Control device, projection system, work guide creation method and program

Abstract

To make it possible to easily acquire information about an object.SOLUTION: A CPU of a control device issues an instruction to project a projection image about an object according to the position of the object based on the object and a first instruction of a first indicator relative to the object recognized from an acquired image, and stores information about the projection image in a storage unit based on a second instruction of a second indicator indicating approval of the projection image projected in accordance with the instruction.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a control device, a projection system, a work guide creation method, and a program. [Background technology]

[0002] Conventionally, projection content showing work procedures tailored to the work object is created on a PC (Personal Computer) or other device, and the image is input into a projector and projected as a work guide at the manufacturing site.

[0003] On the other hand, there is a technology that uses a camera to detect user interactions with a specific location on a physical document, and projects light corresponding to the user interaction with the specific location onto the physical document as visual feedback, or maps the user interaction with the specific location onto a digital document (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-43400 A Summary of the Invention [Problem to be solved by the invention]

[0005] In order to project work guides at manufacturing sites, it is time-consuming to create projection content on a PC or other device that shows work procedures tailored to the work object. Therefore, a technology that can easily acquire information about the object required for projecting work guides, etc. onto the work object is desired. Patent Document 1 describes mapping user interactions with a specific location on a physical document onto a digital document, but it is not possible to acquire information about the object.

[0006] An object of the present invention is to make it possible to easily obtain information about an object. [Means for solving the problem]

[0007] In order to solve the above problem, the control device of the present invention includes a control unit that issues an instruction to project a projection image of an object according to the position of the object based on the object and a first instruction of a first indicator for the object recognized from an acquired image, and stores information about the projection image in a memory unit based on a second instruction of a second indicator indicating approval of the projection image projected in accordance with the instruction. [Effects of the Invention]

[0008] According to the present invention, information about an object can be easily obtained. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the overall configuration of a projection system. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the projection system. [Figure 3] 10 is a flowchart showing the flow of a work guide creation process executed by a CPU of the control device. [Figure 4] FIG. 10 is a diagram showing a state in which a user's finger is pointing at a work object. [Figure 5] FIG. 10 is a diagram showing a state in which an encircling line surrounding a work object is projected. [Figure 6] (a) is a diagram showing a user's finger pointing at a work area in a position that does not overlap with the work object, and (b) is a diagram showing a first guide image projected at the position of the user's finger. [Figure 7] (a) is a diagram showing a user's finger pointing at a position where it overlaps with the work object, and (b) is a diagram showing a second guide image projected at the position of the user's finger. [Figure 8]10 is a flowchart showing the flow of a work guide projection process executed by a CPU of the control device. [Figure 9] FIG. 10 is a diagram showing a state in which a work guide is projected. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are subject to various limitations that are technically preferable for implementing the present invention. Therefore, the technical scope of the present invention is not limited to the following embodiments and illustrated examples.

[0011] First, the configuration of a projection system 1 according to this embodiment will be described. As shown in Fig. 1, the projection system 1 includes a control device 10, a projector (projection unit) 20, and a camera (image acquisition unit) 30. The control device 10 is communicatively connected to the projector 20 and the camera 30 wirelessly or via a wire, and is capable of transmitting and receiving data such as control signals and image data between the control device 10 and the projector 20 and the camera 30.

[0012] In the projection system 1 of this embodiment, the projector 20 and the camera 30 are attached to an arm 50 fixed on a table 40, and can move both in the same direction and by the same distance by movement of the arm 50. The projection system 1 is a system that, for example, when a worker performs work on a work object (object) 60 (see FIG. 4 ) on the table 40, projects a work guide from the projector 20 onto the table 40 as a projection surface, allowing the worker to perform the work while looking at the work guide. The position and direction of the projector 20 are adjusted so that it is oriented so that it can project a projected image onto the table 40. The position and direction of the camera 30 are adjusted so that it can acquire a captured image in the same direction as the projection direction of the projector 20. In other words, the x-axis (horizontal direction of the shooting range) and y-axis (vertical direction of the shooting range) of the projector 20 and the camera 30 are parallel, and the positional relationship (distance) between the projector 20 and the camera 30 is predetermined.

[0013] In this embodiment, the control device 10, the projector 20, and the camera 30 of the projection system 1 are each described as an example of independent devices, but the projection system of the present invention may also be configured with these devices integrated into a single device. Alternatively, the control device 10 and the projector 20 may be integrated into a single device, with the camera 30 being a separate entity from this single device. For example, the control device 10 may be provided inside the housing of the projector 20. Alternatively, the camera 30 and the projector 20 may be integrated into a single device, with the control device 10 being a separate entity from this single device. Alternatively, the control device 10 and the camera 30 may be integrated into a single device, with the projector 20 being a separate entity from this single device.

[0014] As shown in FIG. 2, the control device 10 includes a CPU 11 (Central Processing Unit), a RAM (Random Access Memory) 12, a storage unit 13, and a communication unit 14, and these units are connected by a bus 15.

[0015] The CPU 11 is a processor that reads and executes a program 131 stored in the storage unit 13 and performs various arithmetic processing to control the operation of each unit of the control device 10. In this embodiment, the CPU 11 corresponds to a "control unit" and a "computer." The control unit may have a single processor or multiple processors (e.g., multiple CPUs), and the multiple processes performed by the CPU 11 may be executed by the multiple processors. In this case, the multiple processors correspond to a "control unit." In this case, the multiple processors may be involved in a common process, or the multiple processors may independently execute different processes in parallel. The RAM 12 provides a working memory space for the CPU 11 and temporarily stores data.

[0016] The storage unit 13 is a non-transitory recording medium readable by the CPU 11 as a computer, and stores a program 131 and various data. The storage unit 13 includes a non-volatile memory such as a flash memory. The program 131 includes programs for executing a task guide creation process and a task guide projection process, which will be described later. The program 131 is stored in the storage unit 13 in the form of computer-readable program code.

[0017] The storage unit 13 also stores various data used in the task guide creation process and the task guide projection process. For example, the storage unit 13 stores position information of a second instruction recognition area R2, which is preset in the projection range of the projector 20, and a task area (first instruction recognition area) R1 located on the table 40 other than the second instruction recognition area R2. The second instruction recognition area R2 may or may not be on the table 40. The storage unit 13 also stores images showing gestures such as a task guide creation start gesture, an OK gesture, an NG gesture, a task guide creation completion gesture, and a task guide projection start gesture. The images showing the gestures are used when recognizing each gesture from the captured image acquired by the camera 30. Here, the task guide creation start gesture is a gesture indicating the start of task guide creation. The OK gesture is a gesture indicating that the user approves that the positions of guide images such as a first guide image 71 and a second guide image 72, which will be described later, are appropriate. The NG gesture is a gesture indicating that the user disapproves of a guide image because the position is inappropriate. The work guide creation completion gesture is a gesture indicating that creation of the work guide is complete. The work guide projection start gesture is a gesture indicating an instruction to start projecting the work guide. The storage unit 13 also stores images of the work completion state of each work process for the work object 60 and images of the work completion state of all work processes, display position information (relative position information with the work object 60 as the reference) of the third guide image 73 for the work object 60 for all work processes (see FIG. 9), and image data of the third guide image 73 for all work processes. In this embodiment, the third guide image 73 is projected to have a predetermined relative positional relationship with the work object 60 so as not to overlap with the work object 60 or the worker, which may be placed at any position.

[0018] The communication unit 14 performs a communication operation in accordance with a predetermined communication standard. As this communication operation, the communication unit 14 performs at least one of wireless or wired data transmission and reception with the projector 20 and the camera 30. The communication unit 14 may also be capable of at least one of data transmission and reception with an external device other than the projector 20 and the camera 30.

[0019] The projector 20 includes a CPU 21, a RAM 22, a storage unit 23, a projection unit 24, and a communication unit 25, and these units are connected to each other via a bus .

[0020] The CPU 21 is a processor that reads and executes a program 231 stored in the storage unit 23 and performs various arithmetic processing to control the operation of each unit of the projector 20. The projector 20 may have a single processor or multiple processors (e.g., multiple CPUs), and the multiple processes performed by the CPU 21 may be executed by the multiple processors. In this case, the multiple processors may be involved in a common process, or the multiple processors may independently execute different processes in parallel. The RAM 22 provides a working memory space for the CPU 21 and stores temporary data.

[0021] The storage unit 23 is a non-transitory recording medium readable by the CPU 21 as a computer, and stores the program 231 and various data. The storage unit 23 includes a non-volatile memory such as a flash memory. The program 231 is stored in the storage unit 23 in the form of a computer-readable program code.

[0022] In this embodiment, either the storage unit 13 or the storage unit 23 (storage unit 13 in this embodiment) stores image data of first guide image 71 to third guide image 73 as work guide images to be projected in a work process, in association with identification information (process number) of each work process in a series of work on the work object 60. The first guide image 71 to third guide image 73 are images such as marks, symbols, and characters that are projected onto a projection surface using an OSD (On Screen Display) function. The first guide image 71 is, for example, an image such as an arrow that points to a work location on the work object 60 from a position that does not overlap with the work object 60. The second guide image 72 is, for example, an image of a mark (e.g., circle, triangle, square, etc.) that indicates a work location on the work object 60 on the work object 60, and is projected onto the actual work object 60. For this reason, the second guide image 72 is an image of a different color from the work object 60, and is preferably an image brighter than ambient light or a flashing image to make it easier to recognize. The third guide image 73 is, for example, an image including text showing an explanation of the work. Note that the type of image to be used for each guide image can be set by the user via the US (User Interface) of the projector 20 or the control device 10 (not shown).

[0023] The projection unit 24 is configured to include a light source unit, a display element such as a digital micromirror device (DMD), an optical system such as a group of projection lenses, an optical system drive unit, etc. The projection unit 24 adjusts the intensity distribution of light output from the light source unit using the display element in accordance with the image to be projected, and projects the image onto the projection surface by emitting the projection light toward the projection surface through the group of projection lenses driven by the optical system drive unit.

[0024] The communication unit 25 performs a communication operation in accordance with a predetermined communication standard. As this communication operation, the communication unit 25 performs at least one of transmitting and receiving data wirelessly or via a wire with the control device 10. The communication unit 25 may also be capable of transmitting and receiving data to and from an external device other than the control device 10.

[0025] The camera 30 is configured by an optical camera such as a CCD (Charge Coupled Device) camera, a CMOS (Complementary Metal Oxide Semiconductor Device) camera, etc. The camera 30 includes a communication unit, acquires captured images (acquires moving images) at predetermined time intervals (at a predetermined frame rate), and transmits the acquired captured images to the control device 10 via the communication unit.

[0026] Next, the operation of the projection system 1 will be described. Here, the storage unit 13 is used as a memory for storing data of the first guide image 71 to the third guide image 73 and the encircling line 74. The CPU 11 of the control device 10 first causes the projector 20 to project a monochromatic image of a predetermined color different from the color of the table 40 onto the second instruction recognition area R2 on the table 40 (see FIG. 4), making the user (creator of the work guide) able to identify the second instruction recognition area R2. Next, the CPU 11 activates the camera 30 to capture images at predetermined time intervals, analyzes the captured images, and determines whether or not a work guide creation start gesture (third instruction) of a specific shape pattern made by the user with their own finger 80, for example, with only the index finger extended, has been recognized within the second instruction recognition area R2.

[0027] When it is determined that a work guide creation start gesture has been recognized within the second instruction recognition area R2, the CPU 11 starts the work guide creation process. Hereinafter, the work guide creation process will be described with reference to Fig. 3. In the work guide creation process, as shown in Fig. 4, the user places the work object 60 in the work area R1 on the table 40, points (first instruction) at the work object 60 or a work location in each work process in the work area R1 with a finger 80 (first indicator) as an indicator, and when the finger 80 is kept still for a predetermined time (set time) or more, information required to project a work guide (in this embodiment, first guide image 71 to third guide image 73) when the actual worker is working is acquired accordingly.

[0028] In the work guide creation process, first, the CPU 11 sets the variable N to 1 (step S1). Next, the CPU 11 performs a recognition process for the work object 60 and acquires position information of the contour of the work object 60 (step S2). For example, the CPU 11 analyzes the captured image from the camera 30 and determines whether a finger 80 is present in the captured image and whether an object exists in the direction pointed by the finger 80. As shown in FIG. 4, if it is determined that the finger 80 is present in the captured image and an object exists in the direction pointed by the finger 80, the CPU 11 extracts the contour of the object in the direction pointed by the finger 80 from the captured image and recognizes the extracted contour as the contour of the work object 60. Then, the CPU 11 acquires the position coordinates of the contour of the work object 60 (which may be the position coordinates of a predetermined representative point of the contour of the work object 60) as position information of the work object 60. Alternatively, the CPU 11 may analyze the captured image from the camera 30, and when an action of tracing the outline of an object with the finger 80 is detected, the CPU 11 may recognize the outline of the object traced with the finger 80 as the outline of the work target 60, and acquire the position coordinates thereof as position information of the work target 60. In this embodiment, unless otherwise specified, the upper left corner point of the captured image obtained by the camera 30 is set as the origin of the coordinates.

[0029] Next, CPU 11 transmits image data of the encircling line 74 according to the position information of the recognized work object 60 to projector 20, and causes projector 20 to project the encircling line 74 surrounding the work object 60, as shown in Fig. 5 (step S3). Here, the position information of the encircling line 74 relative to the position of the work object 60 and the position information of the work object 60 are temporarily stored in RAM 12, and the image data of the encircling line 74 and the photographed image data of the work object 60 are stored in memory unit 13.

[0030] Next, the CPU 11 determines whether an OK gesture has been recognized in the second instruction recognition area R2 (step S4). Here, the user visually recognizes the projected image, and if the user determines that the encircling line 74 encircles the work object 60 and that the work object 60 has been correctly recognized, the user makes an OK gesture (second instruction) with the finger 80 (second indicator) within the second instruction recognition area R2. If the encircling line 74 is an inappropriate display, such as not encircling the work object 60, the user makes an NG gesture with the finger 80 within the second instruction recognition area R2. The OK gesture and the NG gesture are preset to different shape patterns, such as one with all five fingers 80 extended, representing a paper shape in rock-paper-scissors, and the other with all five fingers 80 bent, representing a rock shape in rock-paper-scissors. The CPU 11 analyzes the image captured by the camera 30 and determines whether an OK gesture has been recognized within the second instruction recognition area R2. If it is determined that an OK gesture has not been recognized in the second instruction recognition area R2 (step S4; NO), the CPU 11 determines whether an NG gesture has been recognized in the second instruction recognition area R2 (step S5). If it is determined that an NG gesture has not been recognized in the second instruction recognition area R2 (step S5; NO), the CPU 11 returns to step S4. If it is determined that an NG gesture has been recognized in the second instruction recognition area R2 (step S5; YES), the CPU 11 returns to step S2. That is, the recognition process of the work object 60 is performed again. Note that if the encircling line 74 is inappropriately displayed due to, for example, erroneous recognition in step S3, the user may again point with the finger 80 as the indicator at the work object 60 or a work location in each work process without making an OK gesture and keep the finger 80 still for a predetermined period of time or more. This overwrites the image data of the newly projected encircling line 74 surrounding the work object 60 and the position information of the encircling line 74. In this case, step S5 is omitted, and if it is determined that the OK gesture has not been recognized (step S4; NO), the process returns to step S2.

[0031] If it is determined that an OK gesture has been recognized within the second instruction recognition area R2 (step S4; YES), the CPU 11 associates the position information of the work object 60 and the position information of the encircling line 74 relative to the position of the work object 60, which were temporarily stored in the RAM 12 in step S3, with the photographed image data of the work object 60 and the image data of the encircling line 74, which were temporarily stored in the memory unit 13, with the identification information of the variable Nth work process (process number N), and stores them in the memory unit 13 as confirmed data (step S6).

[0032] Next, as shown in Fig. 6(a), the CPU 11 determines whether the user's finger 80 remains stationary for a predetermined period of time or longer in a state where it does not overlap the work object 60 and is pointing in the direction of the work object 60 (step S7). In step S7, the CPU 11 analyzes the image captured by the camera 30 to recognize the finger 80, and determines whether the finger 80 remains stationary for a predetermined period of time or longer in a state where it does not overlap the work object 60 and is pointing in the direction of the work object 60. If it is determined that the finger 80 has not remained stationary for a predetermined period of time or longer in a state where it does not overlap the work object 60 and is pointing in the direction of the work object 60 (step S7; NO), the CPU 11 returns to step S7.

[0033] When it is determined that the user's finger 80 is stationary for a predetermined time or longer in a state (first instruction) where the finger 80 does not overlap with the work object 60 and is pointing in the direction of the work object 60, the CPU 11 acquires the position coordinates of the fingertip of the finger 80 as position information of the finger 80 (position information of the fingertip) (step S8). Next, the CPU 11 transmits image data of the first guide image 71 according to the acquired position information of the finger 80 (image data in which the first guide image 71 is arranged based on the position information of the finger 80) to the projector 20 via the communication unit 14, and causes the projector 20 to project the first guide image 71 at the position of the fingertip of the finger 80, as shown in FIG. 6(b) (step S9). Here, the position information of the finger 80 with respect to the position of the work object 60 is temporarily stored in the RAM 12 as position information indicating the position of the first guide image 71, and the image data of the first guide image 71 is stored in the storage unit 13.

[0034] Next, the CPU 11 determines whether an OK gesture (second instruction) indicating that the user approves because the display of the first guide image 71 is appropriate in the second instruction recognition area R2 has been recognized (step S10). Specifically, the CPU 11 analyzes the captured image from the camera 30 and determines whether an OK gesture has been recognized in the second instruction recognition area R2. If it is determined that an OK gesture has not been recognized in the second instruction recognition area R2 (step S10; NO), the CPU 11 determines whether an NG gesture indicating that the user does not approve because the display of the first guide image 71 is inappropriate for reasons such as being misaligned from the desired position has been recognized (step S11). If it is determined that an NG gesture has not been recognized in the second instruction recognition area R2 (step S11; NO), the CPU 11 returns to step S10. If it is determined that an NG gesture has been recognized in the second instruction recognition area R2 (step S11; YES), the CPU 11 returns to step S7. That is, if the position of the first guide image 71 is not correct, the user can adjust the position of the first guide image 71 by moving the finger 80 to point to the location to be worked on.

[0035] When it is determined that the guide setting OK gesture has been recognized within the second instruction recognition area R2 (step S10; YES), the CPU 11 reads out the position information of the first guide image 71 relative to the position of the work object 60, which was temporarily stored in the RAM 12 in step S9 (step S12). The read-out position information of the first guide image 71 relative to the position of the work object 60 is information indicating the relative position of the first guide image 71 when the work object 60 is used as a reference.

[0036] Next, the CPU 11 stores the display position information (position information indicating the projection position) of the first guide image 71 relative to the position of the work object 60 that has been read out and the image data that will become the first guide image 71 as confirmed data in the memory unit 13 in association with the identification information (process number N) of the Nth work process, which is a variable (step S13).

[0037] Next, as shown in Fig. 7(a), the CPU 11 determines whether the user's finger 80 points to a specific part of the work object 60 at a position overlapping the work object 60 (first instruction) and remains stationary for a predetermined time or longer (step S14). In step S14, the CPU 11 analyzes the image captured by the camera 30 to recognize the finger 80, and determines whether the finger 80 (fingertip) remains stationary for a predetermined time or longer at a position overlapping the work object 60. If it is determined that the finger 80 has not remained stationary for a predetermined time or longer at a position overlapping the work object 60 (step S14; NO), the CPU 11 returns to step S14.

[0038] When it is determined that the user's finger 80 has remained stationary at a position overlapping the work object 60 for a predetermined period of time or longer (step S14; YES), the CPU 11 acquires the position coordinates of the fingertip of the finger 80 as position information of the finger 80 (position information of the fingertip) (step S15). Next, the CPU 11 transmits image data of the second guide image 72 according to the acquired position information (image data in which the second guide image 72 is arranged based on the position information of the finger 80) to the projector 20 via the communication unit 14, and causes the projector 20 to project the second guide image 72 at the position of the fingertip of the finger 80, as shown in FIG. 7(b) (step S16). Here, the position information of the finger 80 with respect to the position of the work object 60 is temporarily stored in the RAM 12 as position information indicating the position of the second guide image 72, and the image data of the second guide image 72 is stored in the storage unit 13.

[0039] Next, the CPU 11 determines whether an OK gesture (second instruction) indicating that the user approves because the display of the second guide image 72 is appropriate has been recognized in the second instruction recognition area R2 (step S17). The CPU 11 analyzes the captured image from the camera 30 and determines whether an OK gesture has been recognized in the second instruction recognition area R2. If it is determined that an OK gesture has not been recognized in the second instruction recognition area R2 (step S17; NO), the CPU 11 determines whether an NG gesture indicating that the user does not approve because the display of the second guide image 72 is inappropriate for reasons such as being misaligned from the desired position has been recognized (step S18). If it is determined that an NG gesture has not been recognized in the second instruction recognition area R2 (step S18; NO), the CPU 11 returns to step S17. If it is determined that an NG gesture has been recognized in the second instruction recognition area R2 (step S18; YES), the CPU 11 returns to step S14. That is, if the position of the second guide image 72 is not correct, the user can adjust the position of the second guide image 72 by moving the finger 80 to point to the location to be worked on.

[0040] When it is determined that the OK gesture has been recognized within the second instruction recognition area R2 (step S17; YES), the CPU 11 reads out the position information (relative position information) of the second guide image 72 with respect to the position of the work object 60, which was temporarily stored in the RAM 12 in step S16 (step S19). The process of step S19 is the same as that of step S12 described above, and therefore the description thereof will be used.

[0041] Next, the CPU 11 stores the display position information (information indicating the projection position) of the second guide image 72 relative to the position of the work object 60 that has been read out and the image data of the second guide image 72 as confirmed data in the memory unit 13, in association with the identification information of the Nth work process (process number N) (step S20).

[0042] Next, the CPU 11 determines whether a work guide creation completion gesture has been recognized within the second instruction recognition area R2 within a set time (step S21). The CPU 11 analyzes the image captured by the camera 30 and determines whether a work guide creation completion gesture by the user has been recognized within the second instruction recognition area R2. If it determines that a work guide creation completion gesture has not been recognized within the second instruction recognition area R2 (step S21; NO), the CPU 11 increments the variable N (step S22), returns to step S7, and repeats the processes of steps S7 to S21 for the next work process. Note that if the image of the work object 60 differs from the image of the previous process because, for example, another part is combined with the work object 60 as the work on the work object 60 progresses, the CPU 11 returns to step S2 and performs the processes of steps S2 to S21 to recognize the work object 60 processed appropriately according to the variable N. When it is determined that the work guide creation completion gesture has been recognized within the second instruction recognition area R2 (step S21; YES), the CPU 11 ends the work guide creation process.

[0043] According to the above-described work guide creation process, when the user points with the finger 80 a work location on the work object 60, the CPU 11 acquires position information of the finger 80 relative to the position of the work object 60, issues an instruction (first instruction) to the projector 20 to project a guide image related to the work object 60 at a position according to the acquired position information, and, based on an OK gesture (second instruction) indicating approval of the guide image projected in accordance with the projection instruction, can store the position information (relative position information) of the finger 80 relative to the position of the work object 60 in the storage unit 13 as position information indicating the projection position when projecting a guide image indicating the work location on the work object 60. Thus, the user can easily acquire information related to the work object 60, such as the position information of the projected images related to the work object 60 (the first to third guide images 71 to 73 and the encircling line 74) simply by making a simple gesture of pointing to the work location on the work object 60 and then making a gesture of approval after checking the projected guide image in response to the gesture. Furthermore, the CPU 11 of the control device 10 controls the recognition so that, in the work area R1, it can recognize a first instruction as an instruction to project a candidate image of a work guide, but cannot recognize a second instruction as an instruction to approve or not the candidate image, an instruction to start creating a work guide, or an instruction to complete creating a work guide, and in the second instruction recognition area R2, it can recognize the second instruction but cannot recognize the first instruction. Therefore, by previously distinguishing the instruction area dedicated to the first instruction from the instruction area dedicated to the second instruction, it is possible to prevent erroneous recognition of the type of instruction given by the user even with a single instruction information acquisition means (camera 30). Furthermore, the CPU 11 of the control device 10 may control the projector 20 to automatically project a first guide image 71 when the finger 80 points to a position that does not overlap with the work object 60, and to automatically project a second guide image 72 when the finger 80 points to a position that overlaps with the work object 60.In this case, the user does not need to always remember the order of steps S6 to S13 and steps S14 to S20. If the user first points to a position where the finger 80 overlaps the work object 60 and then points to a position where the finger 80 does not overlap the work object 60, the CPU 11 of the control device 10 will control the process to first perform steps S14 to S20 and then perform steps S6 to S13.

[0044] Next, a description will be given of the work guide projection process that is executed by the CPU 11 in cooperation with the program 131 when a work guide projection start gesture is recognized in the second instruction recognition area R2. The worker makes a work guide projection start gesture in the second instruction recognition area R2, and performs work on the work object 60 placed on the table 40 according to the projected guide.

[0045] As shown in FIG. 8, in the work guide projection process during work, first, the CPU 11 sets the variable N to 1 (step S31). Next, the CPU 11 performs a recognition process for the work object 60 (step S32). The CPU 11 searches for an image of the work object 60 in the Nth work process stored in the storage unit 13 from the captured images acquired by the camera 30, and performs a recognition process for the work object 60. Note that the position of the work object 60 in the work guide projection process does not necessarily match the position in the work guide creation process. The CPU 11 acquires the position coordinates of the contour of the recognized work object 60 as position information of the work object 60.

[0046] Next, in the Nth work process, CPU 11 causes communication unit 14 to transmit to projector 20 image data of encircling line 74 (fourth guide image) whose position has been adjusted based on the position information of encircling line 74 relative to the position of work object 60 and the position information of work object 60 in step S32, and causes projector 20 to project encircling line 74 so as to surround work object 60 (step S33). This allows the worker to confirm which object on table 40 is work object 60 that he or she is currently working on.

[0047] Next, the CPU 11 reads out the display position information of the first guide image 71 to the third guide image 73 relative to the position of the work target 60 in the Nth work process from the storage unit 13 (step S34). Next, the CPU 11 specifies the display position information (absolute position) of each of the first guide image 71 to the third guide image 73 in the Nth work process based on the display position information (relative position) of the first guide image 71 to the third guide image 73 in the Nth work process and the position information of the work target 60 acquired in step S32 (step S35). That is, the CPU 11 converts the display position information of the first guide image 71 to the third guide image 73, which is shown based on the position of the work target 60, into display position information in the coordinate space on the captured image based on the position information on the captured image of the work target 60 currently being worked on.

[0048] Next, the CPU 11 transmits image data, in which the first guide image 71 to the third guide image 73 are combined, to the projector 20 via the communication unit 14 in accordance with the display position information (absolute positions) of the first guide image 71 to the third guide image 73, the positions of which have been adjusted based on the position of the work target object 60 in the Nth work process, and the projector 20 projects the first guide image 71 to the third guide image 73 for the Nth work process based on the image data (step S36). Note that in this embodiment, the CPU 11 transmits image data, in which the encircling line 74 is also combined with the first guide image 71 to the third guide image 73, to the projector 20 and projects them.

[0049] As shown in FIG. 9 , in the work guide projection process, an encircling line 74 is displayed around the work target 60 placed on the table 40. Furthermore, a first guide image 71 and a second guide image 72 indicating the work location for each work step are projected. Furthermore, a third guide image 73 is projected near the work target 60 as an explanation of the work for that step. Preferably, the CPU 11 creates image data based on an image including the worker captured by the camera 30, with the third guide image 73 projected at an arbitrary position in the work area R1 on the table 40 so that the projected third guide image 73 does not overlap with the worker. Therefore, the worker can easily recognize which is the actual work target 60, which part of the work target 60 is the work location, and what kind of work should be performed at the work location.

[0050] Next, CPU 11 compares the image captured by camera 30 with the image of the completed state of the Nth work process stored in storage unit 13, and determines whether or not the Nth work process has been completed (step S37). If CPU 11 determines that the Nth work process has not been completed (step S37; NO), CPU 11 repeats step S37.

[0051] When it is determined that the Nth work process has been completed (step S37; YES), the CPU 11 compares the image captured by the camera 30 with the images of the work completion states of all work processes stored in the storage unit 13, and determines whether all work processes have been completed (step S38). When it is determined that all work processes have not been completed (step S38; NO), the CPU 11 increments the variable N (step S39), returns to step S32, and repeats the processes of steps S32 to S38. Here, during the process loop of steps S32 to S38, the CPU 11 monitors whether there has been any change in the position of the work target 60 based on the image captured by the camera 30. When there has been a change in the position of the work target 60, the CPU 11 updates the position information of the work target 60 and the display position information (absolute positions) of the first guide image 71 to the third guide image 73, and transmits them to the projector 20, and changes the projection positions of the encircling line 74 and the first guide image 71 to the third guide image 73.

[0052] If it is determined that all work steps have been completed (step S38; YES), the CPU 11 ends the work guide projection process.

[0053] As described above, the CPU 11 of the control device 10 recognizes the work object 60 and the finger 80 pointing at the work object 60 from the image acquired by the camera 30 that acquires an image in the projection direction of the projector 20, and acquires position information of the finger 80 relative to the work object 60. Based on the acquired position information of the finger 80 relative to the work object 60, the CPU 11 stores in the storage unit 13 the position information indicating the projection position when the projection unit 24 projects a guide image indicating the work location, etc. of the work object 60 onto the work object 60. Therefore, the user can easily acquire the position information of the projection position of the guide image, which is information necessary for projecting a work guide onto the work object 60, by simply making a simple gesture of pointing to the work location on the work object 60.

[0054] For example, after recognizing the finger 80 from the image acquired by the camera 30, the CPU 11 causes the projection unit 24 to project a guide image at the position of the finger 80, and when an OK gesture indicating approval of the position of the guide image is recognized from the image acquired by the camera 30, the CPU 11 causes the storage unit 13 to store position information of the finger 80 relative to the position of the work object 60. Therefore, after the user confirms whether the guide image is correctly projected, the storage unit 13 can store the acquired position information of the finger 80 relative to the work object 60 as position information indicating the projection position of the guide image indicating the work location.

[0055] Furthermore, when the OK gesture is not recognized from the image acquired by the camera 30, the CPU 11 repeats the process of recognizing the finger 80 from a new image acquired by the camera 30 and causing the projection unit 24 to project a guide image at the position of the finger 80 until the OK gesture is recognized from the image acquired by the camera 30. Therefore, the user can adjust the guide image so that it is projected at an optimal position.

[0056] Furthermore, for each work process on the work object 60, the CPU 11 recognizes the finger 80 from the image acquired by the camera 30, acquires position information of the finger 80 relative to the work object 60 in that work process, and stores the acquired position information in association with the work process in the storage unit 13. Therefore, it is possible to acquire information necessary for projecting a work guide for each work process on the work object.

[0057] Furthermore, when a work guide creation start gesture indicating the start of work guide creation is recognized from an image acquired by camera 30, CPU 11 starts a process of acquiring an image of finger 80 with camera 30 and acquiring position information of finger 80 relative to work object 60, and when a work guide creation completion gesture indicating the completion of work guide creation is recognized from an image acquired by camera 30, CPU 11 ends the process of acquiring an image of finger 80 with camera 30 and acquiring position information of finger 80 relative to work object 60. Therefore, the user can easily give instructions to start work guide creation and to end work guide creation with gestures.

[0058] Furthermore, when the CPU 11 recognizes, based on the image acquired by the camera 30, that the finger 80 has been stationary for a predetermined time or longer in a state in which it is pointing in the direction of the work object 60 at a position that does not overlap with the work object 60, the CPU 11 causes the projection unit 24 to project the first guide image 71 at the position of the finger 80. Then, when an OK gesture indicating approval of the position of the first guide image 71 is recognized from the image acquired by the camera 30, the CPU 11 causes the storage unit 13 to store position information of the finger 80 relative to the position of the work object 60 as position information indicating the projection position when the projection unit 24 projects the first guide image 71 onto the work object 60. Therefore, the user can easily obtain position information indicating the position at which the first guide image 71 is to be projected by a simple gesture of simply holding the finger 80 stationary for a predetermined time or longer in a state in which it is pointing in the direction of the work object 60 at a position that does not overlap with the work object 60.

[0059] Furthermore, when the CPU 11 recognizes, based on the image acquired by the camera 30, that the finger 80 has remained stationary for a predetermined time at a position overlapping the work object, it causes the projection unit 24 to project the second guide image 72 onto the position of the finger 80, and when an OK gesture indicating approval of the position of the second guide image 72 is recognized from the image acquired by the camera 30, it causes the memory unit 13 to store the position information of the finger 80 relative to the work object 60 as position information indicating the projection position when the projection unit 24 projects the second guide image 72 onto the work object 60. Therefore, the user can easily obtain the position information indicating the position at which the second guide image 72 is to be projected, with a simple gesture of simply remaining stationary for a predetermined time at a position overlapping the work object. Furthermore, the projection image (first guide image 71) that instructs projection when it is recognized that the finger 80 is stationary in a position that does not overlap the work object 60 for a predetermined period of time or more, and the projection image (second guide image 72) that instructs projection when it is recognized that the finger 80 is stationary in a position that overlaps the work object 60 for a predetermined period of time or more are different from each other, so different projection images can be projected depending on the gesture made by the finger 80.

[0060] Furthermore, when the CPU 11 causes the projection unit 24 to project the first guide image 71 or the second guide image 72 onto the work object 60, it recognizes the work object 60 from the image acquired by the camera 30, acquires position information of the work object 60, and causes the projection unit 24 of the projector 20 to project the first guide image 71 and the second guide image 72 based on the acquired position information of the work object 60 and the position information of the finger 80 relative to the work object 60 stored in the memory unit 13. Therefore, during actual work, the worker can easily recognize the work location on the work object 60.

[0061] Furthermore, based on the position information of the work target 60, the CPU 11 further causes the projection unit 24 of the projector 20 to project at least one of a third guide image showing an explanation of the work or a fourth guide image surrounding the work target. Therefore, when working, the worker can easily identify the work target 60 even if various objects are placed on the table 40. Also, the worker can easily recognize what kind of work should be done at the work location.

[0062] The above-described embodiment is a preferred example of the control device, projection system, work guide creation method, and program according to the present invention, and is not limited thereto. For example, in the above-described embodiment, the position of the third guide image 73 relative to the work object 60 is predetermined, but it may be variable. For example, the CPU 11 acquires position information of the first guide image 71 and the second guide image 72 for all work steps in the work guide creation process and stores the information in the memory unit 13, and then instructs the projector 20 to project the third guide image 73 onto the work area R1 in accordance with the operation of the finger 80 in the work area R1. Specifically, the CPU 11 transmits image data to the projector 20 so as to project the third guide image 73 at a position that does not overlap with the first guide image 71 and the second guide image 72 for any work step and that is easily visible and corresponds to the position of the work object 60. After the projector 20 projects an image based on the image data, the CPU 11 stores the position information of the third guide image 73 relative to the position of the work object 60 and the image data of the third guide image 73 in the storage unit 13 in accordance with the user's approval instruction in the second instruction recognition area R2. Alternatively, unlike determining the position information of the third guide image 73 in the work guide creation process as described above, in each work process, the CPU 11 may acquire the position information of the first guide image 71 and the second guide image 72, and then determine the position of the third guide image 73 in that work process at a position that does not overlap with the first guide image 71, the second guide image 72, and the worker in that work process. Alternatively, in the work guide creation process, the user may point to an arbitrary position to determine the projection position of the third guide image 73, just like the first guide image 71 and the second guide image 72.In addition, while a worker is working on the work object 60 based on the work guide instructions of the guide image 73, in order to prevent the second guide image 72 projected onto the work object 60 from overlapping with the worker or a work tool such as a screwdriver held by the worker and thereby interfering with the work, if it is recognized from the image of the camera 30 that the worker or a work tool has overlapped with the second guide image 72 and performed work on a part of the second guide image 72, the CPU 11 may instruct the projector 20 to stop projecting the second guide image 72 midway through the work.

[0063] In the above embodiment, the positions of both the first guide image 71 and the second guide image 72 are determined and projected in the work guide projection process, but the position of only one of them may be determined and projected. Also, with regard to the third guide image 73 and the encircling line 74, only one of them may be projected.

[0064] Furthermore, in the above embodiment, after the processing of steps S7 to S20 for one work process is completed, if a work guide creation completion gesture is not recognized, the variable N is automatically incremented and processing of step S7 for the next work process is started. However, if it is recognized that the user has made a gesture to start work guide creation after step S20 is completed, processing of step S7 for the next work process may also be started.

[0065] In addition, in the final work step, the user may perform a work guide creation completion gesture instead of an OK gesture. Then, when the CPU 11 recognizes the work guide creation completion gesture in step S10 or step S14, it may end the work guide creation process without recognizing the work guide creation completion gesture after the process of step S20 is completed.

[0066] In addition, in the above embodiment, if the position of the guide image projected in step S9 or step S16 is incorrect, the user can reset the position of the guide image by performing an NG gesture, but the NG gesture may be omitted.

[0067] In addition, in the above embodiment, the projector 20 stores the data for projecting the first guide image 71 to the third guide image 73 and the encircling line 74, but the data may be stored in the memory unit 13 of the control device 10, and the projection data may be transmitted to the projector 20 together with the display position information of each image.

[0068] In the above embodiment, the position information of each guide image relative to the work object 60 is acquired as relative position information so that the position of the work object 60 during the work guide creation process may differ from the position of the work object 60 during the work guide projection process. On the other hand, if the position of the work object 60 is fixed, the position information of each guide image relative to the work object 60 may be acquired as, for example, position information (absolute position) in the coordinate space of the captured image.

[0069] In the above embodiment, the case where the indicator pointing to the work location is a finger 80 has been described as an example, but the indicator is not limited to a finger and may be, for example, a pen or a wand. Furthermore, when there is only one user, the indicator pointing in the work area R1 (first indicator) and the indicator in the second instruction recognition area R2 (second indicator) may be the same indicator such as a hand, or different indicators such as different hands may be used. Furthermore, when there are multiple users, a first user may point in the work area R1, and a second user different from the first user may point in the second instruction recognition area R2.

[0070] In the above embodiment, an example has been disclosed in which an HDD or semiconductor memory is used as the computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. Other computer-readable media may be used, such as flash memory or CD-ROM. Furthermore, a carrier wave may also be used as a medium for providing data for the program according to the present invention via a communication line.

[0071] Furthermore, it goes without saying that the detailed configurations and operations of the components of the control device 10, projector 20, and camera 30 in the above embodiment can be modified as appropriate without departing from the spirit of the present invention. [Explanation of symbols]

[0072] 10 control device, 11 CPU, 13 memory unit, 20 projector, 30 camera

Claims

1. giving an instruction to project a projection image relating to the object according to a position of the object, based on the object and a first instruction of a first indicator with respect to the object, which are recognized from the acquired image; storing information about the projected image in a storage unit based on a second instruction from a second indicator indicating approval of the projected image projected in accordance with the instruction; A control device having a control unit.

2. the first indicator is the second indicator, The control unit recognizing the second instruction from an image including the second instruction of the second indicator; The control device according to claim 1 , further comprising: storing, in the storage unit, position information of the projection image relating to the object relative to a position of the object as information relating to the projection image in accordance with the recognition of the second instruction.

3. The control unit In a first instruction recognition area, the first instruction can be recognized but the second instruction cannot be recognized; The control device according to claim 1 , wherein the second instruction can be recognized but the first instruction cannot be recognized in a second instruction recognition area.

4. The control unit 3. The control device according to claim 2, wherein, if the second instruction is not recognized from the image, the instruction to project a new projection image is repeated based on a new first instruction recognized from a newly acquired new image until the second instruction is recognized.

5. The control unit and storing an instruction to project the projection image based on the first instruction and information about the projection image based on the second instruction for each step on the object. The control device according to claim 1 .

6. The control unit 2. The control device according to claim 1, wherein the projection image that instructs projection when it is recognized based on an image that the first indicator has remained stationary for a predetermined period of time or more in a position that does not overlap with the object, and the projection image that instructs projection when it is recognized that the first indicator has remained stationary for a predetermined period of time or more in a position that overlaps with the object, are different from each other.

7. The control unit and after storing information about the projection image in the storage unit, further instructing the projection of a projection image showing an explanatory text about the object based on the position information of the object. The control device according to claim 1 .

8. A control device according to any one of claims 1 to 7; a projection unit, an image acquisition unit, A projection system comprising:

9. The computer giving an instruction to project a projection image relating to the object according to a position of the object, based on the object and a first instruction of a pointer to the object, which are recognized from the acquired image; storing information about the projected image in a storage unit based on a second instruction by the indicator indicating approval of the projected image projected in accordance with the instruction; How to create a work guide.

10. On the computer, instructing projection of a projection image relating to the object according to a position of the object, based on the object and a first instruction of a pointer to the object, which are recognized from the acquired image; storing information about the projected image in a storage unit based on a second instruction by the indicator indicating approval of the projected image projected in accordance with the instruction; A program for executing a process.

Images