Control device, industrial machine, and computer-readable storage medium

Abstract

A control device according to the present invention acquires a three-dimensional image from an imaging device, detects a gesture from the three-dimensional image, and further detects that the gesture has been performed in the vicinity of a spatial region. When a prescribed condition is met, the control device adjusts the spatial region so that the gesture is performed at a position close to the center of the spatial region.

Description

Storage media readable by control devices, industrial machinery, and computers.

[0001] This disclosure relates to a storage medium readable by a control device, industrial machinery, and computer.

[0002] For operating industrial machinery such as machine tools and robots, methods include remote control using tablet devices and operation using augmented reality (AR) technologies such as smart glasses. For example, see Patent Document 1.

[0003] Industrial machinery operators often wear safety equipment such as helmets, goggles, and gloves. Cutting oil and chips can adhere to gloves. These cutting oils and chips on gloves may then adhere to tablet devices. Furthermore, using smart glasses while wearing a helmet and goggles is difficult, and removing safety equipment to wear smart glasses is not advisable from a safety standpoint.

[0004] Japanese Patent Publication No. 2020-067956

[0005] Traditionally, technologies have existed that allow for contactless operation of industrial machinery using gestures. However, the area and boundaries where gestures are detected are not explicitly defined; they are simply empty space. In other words, the area where gestures are detected is not clear to the operator. The default area is not necessarily suitable for detecting the operator's gestures. If the operator's body movements do not match the area where gestures are detected, the accuracy of gesture detection will decrease.

[0006] There is a need to improve the detection accuracy of non-contact operations in industrial machinery.

[0007] The control device according to this disclosure includes: an information acquisition unit that acquires range information defining the range of a spatial area virtually provided near an industrial machine, and operation information defining the content of gesture operations received in the spatial area; a three-dimensional image acquisition unit that acquires a three-dimensional image from an imaging device; a boundary operation detection unit that detects a gesture from the three-dimensional image and further detects that the gesture was performed near the boundary of the spatial area; a range adjustment unit that adjusts the spatial area so that the gesture is performed at a position close to the center of the spatial area when predetermined conditions are met; a gesture determination unit that determines that the gesture is performed in the spatial area and that the gesture is defined in the operation information; an operation determination unit that determines the content of an operation corresponding to the gesture; and an operation application unit that applies the content of an operation to the industrial machine.

[0008] This is a block diagram of the control device. This is a perspective view of an industrial machine to explain the spatial domain. This is a diagram explaining spatial domain information. This is a diagram explaining gestures performed near the boundary of the spatial domain. This is a diagram explaining boundary crossing and movement within the spatial domain by gesture. This is a diagram explaining boundary crossing and movement within the spatial domain by gesture. This is a diagram explaining boundary crossing and movement within the spatial domain by gesture. This is a diagram explaining boundary crossing and expansion within the spatial domain by gesture. This is a diagram explaining the execution of a gesture near the boundary and movement within the spatial domain. This is a diagram explaining the execution of a gesture near the boundary and movement within the spatial domain. This is a diagram explaining the confirmation of whether the spatial domain can be adjusted. This is a hardware configuration diagram of the control device.

[0009] Embodiments of this disclosure will be described below with reference to the drawings. In the following description, components having the same or similar functions will be denoted by the same reference numerals. Duplication of these components may be omitted.

[0010] In this application, "based on XX" means "based on at least XX," and includes cases where it is based on another element in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on something that has been calculated or processed. "XX" is any element (for example, any information).

[0011] As a prerequisite, the relationship between the control device 100, the industrial machine 200, and the imaging device will be explained. The control device 100 is a numerical control device that controls machine tools, a robot controller that controls robots, etc. The control device 100 acquires a three-dimensional image from the imaging device. The imaging device is a ToF camera, a stereo camera, etc. A ToF camera measures the distance to an object using reflected light. A stereo camera measures the distance to an object using parallax. The image captured by the ToF camera is called a depth image. Images captured by ToF cameras and stereo cameras cannot be used directly for gesture detection. In order to recognize an object, it is necessary to derive three-dimensional spatial coordinates that indicate the position of the object. In this embodiment, the three-dimensional spatial information used for gesture detection is called a three-dimensional image. The type of data acquired from the imaging device is not particularly limited. Other means may be used as long as three-dimensional spatial information can ultimately be obtained.

[0012] A virtual spatial area is set up in the vicinity of the control device 100. This spatial area is an area for detecting gestures to operate the industrial machine 200. The spatial area is associated with the objects of operation of the industrial machine 200. These objects of operation include feed axes, operation screens, turrets, and control panels. The operator performs gestures, such as moving their hands, while looking at the objects of operation. The types of objects of operation are examples only and are not particularly limited.

[0013] Each spatial domain has spatial domain information defined within it. This spatial domain information defines the range of the spatial domain, the gestures associated with that spatial domain, and the operations associated with those gestures.

[0014] Figure 1 is a block diagram of the control device 100. The control device 100 includes an information acquisition unit 11, a detection unit 12, a gesture determination unit 13, an operation determination unit 14, an operation application unit 15, a drive control unit 16, a boundary motion detection unit 17, an area adjustment unit 18, a notification unit 19, a display unit 20, and a 3D image acquisition unit 21.

[0015] The information acquisition unit 11 acquires information related to the spatial domain (referred to as spatial domain information). The spatial domain information includes range information indicating the extent (location and size) of the spatial domain, and operation information relating to gestures accepted in the spatial domain and the corresponding operation content. The spatial domain information (range information and operation information) may be acquired from within the control device 100, or from an external cloud or network storage. The storage location of the spatial domain information is not limited.

[0016] Figure 2 shows an example of a spatial area provided in a machining center. A machining center is an industrial machine 200 in which a control device 100 (numerical control device) and a machine tool are integrated. In this embodiment, the spatial area of ​​a machining center is described, but a spatial area can also be defined for machine tools other than machining centers, robots, etc.

[0017] In Figure 2, the object of operation in spatial region 1 is the turret. The object of operation in spatial region 2 is the operation screen of the control device 100. The object of operation in spatial region 3 is the feed axis of the machine tool (industrial machine 200).

[0018] Spatial domain information includes spatial domain range information and operation information. An example of spatial domain information is explained with reference to Figure 3. In the range information of Figure 3, the position and size of the spatial domain are represented by a position vector and the spatial domain itself. The position vector indicates the position of the spatial domain relative to a certain point (reference position). The spatial length represents the size of the spatial domain. The spatial length is expressed as the length in the X, Y, and Z axis directions. In this embodiment, the reference position is the position of the imaging device.

[0019] The range of spatial region 1 is defined by the position vector (X １ , Y １ , Z １ ) and spatial length (X １ｌ , Y１ｌ , Z １ｌ ) is expressed. Similarly, the range information of the spatial region 2 is the position vector (X ２ , Y ２ , Z ２ ), the spatial length (X ２ｌ , Y ２ｌ , Z ２ｌ ), and the range information of the spatial information 3 is the position vector (X ３ , Y ３ , Z ３ ), the spatial length (X ３ｌ , Y ３ｌ , Z ３ｌ ). In FIG. 3, the spatial region is a rectangular parallelepiped, but the shape of the spatial region is not limited to this.

[0020] In the operation information of the spatial region, a gesture for the operation target and the operation content associated with the gesture are defined. In FIG. 3, as the operation information 1 of the operation target "turret" in the spatial region 1, "gesture 11" and "turret operation 1", "gesture 12" and "turret operation 2", etc. are defined. As the operation information 2 of the operation target "display screen" in the spatial region 2, "gesture 21" and "screen operation 1", "gesture 22" and "screen operation 2", etc. are defined. As the operation information 3 of the operation target "feed shaft" in the spatial region 3, "gesture 31" and "feed shaft operation 1", "gesture 32" and "feed shaft operation 2", etc. are defined.

[0021] The 3D image acquisition unit 21 acquires data that is the source of the 3D image or the 3D image from the imaging device. The imaging device is a ToF camera, a stereo camera, etc., but its type is not particularly limited. Other means may be used as long as 3D spatial information can be finally acquired.

[0022] The detection unit 12 detects a gesture from the acquired 3D image. The detection range of the gesture does not necessarily have to be only inside the spatial region. It may include the vicinity of the spatial region.

[0023] The gesture determination unit 13 determines a gesture based on the 3D image detected in the spatial region and the spatial region information acquired by the information acquisition unit 11. The determination of the gesture is an existing technology, so the explanation is omitted.

[0024] The operation determination unit 14 determines the operation content from the gesture based on the operation information.

[0025] The operation application unit 15 applies the operation content determined by the operation determination unit 14. Specifically, it outputs control information based on the operation content to the drive control unit 16. The drive control unit 16 drives the industrial machine 200 by controlling the drive unit 210 of the industrial machine 200 according to the gesture operation content.

[0026] The boundary motion detection unit 17 detects that a gesture was performed near the boundary of a spatial domain. The boundary of a spatial domain refers to the boundary between the inside and outside of the spatial domain. Gestures near a spatial domain include gestures that do not cross the boundary of the spatial domain and gestures that cross the boundary of the spatial domain, as shown in Figure 4.

[0027] The boundary motion detection unit 17 counts the number of times a gesture is performed near the boundary of a spatial area (within a predetermined threshold from the boundary) if it detects a gesture that does not cross the boundary. Furthermore, if the boundary motion detection unit 17 detects a gesture that crosses the boundary, it counts the number of times the gesture crosses the boundary of the spatial area.

[0028] The region adjustment unit 18 adjusts the spatial region so that the gesture is performed near the center of the spatial region when the number of times exceeds a predetermined threshold. Adjustment methods include moving the spatial region by a predetermined distance, expanding the spatial region by a predetermined distance, or moving or expanding the spatial region along each axis.

[0029] The adjustment of the spatial domain will be explained with reference to the diagram. Referring to Figure 5, an example of spatial domain adjustment when a gesture crossing the boundary of the spatial domain is detected will be explained. The target of the spatial domain information in Figure 5 is the "feed axis". The initial settings for the range information are the position vector (0, -100, -1200) and the spatial length (800, 400, 500). The operation information is defined as follows: gesture "swipe up with two fingers" and operation content "move tool axis upward", gesture "swipe down with two fingers" and operation content "move tool axis downward", gesture "swipe right with two fingers" and operation content "move table axis to the right", gesture "swipe left with two fingers" and operation content "move table axis to the left", gesture "push back with two fingers" and operation content "move table axis back", and gesture "pull forward with two fingers" and operation content "move table axis forward".

[0030] The count threshold is a threshold for adjusting the spatial domain. In Figure 5, the count threshold is defined as "2". When the number of times the boundary of the spatial domain is crossed exceeds the threshold (2 times), the domain adjustment unit 18 adjusts the position or size of the spatial domain. The adjustment distance is the distance by which the spatial domain is adjusted. In Figure 5, the adjustment distance is defined as "200 mm". When the number of times the spatial domain has been crossed exceeds the threshold (2 times), the spatial domain is moved by "200 mm". As a modification, the adjustment distance may be defined separately for each axis and direction. For example, the spatial domain may be adjusted with different adjustment distances for each axis, such as "100 mm" in the X-axis direction and "50 mm" in the upward direction.

[0031] An example of adjusting the spatial domain will be explained with reference to Figures 6 to 11. Figure 6 is an example in which a gesture detects boundary crossing, and in particular, the number of crossings is counted in a common way regardless of which axial direction the crossing includes outside the domain, and the area outside the domain included during the crossing is newly included as part of the spatial domain. Three operations are performed: swiping two fingers to the right, swiping two fingers up, and swiping two fingers to the left. In the first operation, swiping two fingers to the right, the boundary is crossed so as to include the area outside the domain in the negative X-axis direction. Therefore, the number of crossings for the gesture becomes "1". In the second operation, swiping two fingers up, the boundary is crossed so as to include the area outside the domain in the positive Z-axis direction. Therefore, the number of crossings for the gesture becomes "2". In the third operation, swiping two fingers to the left, the boundary is crossed so as to include the area outside the domain in the negative X-axis direction. Here, since the number of crossings including outside the region exceeds the threshold "2", the region adjustment unit 18 moves the spatial region by an adjustment distance of "200 mm" in the negative X-axis direction and the positive Z-axis direction where the gesture is outside the region. The position vector of the adjusted spatial region moves from (0, -100, -1200) to (-200, -100, -1000). Here, the spatial region is moved, but the spatial length (800, 400, 500) does not change. The region adjustment unit 18 updates the range information, moves the spatial region, and resets the counter to zero. The notification unit 19 notifies the user of the movement of the spatial region by voice or screen display, for example, "The spatial region for detecting gestures has been moved 200 mm in the negative X-axis direction and the positive Z-axis direction."

[0032] Referring to Figure 7, an example of adjustment when a gesture crosses a boundary will be explained. The initial settings and operation information for the range information are the same as in Figure 5, so the explanation will be omitted. In the example in Figure 7, the number of times the gesture crosses the boundary is counted for each axis and direction, and the spatial area is adjusted for each axis. Figure 7 shows an example of performing four operations: swiping left with two fingers, swiping right with two fingers, swiping up with two fingers, and swiping left with two fingers. In the first operation, the boundary is crossed so as to include the area outside the negative X-axis direction. In the second operation, the boundary is also crossed so as to include the area outside the negative X-axis direction. In the third operation, the boundary is crossed so as to include the area outside the positive Z-axis direction. In the fourth operation, the boundary is crossed so as to include the area outside the negative X-axis direction. Here, since the number of crossings in the negative X-axis direction exceeds the threshold "2", the area adjustment unit 2 moves the spatial area in the negative X-axis direction by an adjustment distance of "200 mm". The position vector of the adjusted spatial region moves to (-200, -100, -1200). Here, the spatial region is moved, but the spatial length (800, 400, 500) remains unchanged. The region adjustment unit 18 updates the range information and, after moving the spatial region, resets the number of crossings in the negative X-axis direction to "0". Resetting the number of crossings for the axis that is not adjusted (in this case, the positive Z-axis direction) is optional. The notification unit 19 notifies the user of the movement of the spatial region by voice or screen display, for example, "The spatial region for detecting gestures has been moved 200 mm in the negative X-axis direction."

[0033] An example of expanding the spatial domain will be explained with reference to Figures 8 and 9. In Figure 8, the object of manipulation in the spatial domain is a "turret". The initial settings for the range information are a position vector (-300, -100, -800) and a spatial length (400, 400, 400). In the operation information, the gesture "rotate right with two fingers" is linked to the operation content "select the next tool", and the gesture "rotate left with two fingers" is linked to the operation content "select the previous tool". In Figure 8, the count threshold is defined as "2". The distance space is defined as "100 mm". When the number of times the spatial domain is traversed exceeds the threshold (2 times), the domain adjustment unit 18 expands the spatial length of the X and Z axes by "100 mm". The expanded spatial length becomes (500, 400, 500).

[0034] Referring to Figure 9, an example of expanding the spatial domain when the number of times a gesture crosses the boundary exceeds a threshold is explained. In Figure 9, four operations are performed: right rotation with two fingers, left rotation with two fingers, right rotation with two fingers, and right rotation with two fingers. In the first operation, the boundary is crossed so as to include the areas outside the X-axis positive direction and the Z-axis positive direction. In the second operation, the boundary is crossed so as to include the areas outside the X-axis positive direction and the Z-axis positive direction. The third operation is performed near the center of the spatial domain and does not include the areas outside the domain. In the fourth operation, the boundary is crossed so as to include the areas outside the X-axis positive direction and the Z-axis positive direction. Here, the number of crossings in the X-axis positive direction and the Z-axis positive direction exceeds the threshold "2", so the domain adjustment unit 18 expands the spatial domain by the adjustment distance (100 mm) in the X-axis positive direction and the Z-axis positive direction. The domain adjustment unit 18 expands the spatial domain. The spatial length of the adjusted spatial domain becomes (500, 400, 500). Here, only the spatial domain is expanded, and the position vectors (-300, -100, -800) are not changed. The domain adjustment unit 18 updates the range information and resets the counters in the positive X-axis and positive Z-axis directions to zero. The notification unit 19 notifies the user of the expansion of the spatial domain by displaying an image, such as "The spatial domain for detecting gestures has been expanded by 100 mm in the positive X-axis and positive Z-axis directions."

[0035] Referring to FIG. 10, an example of comparing the number of gestures detected in the vicinity of the boundary (within the threshold dt from the boundary of the spatial region) with the threshold value is shown regardless of the axial direction. In this example, the movement of the spatial region is performed for each axis. Since the initial setting of the range information and the operation information are the same as those in FIG. 4, the description thereof is omitted. In FIG. 10, three operations are performed: swiping down with two fingers in the range within the distance dt from the boundary of the spatial region, swiping right with two fingers in the range within the distance dt from the boundary of the spatial region, and swiping up with two fingers in the range outside the distance dt from the boundary of the spatial region. In the first operation, a gesture is detected in the vicinity of the boundary in the negative X-axis direction from the center of the spatial region. In the second operation, a gesture is detected in the vicinity of the boundary in the positive Z-axis direction from the center of the spatial region. In the third operation, a gesture is detected in the vicinity of the boundary in the negative X-axis direction from the center of the spatial region. Here, since the total number of gestures in the vicinity of the boundary exceeds the threshold value of "2", the region adjustment unit moves the spatial region by an adjustment distance of "200 mm" in the positive X-axis direction and the positive Z-axis direction. The position vector of the spatial region after adjustment moves to (-200, -100, -1200). Here, the spatial region is moved, and the spatial lengths (800, 400, 500) are not changed. After the region adjustment unit 18 updates the range information and moves the spatial region, it resets the number of gestures in the vicinity of the boundary to "0". The notification unit 19 notifies the movement of the spatial region, such as "The spatial region where the gesture is detected has been moved 200 mm in the negative X-axis direction and the positive Z-axis direction" by voice or image display, etc.

[0036] Referring to Figure 11, an example of detecting gestures performed in the vicinity of a spatial domain (within a threshold dt from the boundary of the spatial domain) will be explained. The initial settings and operation information for the range information are the same as in Figure 5, so the explanation will be omitted. In the example in Figure 11, the number of times a gesture is performed in the vicinity of the boundary is counted for each axis and direction. In Figure 11, four operations are performed: swiping up with two fingers within a distance dt inside the left boundary of the spatial domain, swiping down with two fingers within a distance dt inside the left boundary of the spatial domain, swiping left with two fingers within a distance dt inside the top boundary of the spatial domain, and swiping up with two fingers outside the left boundary of the spatial domain. In the first operation, the gesture is detected in the vicinity of the boundary in the negative X-axis direction from the center of the spatial domain. In the second operation, the gesture is detected in the vicinity of the boundary in the negative X-axis direction from the center of the spatial domain. In the third operation, the gesture is detected in the vicinity of the boundary in the positive Z-axis direction from the center of the spatial domain. In the fourth operation, gestures are detected near the boundary in the negative X-axis direction from the center of the spatial domain. Here, since the total number of gestures near the boundary exceeds the threshold "2", the spatial domain is moved by an adjustment distance (200 mm) in the negative X-axis direction. The position vector of the adjusted spatial domain moves to (-200, -100, -1200). Here, the spatial domain is moved, but the spatial length (800, 400, 500) is not changed. The domain adjustment unit 18 updates the range information, moves the spatial domain, and resets the number of gestures near the boundary in the negative X-axis direction to "0". Resetting the axis that is not adjusted (in this case, the positive Z-axis direction) is optional. The notification unit 19 notifies the operator of the adjustment of the spatial domain by voice or screen display, for example, "The spatial domain for detecting gestures has been moved 200 mm in the negative X-axis direction".

[0037] The region adjustment unit 18 checks whether the moved or expanded spatial region overlaps with other spatial regions. If there is an overlap, the region adjustment unit 18 disables the movement or expansion of the spatial region. Alternatively, the overlap with other spatial regions may be resolved by adjusting the method and length of movement of the spatial region, the axis and length of expansion, etc.

[0038] The notification unit 19 may confirm with the operator whether the execution of the adjustment of the spatial region is possible. For example, in the example of FIG. 12, the notification unit 19 asks whether the execution of the adjustment is possible, such as "A gesture has been made outside the spatial region. Do you want to adjust the spatial region?" In response to the notification, the operator answers using a gesture. In the example of FIG. 12, the operator presents a "peace" sign toward the imaging device. When receiving the permission of the operator, the region adjustment unit 18 adjusts the spatial region. In the example of FIG. 12, the spatial region is moved 200 mm in the negative X-axis direction and the positive Z-axis direction. The notification unit 19 notifies the operator that the adjustment has been made. In the example of FIG. 1, it notifies that "The spatial region for detecting the gesture has been moved 200 mm in the negative X-axis direction and the positive Z-axis direction."

[0039] As described above, the control device 100 of the present embodiment and the industrial machine 200 including the control device 100 define a spatial region for an operation target of the industrial machine 200 and receive an operation input by a gesture. The control device 100 detects that a gesture has been performed in the vicinity of the spatial region, and when a predetermined condition is satisfied, adjusts the spatial region so that the gesture is performed at a position closer to the center of the spatial region. Thereby, the detection accuracy of non-contact operations such as gestures can be improved.

[0040] The operator of the industrial machine 200 needs to wear safety equipment such as a helmet, goggles, and gloves. Also, cutting oil and chips may adhere to the gloves. By operating the industrial machine 200 with a gesture, the industrial machine 200 can be operated without wearing an AR terminal such as smart glasses and without contacting an operation panel or a tablet terminal.

[0041] Hereinafter, the hardware configuration of the control device 100 to which the present disclosure is applied will be described. FIG. 13 is a hardware configuration diagram of the control device 100. As shown in FIG. 13, the control device 100 includes a CPU 111 that controls the entire control device 100, a ROM 112 that records programs and data, and a RAM 113 for temporarily expanding data. The CPU 111 reads out the system program recorded in the ROM 112 via a bus.

[0042] The non-volatile memory 114 is backed up, for example, by a battery (not shown), so that its stored state is maintained even when the power to the control device 100 is turned off. The non-volatile memory 114 stores various data, such as programs read from external devices 120 via interfaces 115, 118, and 119, and operation inputs input via input device 30. The non-volatile memory 114 may also store programs and data for executing the control device 100 of this embodiment.

[0043] Interface 115 is an interface for connecting the control device 100 to an external device 120 such as an adapter. Programs and various parameters are read from the external device 120. Interface 118 is an interface for connecting the control device 100 to a display device 40 such as a liquid crystal display. The display device 40 displays data read into memory, data obtained as a result of executing programs, etc. Interface 119 is an interface for connecting the control device 100 to an input device 30 such as a keyboard or pointing device. The input device 30 passes commands, data, etc. based on operator operations to the CPU 111 via interface 119.

[0044] While embodiments of this disclosure have been described in detail above, this disclosure is not limited to the individual embodiments described above. These embodiments can be added, replaced, modified, partially deleted, etc., in any way that does not depart from the spirit of the invention or from the idea and intent of this disclosure derived from the claims and their equivalents. For example, the order of operations and processes in the embodiments described above are shown as examples only and are not limited thereto. The same applies when numerical values ​​or mathematical formulas are used in the description of the embodiments described above.

[0045] The following are annotations relating to embodiments of the present disclosure. (Annotation 1) A control device (100) according to one aspect of the present disclosure includes: an information acquisition unit (11) that acquires range information defining the range of a spatial area virtually provided near an industrial machine, and operation information defining the content of gesture operations to be received in the spatial area; a three-dimensional image acquisition unit (21) that acquires a three-dimensional image from an imaging device; a boundary operation detection unit (17) that detects a gesture from the three-dimensional image and further detects that the gesture was performed near the boundary of the spatial area; a range adjustment unit (18) that adjusts the spatial area so that the gesture is performed at a position close to the center of the spatial area when predetermined conditions are met; a gesture determination unit (13) that determines that the gesture is performed in the spatial area and that the gesture is defined in the operation information; an operation determination unit (14) that determines the content of an operation corresponding to the gesture; and an operation application unit (15) that applies the content of the operation to the industrial machine. (Note 2) The area adjustment unit (18) moves or expands the position of the spatial area by a predetermined distance so that the gesture is performed at a position close to the center of the spatial area. (Note 3) If the area adjustment unit (18) determines that the movement or expansion of the spatial area will result in overlap with other spatial areas, it will invalidate the movement or expansion. (Note 4) The boundary motion detection unit (17) detects that the gesture was performed near the boundary of the spatial area if the gesture is performed across the boundary of the spatial area. (Note 5) The boundary motion detection unit (17) detects that the gesture was performed across the boundary of the spatial area in each direction. (Note 6) The boundary motion detection unit (17) detects that the gesture was performed within a predetermined distance from the boundary of the spatial area. (Note 7) The boundary motion detection unit (17) counts the number of times the gesture is performed near the boundary of the spatial area, and the area adjustment unit (18) adjusts the spatial area so that the gesture is performed in a position closer to the center of the spatial area when the count exceeds a predetermined number. (Note 8) The boundary motion detection unit (17) resets the count when the area adjustment unit (18) moves the position of the spatial area or expands the spatial area.(Note 9) An industrial machine (200) according to one aspect of the present disclosure comprises any one of the control devices described in claims 1 to 8. (Note 10) A computer-readable storage medium (112, 113, 114) according to one aspect of the present disclosure stores in one or more processors (111) of the present disclosure range information defining the range of a spatial area virtually provided near the industrial machine, and operation information defining the content of gesture operations accepted in the spatial area, a three-dimensional image from an imaging device, a gesture detected from the three-dimensional image, and further detection that the gesture was performed near the boundary of the spatial area, and if predetermined conditions are met, the spatial area is adjusted so that the gesture is performed near the center of the spatial area, it is determined that the gesture was performed in the spatial area and that the gesture is defined in the operation information, it is determined the content of the operation corresponding to the gesture, and an instruction to perform a process to apply the content of the operation to the industrial machine.

[0046] 100 Control device 200 Industrial machine 11 Information acquisition unit 12 Detection unit 13 Gesture judgment unit 14 Operation judgment unit 15 Operation application unit 16 Drive control unit 17 Boundary motion detection unit 18 Area adjustment unit 19 Notification unit 20 Display unit 21 3D image acquisition unit 30 Input device 40 Display device 111 CPU 112 ROM 113 RAM 114 Non-volatile memory 115 Interface 118 Interface 119 Interface 120 External device 210 Drive unit

Claims

1. A control device comprising: an information acquisition unit that acquires range information defining the range of a spatial area virtually provided near an industrial machine, and operation information defining the content of gesture operations accepted in the spatial area; a three-dimensional image acquisition unit that acquires a three-dimensional image from an imaging device; a boundary action detection unit that detects a gesture from the three-dimensional image and further detects that the gesture was performed near the boundary of the spatial area; a range adjustment unit that adjusts the spatial area so that the gesture is performed at a position close to the center of the spatial area when predetermined conditions are met; a gesture determination unit that determines that the gesture was performed in the spatial area and that the gesture is defined in the operation information; an operation determination unit that determines the content of the operation corresponding to the gesture; and an operation application unit that applies the content of the operation to the industrial machine.

2. The control device according to claim 1, wherein the area adjustment unit moves or expands the position of the spatial area by a predetermined distance so that the gesture is performed at a position close to the center of the spatial area.

3. The control device according to claim 2, wherein the region adjustment unit determines that the movement or expansion of a spatial region would result in overlap with another spatial region, and disables the movement or expansion.

4. The control device according to claim 1, wherein the boundary motion detection unit detects that the gesture was performed near the boundary of the spatial region when the gesture is performed across the boundary of the spatial region.

5. The control device according to claim 1, wherein the boundary motion detection unit detects, in each direction, that the gesture was performed across the boundary of the spatial region.

6. The control device according to claim 1, wherein the boundary motion detection unit detects that the gesture was performed within a predetermined distance from the boundary of the spatial area.

7. The control device according to claim 1, wherein the boundary motion detection unit counts the number of times the gesture is performed near the boundary of the spatial region, and the region adjustment unit adjusts the spatial region so that the gesture is performed at a position closer to the center of the spatial region when the count exceeds a predetermined number of times.

8. The control device according to claim 7, wherein the boundary motion detection unit resets the count when the area adjustment unit moves the position of the spatial area or expands the spatial area.

9. An industrial machine equipped with any one of the control devices described in claims 1 to 8.

10. A computer-readable storage medium that stores instructions for executing a process which involves one or more processors acquiring range information defining the extent of a spatial region virtually provided near an industrial machine, and operation information defining the content of gesture operations accepted in the spatial region; acquiring a three-dimensional image from an imaging device; detecting a gesture from the three-dimensional image; further detecting that the gesture was performed near the boundary of the spatial region; adjusting the spatial region so that the gesture is performed near the center of the spatial region if predetermined conditions are met; determining that the gesture was performed in the spatial region and that the gesture is defined in the operation information; determining the content of the operation corresponding to the gesture; and applying the content of the operation to the industrial machine.

Images