Control device, industrial machine, and computer-readable storage medium

Abstract

This control device acquires range information indicating a range of a space region virtually defined in the vicinity of an industrial machine and operation information indicating contents of operations for gestures received in the space region, acquires a three-dimensional image from an imaging device, detects a gesture from the three-dimensional image, determines that the gesture is performed in the space region and that the gesture is defined by the operation information, determines the contents of an operation corresponding to the gesture, and applies the contents of the operation to the industrial machine.

Description

Storage media readable by control devices, industrial machinery, and computers.

[0001] This disclosure relates to a storage medium readable by a control device, industrial machinery, and computer.

[0002] For operating industrial machinery such as machine tools and robots, methods include remote control using tablet devices and operation using augmented reality (AR) technologies such as smart glasses. For example, see Patent Document 1.

[0003] Industrial machinery operators often wear safety equipment such as helmets, goggles, and gloves. Cutting oil and chips can adhere to gloves. These cutting oils and chips on gloves may then adhere to tablet devices. Furthermore, using smart glasses while wearing a helmet and goggles is difficult, and removing safety equipment to wear smart glasses is not advisable from a safety standpoint.

[0004] Japanese Patent Publication No. 2020-067956

[0005] Traditionally, technologies exist for operating industrial machinery without physical contact using gestures. However, improving the accuracy of gesture detection and the ease of operation are necessary to increase the reliability of non-contact operation.

[0006] There is a growing demand for non-contact operation of industrial machinery.

[0007] The control device according to this disclosure includes: an information acquisition unit that acquires range information defining the range of a spatial area virtually provided in the vicinity of an industrial machine, and operation information defining the content of gesture operations received in the spatial area; a three-dimensional image acquisition unit that acquires a three-dimensional image from an imaging device; a detection unit that detects a gesture from the three-dimensional image; a gesture determination unit that determines that the gesture is performed in the spatial area and that the gesture is defined in the operation information; an operation determination unit that determines the content of an operation corresponding to the gesture; and an operation application unit that applies the content of an operation to the industrial machine.

[0008] This is a block diagram of the control device. This is a perspective view of an industrial machine to explain the spatial domain. This is a diagram explaining spatial domain information. This is a diagram specifically explaining gesture detection using the spatial domain. This is a diagram explaining an example of operating the feed axis. This is a diagram of operation information 1. This is a diagram of operation information 2. This is a diagram of operation information 3. This is a diagram explaining an example of switching operation content depending on the state. This is a flowchart explaining the operation of the control device and the industrial machine. This is a hardware configuration diagram of the control device.

[0009] Embodiments of this disclosure will be described below with reference to the drawings. In the following description, components having the same or similar functions will be denoted by the same reference numerals. Duplication of these components may be omitted.

[0010] In this application, "based on XX" means "based on at least XX," and includes cases where it is based on another element in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on something that has been calculated or processed. "XX" is any element (for example, any information).

[0011] As a prerequisite, the relationship between the control device 100, the industrial machine 200, and the imaging device will be explained. The control device 100 is a numerical control device that controls machine tools, a robot controller that controls robots, etc. The control device 100 acquires a three-dimensional image from the imaging device. The imaging device is a ToF camera, a stereo camera, etc. A ToF camera measures the distance to an object using reflected light. A stereo camera measures the distance to an object using parallax. The image captured by the ToF camera is called a depth image. Images captured by ToF cameras and stereo cameras cannot be used directly for gesture detection. In order to recognize an object, it is necessary to derive three-dimensional spatial coordinates that indicate the position of the object. In this embodiment, the three-dimensional spatial information used for gesture detection is called a three-dimensional image. The type of data acquired from the imaging device is not particularly limited. Other means may be used as long as three-dimensional spatial information can ultimately be obtained.

[0012] A virtual spatial area is defined in the vicinity of the control device 100. This spatial area is an area for detecting gestures to operate the industrial machine 200. The spatial area is associated with the objects of operation of the industrial machine 200. Examples of objects of operation include feed axes, operation screens, turrets, and control panels. The operator performs gestures, such as moving their hands, while looking at the object of operation. This allows for contactless operation of the object of operation. The type of object of operation is not particularly limited.

[0013] Each spatial domain has spatial domain information defined within it. This spatial domain information defines the range of the spatial domain, the gestures associated with that spatial domain, and the operations associated with those gestures.

[0014] Figure 1 is a block diagram of the control device 100. The control device 100 includes an information acquisition unit 11, a detection unit 12, a state acquisition unit 13, a gesture determination unit 14, an operation determination unit 15, an operation application unit 16, a display unit 17, a drive control unit 18, and a 3D image acquisition unit 19.

[0015] Spatial domain information may be obtained from within the control device, or from an external cloud or network storage. The storage location of the spatial domain information is not limited.

[0016] The information acquisition unit 11 acquires information related to the spatial domain (referred to as spatial domain information). The spatial domain information includes range information indicating the extent (location and size) of the spatial domain, and operation information relating to gestures accepted in the spatial domain and the corresponding operation content. The spatial domain information may be acquired from within the control device, or from an external cloud or network storage. The storage location of the spatial domain information is not limited.

[0017] FIG. 2 shows an example of a spatial region provided in a machining center. The machining center is an industrial machine 200 in which a control device (numerical control device) 100 and a machine tool are integrated. In FIG. 2, the operation target of the spatial region 1 is the turret. The operation target of the spatial region 2 is the operation screen of the control device 100. The operation target of the spatial region 3 is the feed axis of the machine tool (industrial machine 200). Note that, in the present embodiment, the spatial region of the machining center will be described, but a spatial region can also be defined for machine tools, robots, etc. other than the machining center.

[0018] The spatial region information includes the range information of the spatial region and the operation information. In the range information of FIG. 3, the position and size of the spatial region are expressed by a position vector and a spatial length. The position vector indicates the position of the spatial region. The position vector indicates the position of the spatial region with respect to a certain point (reference position). The spatial length represents the size of the spatial region. The spatial length expresses the size of the spatial region by the lengths in the X-axis, Y-axis, and Z-axis directions. Note that, in the present embodiment, the position of the imaging device is used as the reference position.

[0019] The range of the spatial region 1 is expressed by the position vector (X １ , Y １ , Z １ ) and the spatial length (X １ｌ , Y １ｌ , Z １ｌ ). Similarly, the range information of the spatial region 2 is the position vector (X ２ , Y ２ , Z ２ ), the spatial length (X ２ｌ , Y ２ｌ , Z ２ｌ ), the range information of the spatial region 3 is the position vector (X ３ , Y ３ , Z ３ ), and the spatial length (X ３ｌ , Y ３ｌ , Z ３ｌ ). Note that, in the example of FIG. 3, the spatial region is a rectangular parallelepiped, but the shape of the spatial region is not limited to this.

[0020] In the spatial domain operation information, gestures for the target being operated on and the operations associated with those gestures are defined. In Figure 3, operation information 1 for the target "turret" in spatial domain 1 is defined as "gesture 11" and "turret operation 1", "gesture 12" and "turret operation 2", etc. Operation information 2 for the target "display screen" in spatial domain 2 is defined as "gesture 21" and "screen operation 1", "gesture 22" and "screen operation 2", etc. Operation information 3 for the target "feed axis" in spatial domain 3 is defined as "gesture 31" and "feed axis operation 1", "gesture 32" and "feed axis operation 2", etc.

[0021] In spatial domain 3, the operation content corresponding to the gesture switches depending on the state of the industrial machine 200 (for example, mode). The switching of operation content will be described later.

[0022] The 3D image acquisition unit 19 acquires data that will form the basis of a 3D image, or a 3D image, from the imaging device. The imaging device can be a ToF camera, a stereo camera, etc., but is not particularly limited in type. Other means may be used as long as 3D spatial information can ultimately be obtained.

[0023] The detection unit 12 detects gestures from the acquired 3D image. The gesture detection range does not necessarily have to be limited to the interior of the spatial region; it may also include the periphery of the spatial region.

[0024] The gesture determination unit 14 determines the gesture based on the three-dimensional image detected in the spatial domain and the spatial domain information acquired by the information acquisition unit 11. The detection of gestures using the spatial domain will be explained in detail with reference to Figure 4. In Figure 4, the operator operates the turret with gestures while looking at the turret. The operator makes gestures by moving their hand around the spatial domain 1. In spatial domain 1, operation information such as gesture 11 "rotate right with two fingers" and gesture 12 "rotate left with two fingers" are defined. The gesture determination unit 14 determines the input gesture from the operator's hand movements. Note that the gesture determination is based on existing technology, so the explanation will be omitted.

[0025] The gesture determination unit 14 also determines if the detected gesture is not defined in the spatial domain information. For example, the gesture "pinch in" is not defined in operation information 1. The gesture determination unit 14 determines that there is no operation corresponding to the detected hand movement. The control device 100 may notify the user that the gesture is not registered by displaying it on the screen or by other means.

[0026] Referring to Figure 5, an example of operating the feed axis will be explained. In Figure 5, the operator operates the feed axis using gestures while looking at the feed axis inside the machining center. Operation information 3 defines gestures such as gesture 31 "swipe up with two fingers", gesture 32 "swipe down with two fingers", and gesture 33 "swipe right with two fingers" as ways to operate the feed axis.

[0027] The status acquisition unit 13 acquires information regarding the status of the industrial machine 200. The status of the industrial machine 200 includes mode status, operating status, alarm status, etc. Information regarding the status of the industrial machine 200 can be acquired from within the control device 100.

[0028] The operation determination unit 15 determines the operation content from the gesture based on the operation information. Figures 6, 7, and 8 are specific examples of operation information. Figure 6 shows operation information 1 for the target of operation, "turret". In operation information 1 in Figure 6, the operation content "select next tool" is defined for the gesture "rotate right with two fingers", and the operation content "select previous tool" is defined for the gesture "rotate left with two fingers".

[0029] Figure 7 shows operation information 2 for the target of operation, "display screen". In operation information 2 of Figure 7, the operation content "Proceed to next screen" is defined for the gesture "right flick", the operation content "Return to previous screen" for the gesture "left flick", "Zoom in on screen" for the gesture "pinch out", and "Zoom out on screen" for the gesture "pinch in".

[0030] Figure 8 shows the operation information 3 for the target of operation, the "feed axis". Different operations are defined for each mode in the feed axis operation information 3. In the example in Figure 8, three types of operations are defined: "jog mode", "handle mode", and "other than jog mode / handle mode". In jog mode, the following operations are defined: for the gesture "swipe up with two fingers", the operation is "move the tool axis upward at the jog feed speed"; for the gesture "swipe down with two fingers", the operation is "move the tool axis downward at the jog feed speed"; for the gesture "swipe right with two fingers", the operation is "move the table axis to the right at the jog feed speed"; for the gesture "swipe left with two fingers", the operation is "move the table axis to the left at the jog feed speed"; for the gesture "push back with two fingers", the operation is "move the table axis in the back at the jog feed speed"; and for the gesture "pull back with two fingers", the operation is "move the table axis in the front at the jog feed speed". In handle mode, the following actions are defined: "Swipe up with two fingers" moves the tool axis upward at the selected handle magnification; "Swipe down with two fingers" moves the tool axis downward at the selected handle magnification; "Swipe right with two fingers" moves the table axis to the right at the selected handle magnification; "Swipe left with two fingers" moves the table axis to the left at the selected handle magnification; "Push back with two fingers" moves the table axis backward at the selected handle magnification; and "Pull back with two fingers" moves the table axis forward at the selected handle magnification. In modes other than jog mode / handle mode, no actions are defined.

[0031] Referring to Figure 9, an example of switching operations depending on the state will be explained. In Figure 9, the left column shows gestures, and the middle column shows the mode or handle magnification. If the gesture is "swipe down with two fingers" and the mode state is "jog mode", the operation determination unit 15 determines that the operation is "move the tool axis downward at the jog feed speed (Z minus direction)". If the gesture is "swipe right with two fingers", the mode state is "handle mode", and the magnification selection is "×100 (100μ)", the operation determination unit 15 determines that the operation is "move the table axis 100μ to the right with one swipe". Note that this movement is equivalent to the tool moving 100μ in the X minus direction relatively. If the gesture is "push in with two fingers" and the mode state is "memory mode", the operation determination unit 15 determines that "there is no operation to apply". If there are no applicable operations, the operator may be notified of this fact using a screen display or the like. If the gesture is "pinch out" and the mode status is "jog mode", the operation determination unit 15 determines that "there are no applicable operations". If there are no applicable operations, the operator may be notified of this fact using a screen display or the like.

[0032] The operation application unit 16 applies the operation content determined by the operation determination unit 15. Specifically, it outputs control information based on the operation content to the drive control unit 18. The drive control unit 18 drives the industrial machine 200 by controlling the drive unit 21 of the industrial machine 200 according to the gesture operation content.

[0033] Referring to the flowchart in Figure 10, the operation of the control device 100 and the industrial machine 200 including the control device 100 in this embodiment will be explained. When an operator makes a gesture in any spatial area, the detection unit 12 detects the gesture (step S1). The gesture determination unit 14 determines the gesture based on the operation information. If there is no corresponding gesture (step S2; No), the operator is notified of this fact by a screen display or the like (step S3).

[0034] If a corresponding gesture exists (step S2; Yes) and the operation details for each state are not defined (step S4; No), the operation determination unit 15 determines the operation details based on the spatial region and the gesture (step S5). If a corresponding gesture exists (step S2; Yes) and the operation details for each state are defined (step S4; Yes), the operation determination unit 15 determines the operation details based on the spatial region, the gesture, and the state (step S6).

[0035] The operation application unit 16 outputs the operation details to the drive control unit 18, and the drive control unit 18 controls the industrial machine 200 according to the operation details (step S7).

[0036] As described above, the control device 100 of the present embodiment and the industrial machine 200 equipped with the control device 100 define a spatial region around the industrial machine 200. The spatial region is associated with the operation target. When a gesture is detected in the spatial region, the operation details corresponding to the gesture are determined, and the industrial machine 200 is operated.

[0037] The operator of the industrial machine 200 needs to wear safety equipment such as helmets, goggles, and gloves. Also, cutting oil and chips may adhere to the gloves. By operating the industrial machine 200 with gestures, it is possible to operate the industrial machine 200 without wearing an AR terminal such as smart glasses and without contacting the operation target (operation panel, tablet terminal, etc.).

[0038] Also, by setting the spatial region, gestures can be defined for each operation target. Pinch-in, pinch-out, and pressing are gestures suitable for screen operations. Dial rotation, button pressing, etc. are gestures suitable for operating the operation panel. In this way, by defining gestures suitable for each operation target, intuitive operation becomes possible and the operability is improved. Also, since gestures can be defined for each operation target, it is possible to increase the types of gestures.

[0039] Defining a spatial region also improves the detection accuracy of gestures. Specifically, the imaging device images all objects that enter the viewing angle. Among the imaged objects, there are some that are unnecessary for gesture recognition. By defining a spatial region, unnecessary objects can be excluded from the three-dimensional image, and only the target object can be extracted.

[0040] In addition, the operation content corresponding to the state of the industrial machine 200 is defined in the spatial region state. Thereby, it is also possible to switch the operation content according to the state of the industrial machine 200.

[0041] Hereinafter, the hardware configuration of the control device 100 to which the present disclosure is applied will be described. FIG. 11 is a hardware configuration diagram of the control device 100. As shown in FIG. 11, the control device 100 includes a CPU 111 that controls the entire control device 100, a ROM 112 that records programs and data, and a RAM 113 for temporarily expanding data. The CPU 111 reads out the system program recorded in the ROM 112 via a bus.

[0042] The non-volatile memory 114 maintains its storage state even when the power of the control device 100 is turned off, for example, by being backed up by a battery (not shown). The non-volatile memory 114 stores various data such as programs read from an external device 120 via interfaces 115, 118, 119 and operation inputs input via an input device 20. The non-volatile memory 114 may store the program and data for executing the control device 100 of the present embodiment.

[0043] Interface 115 is an interface for connecting the control device 100 to an external device 120 such as an adapter. Programs and various parameters are read from the external device 120. Interface 118 is an interface for connecting the control device 100 to a display device 30 such as a liquid crystal display. The display device 30 displays data read into memory, data obtained as a result of executing programs, etc. Interface 119 is an interface for connecting the control device 100 to an input device 20 such as a keyboard or pointing device. The input device 20 passes commands, data, etc. based on operator operations to the CPU 111 via interface 119.

[0044] While embodiments of this disclosure have been described in detail above, this disclosure is not limited to the individual embodiments described above. These embodiments can be added, replaced, modified, partially deleted, etc., in any way that does not depart from the spirit of the invention or from the idea and intent of this disclosure derived from the claims and their equivalents. For example, the order of operations and processes in the embodiments described above are shown as examples only and are not limited thereto. The same applies when numerical values ​​or mathematical formulas are used in the description of the embodiments described above.

[0045] The following are annotations relating to embodiments of the present disclosure. (Annotation 1) A control device (100) according to one aspect of the present disclosure includes: an information acquisition unit (11) that acquires range information defining the range of a spatial area virtually provided in the vicinity of an industrial machine, and operation information defining the content of a gesture operation to be received in the spatial area; a three-dimensional image acquisition unit (19) that acquires a three-dimensional image from an imaging device; a detection unit (12) that detects a gesture from the three-dimensional image; a gesture determination unit (14) that determines that the gesture is performed in the spatial area and that the gesture is defined in the operation information; an operation determination unit (15) that determines the content of an operation corresponding to the gesture; and an operation application unit (16) that applies the content of the operation to the industrial machine. (Annotation 2) The range information includes a position vector from a predetermined reference position, and the position vector indicates the position of the spatial area. (Annotation 3) The range information includes a spatial length in the three-dimensional direction, and the spatial length indicates the size of the spatial area. (Note 4) The operation information defines different operation contents for each state of the industrial machine. (Note 5) The state of the industrial machine includes at least one of a mode state, an operating state, and an alarm state. (Note 6) The spatial region is associated with the object to be operated. (Note 7) The operation information defines a gesture suitable for the object to be operated. (Note 8) An industrial machine (200) according to one aspect of the present disclosure comprises any one of the control devices described in claims 1 to 7. (Note 9) A computer-readable storage medium (112, 113, 114) according to one aspect of the present disclosure stores in one or more processors (111) an instruction to execute a process which includes: range information defining the range of a spatial area virtually provided in the vicinity of an industrial machine, and operation information defining the content of a gesture operation accepted in the spatial area; acquiring a three-dimensional image from an imaging device; detecting a gesture from the three-dimensional image; determining that the gesture is performed in the spatial area and that the gesture is defined in the operation information; determining the content of an operation corresponding to the gesture; and applying the content of the operation to the industrial machine.

[0046] 100 Control device 200 Industrial machine 11 Information acquisition unit 12 Detection unit 13 State acquisition unit 14 Gesture judgment unit 15 Operation judgment unit 16 Operation application unit 17 Display unit 18 Drive control unit 20 Input device 21 Drive unit 30 Display device 111 CPU 112 ROM 113 RAM 114 Non-volatile memory 115 Interface 118 Interface 119 Interface 120 External device

Claims

1. A control device comprising: an information acquisition unit that acquires range information defining the range of a spatial area virtually provided near an industrial machine, and operation information defining the content of gesture operations accepted in the spatial area; a three-dimensional image acquisition unit that acquires a three-dimensional image from an imaging device; a detection unit that detects a gesture from the three-dimensional image; a gesture determination unit that determines that the gesture is performed in the spatial area and that the gesture is defined in the operation information; an operation determination unit that determines the content of an operation corresponding to the gesture; and an operation application unit that applies the content of an operation to the industrial machine.

2. The control device according to claim 1, wherein the range information includes a position vector from a predetermined reference position, and the position vector indicates the position of the spatial region.

3. The control device according to claim 1, wherein the range information includes the spatial length in a three-dimensional direction, and the spatial length indicates the size of the spatial region.

4. The control device according to claim 1, wherein the operation information has different operation contents defined for each state of the industrial machine.

5. The control device according to claim 4, wherein the state of the industrial machine includes at least one of a mode state, an operating state, and an alarm state.

6. The control device according to claim 1, wherein the spatial region is associated with an object to be operated.

7. The control device according to claim 6, wherein the operation information defines a gesture suitable for the target of the operation.

8. An industrial machine comprising any one of the control devices described in claims 1 to 7.

9. A computer-readable storage medium that stores instructions for one or more processors to execute a process that includes: range information defining the extent of a spatial area virtually provided near an industrial machine, and operation information defining the content of gesture operations accepted in the spatial area; acquiring a three-dimensional image from an imaging device; detecting a gesture from the three-dimensional image; determining that the gesture is performed in the spatial area and that the gesture is defined in the operation information; determining the content of an operation corresponding to the gesture; and applying the content of the operation to the industrial machine.

Images