Control device, industrial machine, and computer-readable storage medium

Abstract

A control device according to the present invention: acquires identification information of an operator of an industrial machine; acquires range information that defines a range of a space region which is virtually provided in the vicinity of the industrial machine and operation information that defines the operation content of a gesture which is received in the space region; selects the range information that is associated with the identification information of the operator; acquires a three-dimensional image from an imaging device; detects a gesture from the three-dimensional image; and determines that the gesture is performed in the range of the space region associated with the identification information of the operator and that the gesture is defined by the operation information.

Description

Control device, industrial machine, and computer-readable storage medium

[0001] The present disclosure relates to a control device, an industrial machine, and a computer-readable storage medium.

[0002] As operations of industrial machines such as machine tools and robots, there are remote operations by tablet terminals, operations by extended reality technology (AR) such as smart glasses, etc. For example, Patent Document 1.

[0003] Operators of industrial machines often wear safety equipment such as helmets, goggles, and gloves. Cutting oil and chips may adhere to the gloves. The cutting oil and chips adhering to the gloves may adhere to the tablet terminal. Also, it is difficult to use smart glasses when wearing a helmet or goggles, and it is not safe to remove the safety equipment and wear smart glasses.

[0004] Japanese Patent Application Laid-Open No. 2020-067956

[0005] Conventionally, there is also a technology for operating an industrial machine non-contact using gestures. However, since the physical characteristics (height, build, etc.) and movement characteristics (dominant hand, habits, etc.) of the operator are different, there is a possibility that gestures may not be detected or are difficult to detect due to these characteristics.

[0006] It is desired to improve the detection accuracy of non-contact operations of industrial machines.

[0007] The control device according to this disclosure includes: an identification information acquisition unit that acquires identification information of an operator of an industrial machine; an information acquisition unit that acquires range information that defines the range of a spatial area virtually provided near the industrial machine, and operation information that defines the content of gesture operations to be received in the spatial area; a selection unit that selects range information linked to the operator's identification information; a three-dimensional image acquisition unit that acquires a three-dimensional image from an imaging device; a detection unit that detects a gesture from the three-dimensional image; a gesture determination unit that determines that the gesture is performed within the range of the spatial area linked to the operator's identification information and that the gesture is defined in the operation information; an operation determination unit that determines the content of an operation corresponding to the gesture; and an operation application unit that applies the content of an operation to the industrial machine.

[0008] This is a block diagram of the control device. This is a perspective view of the industrial machine to explain the spatial domain. This is a diagram explaining spatial domain information. This is a diagram showing the differences in spatial domains due to physical characteristics. This is a diagram showing the differences in spatial domains due to operational characteristics. This is a diagram explaining the notification of the definition of range information to the operator. This is a flowchart explaining the operation of the control device and the industrial machine. This is a hardware configuration diagram of the control device.

[0009] Embodiments of this disclosure will be described below with reference to the drawings. In the following description, components having the same or similar functions will be denoted by the same reference numerals. Duplication of these components may be omitted.

[0010] In this application, "based on XX" means "based on at least XX," and includes cases where it is based on another element in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on something that has been calculated or processed. "XX" is any element (for example, any information).

[0011] As a prerequisite, the relationship between the control device 100, the industrial machine 200, and the imaging device will be explained. The control device 100 is a numerical control device that controls machine tools, a robot controller that controls robots, etc. The control device 100 acquires a three-dimensional image from the imaging device. The imaging device is a ToF camera, a stereo camera, etc. A ToF camera measures the distance to an object using reflected light. A stereo camera measures the distance to an object using parallax. The image captured by the ToF camera is called a depth image. Images captured by ToF cameras and stereo cameras cannot be used directly for gesture detection. In order to recognize an object, it is necessary to derive three-dimensional spatial coordinates that indicate the position of the object. In this embodiment, the three-dimensional spatial information used for gesture detection is called a three-dimensional image. The type of data acquired from the imaging device is not particularly limited. Other means may be used as long as three-dimensional spatial information can ultimately be obtained.

[0012] A virtual spatial area is defined in the vicinity of the control device 100. This spatial area is an area for detecting gestures to operate the industrial machine 200. The spatial area is associated with the objects of operation of the industrial machine 200. Examples of objects of operation include feed axes, operation screens, turrets, and control panels. The operator performs gestures, such as moving their hands, while looking at the objects of operation. The type of object of operation is not particularly limited.

[0013] Each spatial domain has spatial domain information defined within it. This spatial domain information defines the domain's boundaries, the gestures associated with that domain, and the operations associated with those gestures.

[0014] The spatial domain range information is linked to the operator's identification information. Operators have physical characteristics such as height and build, as well as operational characteristics such as their dominant hand when performing gestures and their habits. By defining the spatial domain range for each operator, the accuracy of gesture detection can be improved.

[0015] Figure 1 is a block diagram of the control device 100. The control device 100 includes an identification information acquisition unit 10, an information acquisition unit 11, a selection unit 12, a detection unit 13, a gesture judgment unit 14, an operation judgment unit 15, an operation application unit 16, a drive control unit 17, a notification unit 18, a display unit 19, and a 3D image acquisition unit 20.

[0016] The identification information acquisition unit 10 acquires identification information of the operator of the industrial machine. Identification information includes biometric information, memorized information, and owned equipment. Biometric information includes fingerprints, iris scans, vein scans, facial scans, and voiceprints. Memorized information is information stored by the operator. Memorized information includes passwords, passphrases, and PINs (passcodes). Owned equipment includes IC cards (smart cards, chip cards), USB dongles, and OTP (One Time Password tokens). The operator's gestures may also be used as identification information. Furthermore, name tags and company badges captured when the operator is imaged by an imaging device may also be used as identification information. The types of identification information are not particularly limited.

[0017] The information acquisition unit 11 acquires information related to the spatial domain (referred to as spatial domain information). The spatial domain information includes range information indicating the extent (location and size) of the spatial domain, and operation information relating to gestures accepted in the spatial domain and the corresponding operation content. The spatial domain information may be acquired from within the control device, or from an external cloud or network storage. The storage location of the spatial domain information is not limited.

[0018] The selection unit 12 selects range information from the spatial domain information that is linked to the operator's identification information. This range information indicates the individual spatial domain of the operator.

[0019] Figure 2 shows an example of a spatial area provided in a machining center. A machining center is an industrial machine 200 in which a control device (numerical control device) 100 and a machine tool are integrated. In this embodiment, the spatial area of ​​a machining center is described, but a spatial area can also be defined for machine tools other than machining centers, robots, etc.

[0020] In Figure 2, the object of operation in spatial region 1 is the turret. The object of operation in spatial region 2 is the operation screen of the control device 100. The object of operation in spatial region 3 is the feed axis of the machine tool (industrial machine 200).

[0021] An example of spatial domain information will be explained with reference to Figure 3. Spatial domain information includes range information and operation information. Range information is linked to operator identification information. In Figure 3, range information is assigned to operator identification information (e.g., "0001", "0002", "0003"). Range information represents the position and size of the spatial domain. In the range information of Figure 3, the position and size of the spatial domain are represented by a position vector and the spatial domain. The position vector indicates the position of the spatial domain. The position vector indicates the position of the spatial domain relative to a certain point (reference position). The spatial length represents the size of the spatial domain. The spatial length represents the size of the spatial domain as the length in the X, Y, and Z axis directions. In this embodiment, the position of the imaging device is used as the reference position.

[0022] The range information of the spatial region 1 associated with operator "0001" is represented by a position vector (-300, -100, -800) and a spatial length (400, 400, 400). In this case, the spatial region is a cube, but the shape of the spatial region is not limited to this. Similarly, the range information of the spatial region 1 for operator "0002" is represented by a position vector (-300, -100, -700) and a spatial length (400, 400, 400), and the range information of the spatial region 1 for operator "0003" is represented by a position vector (-400, -100, -600) and a spatial length (500, 500, 500). In this embodiment, the position of the imaging device is used as the reference position.

[0023] The operation information defines a gesture for the object being operated on, and the operation content associated with that gesture. In Figure 3, the operation information for the object being operated on, the "turret," defines "select next tool" as the operation content for the gesture "rotate right with two fingers," and "select previous tool" as the operation content for the gesture "rotate left with two fingers."

[0024] The 3D image acquisition unit 20 acquires data that will form the basis of a 3D image, or a 3D image, from the imaging device. The imaging device can be a ToF camera, a stereo camera, etc., but is not particularly limited in type. Other means may be used as long as 3D spatial information can ultimately be obtained.

[0025] The detection unit 13 detects gestures in a spatial region from a three-dimensional image. The gesture detection range does not necessarily have to be limited to the interior of the spatial region; it may also include the periphery of the spatial region.

[0026] The gesture determination unit 14 determines the gesture based on the three-dimensional image detected in the spatial domain and the spatial domain information acquired by the information acquisition unit 11.

[0027] The following describes an example where a turret is the target of operation. The operator operates the turret using gestures while looking at it. The spatial area when operating the turret is spatial area 1. The range information of spatial area 1 is defined for each operator. The selection of range information is performed by the selection unit 12. The spatial area suitable for each operator is determined by the operator's physical and operational characteristics. An example of a physical characteristic is height. As shown in Figure 4, the spatial area of ​​a tall person tends to be higher than that of a short person. An example of an operational characteristic is handedness. As shown in Figure 5, the spatial area of ​​a right-handed person is to the right of the turret, and the spatial area of ​​a left-handed person is to the left of the turret.

[0028] Operator-specific range information shall be registered before operating the industrial machine 200. Range information may be entered numerically, or the operator may input a gesture and the area around that gesture may be used as the operator's range information. When the operator is to actually perform a gesture, the control device 100 may guide the operator by saying something like, "Please make a gesture of rotating the dial while looking at the turret."

[0029] The notification unit 18 notifies the operator that new range information has been defined. In Figure 6, when the operator with identification information "0001" registers spatial area 1, the operator is notified with a message such as, "We have set up spatial area for 0001. This is spatial area for recognizing gestures to operate the turret. The range is the position vector (-300, -100, -800), and the spatial length is (400, 400, 400)." Similarly, when the operator with identification information "0002" registers spatial area 1, the operator is notified with a message such as, "We have set up spatial area for 0002. This is spatial area 1 for recognizing gestures to operate the turret. The range is the position vector (-300, -100, -700), and the spatial length is (400, 400, 400)." The means of notification are not particularly limited and may include voice, screen display, etc.

[0030] The operation determination unit 15 determines the operation content from the gesture based on the operation information. As shown in Figure 3, the operation information is defined as a gesture and the operation content. The operation determination unit 15 determines the operation content corresponding to the detected gesture.

[0031] The operation application unit 16 applies the operation content determined by the operation determination unit 15. Specifically, it outputs control information based on the operation content to the drive control unit 17. The drive control unit 17 drives the industrial machine 200 by controlling the drive unit 21 of the industrial machine 200 according to the gesture operation content.

[0032] Referring to the flowchart in Figure 7, the operation of the control device 100 and the industrial machine 200 including the control device 100 in this embodiment will be explained. As a prerequisite, it is assumed that the spatial area for each operator is registered in advance. In the registration of spatial areas, the identification information of the spatial area (spatial area 1, spatial area 2, spatial area 3, etc.) is linked to the identification information of the operator and the range information of the spatial area.

[0033] The operator inputs their own identification information in order to operate the industrial machine 200. The identification information acquisition unit 10 acquires the operator's identification information (step S1). The selection unit 12 acquires range information linked to the operator's identification information (step S2). The information acquisition unit 11 acquires operation information for the spatial area (step S3).

[0034] The detection unit 13 monitors the gesture input (step S4). The detection unit 13 detects a gesture inside or around any of the spatial regions (step S5). The gesture determination unit 14 determines whether the detected gesture is defined in the operation information of the spatial region where the gesture was detected.

[0035] If the spatial domain operation information includes the detected gesture (step S6; Yes), the operation determination unit 15 determines the operation content associated with the gesture (step S7). If the spatial domain operation information does not include the gesture (step S6; No), the process moves to step S4, and the gesture input is monitored.

[0036] The operation application unit 16 outputs the operation content to the drive control unit 17, and the drive control unit 17 controls the industrial machine 200 according to the operation content (step S8).

[0037] As described above, the control device 100 and the industrial machine 200 equipped with the control device 100 of this embodiment define a spatial area corresponding to the object to be operated by the industrial machine 200, and when a gesture is detected in the spatial area, the control device determines the operation content corresponding to that gesture and controls the industrial machine 200.

[0038] The spatial domain is defined for each operator. Each operator has their own physical and operational characteristics. By defining the spatial domain for each operator, discrepancies in the spatial domain caused by physical and operational characteristics can be eliminated. This improves the detection accuracy of non-contact operations of industrial machinery.

[0039] Industrial machinery operators are required to wear safety equipment such as helmets, goggles, and gloves. Gloves may become contaminated with cutting oil and chips. By operating industrial machinery with gestures, it is possible to operate the machinery without wearing AR devices such as smart glasses, and without touching the control panel or tablet device.

[0040] Furthermore, the accuracy of gesture detection is also improved. Specifically, the imaging device captures images of all objects within the field of view. Some of the captured objects are unnecessary for gesture recognition. By defining a spatial region, unnecessary objects can be excluded from the 3D image, and only the target object can be extracted. The hardware configuration of the control device 100 to which this disclosure is applied will be described below. Figure 8 is a hardware configuration diagram of the control device 100. As shown in Figure 8, the control device 100 includes a CPU 111 that controls the control device 100 overall, a ROM 112 that records programs and data, and a RAM 113 for temporarily expanding data. The CPU 111 reads the system program recorded in the ROM 112 via a bus.

[0041] The non-volatile memory 114 is backed up, for example, by a battery (not shown), so that its stored state is maintained even when the power to the control device 100 is turned off. The non-volatile memory 114 stores various data, such as programs read from external devices 120 via interfaces 115, 118, and 119, and operation inputs input via input device 40. The non-volatile memory 114 may also store programs and data for executing the control device 100 of this embodiment.

[0042] Interface 115 is an interface for connecting the control device 100 to an external device 120 such as an adapter. Programs and various parameters are read from the external device 120. Interface 118 is an interface for connecting the control device 100 to a display device 30 such as a liquid crystal display. The display device 30 displays data read into memory, data obtained as a result of executing programs, etc. Interface 119 is an interface for connecting the control device 100 to an input device 40 such as a keyboard or pointing device. The input device 40 passes commands, data, etc. based on operator operations to the CPU 111 via interface 119.

[0043] While embodiments of this disclosure have been described in detail above, this disclosure is not limited to the individual embodiments described above. These embodiments can be added, replaced, modified, partially deleted, etc., in any way that does not depart from the spirit of the invention or from the idea and intent of this disclosure derived from the claims and their equivalents. For example, the order of operations and processes in the embodiments described above are shown as examples only and are not limited thereto. The same applies when numerical values ​​or mathematical formulas are used in the description of the embodiments described above.

[0044] The following are additional notes regarding embodiments of the present disclosure. (Addendum 1) A control device (100) according to one aspect of the present disclosure includes: an identification information acquisition unit (10) that acquires identification information of an operator of an industrial machine (200); an information acquisition unit (11) that acquires range information defining the range of a spatial area virtually provided in the vicinity of the industrial machine (200), and operation information defining the content of gesture operations to be received in the spatial area; a selection unit (12) that selects range information linked to the operator's identification information; a three-dimensional image acquisition unit (20) that acquires a three-dimensional image from an imaging device; a detection unit (13) that detects a gesture from the three-dimensional image; a gesture determination unit (14) that determines that the gesture is performed within the range of the spatial area linked to the operator's identification information and that the gesture is defined in the operation information; an operation determination unit (15) that determines the content of an operation corresponding to the gesture; and an operation application unit (16) that applies the content of the operation to the industrial machine. (Note 2) The range information includes a position vector from a predetermined reference position, and the position vector indicates the position of the spatial region. (Note 3) The range information includes a spatial length in the three-dimensional direction, and the spatial length indicates the size of the spatial region. (Note 4) The identification information is at least one of the operator's stored information, possessed equipment, and biometric information. (Note 5) The identification information is a gesture. (Note 6) The spatial region is associated with the object to be operated. (Note 7) A notification unit is provided to notify the operator that new range information has been defined. (Note 8) The industrial machine (200) includes any one of the control devices (100) described in claims 1 to 7.(Appended Note 9) A computer-readable storage medium (112, 113, 114) causes one or more processors (111) to acquire identification information of an operator of an industrial machine (200), acquire range information defining a range of a space region virtually provided in the vicinity of the industrial machine (200), and operation information defining the operation content of a gesture received in the space region, select the range information associated with the identification information of the operator, acquire a three-dimensional image from an imaging device, detect a gesture from the three-dimensional image, determine that the gesture is performed within the range of the space region associated with the identification information of the operator and that the gesture is defined by the operation information, determine the operation content corresponding to the gesture, and store an instruction for causing the industrial machine (200) to execute a process for applying the operation content.

[0045] 100 Control device 200 Industrial machine 10 Identification information acquisition unit 11 Information acquisition unit 12 Selection unit 13 Detection unit 14 Gesture determination unit 15 Operation determination unit 16 Operation application unit 17 Drive control unit 18 Notification unit 19 Display unit 20 Three-dimensional image acquisition unit 21 Drive unit 30 Display device 40 Input device 111 CPU 112 ROM 113 RAM 114 Non-volatile memory 115 Interface 118 Interface 119 Interface 120 External device

Claims

1. A control device comprising: an identification information acquisition unit for acquiring identification information of an operator of an industrial machine; an information acquisition unit for acquiring range information that defines the range of a spatial area virtually provided near the industrial machine, and operation information that defines the content of gesture operations accepted in the spatial area; a selection unit for selecting range information linked to the operator's identification information; a three-dimensional image acquisition unit for acquiring a three-dimensional image from an imaging device; a detection unit for detecting a gesture from the three-dimensional image; a gesture determination unit for determining that the gesture is performed within the range of the spatial area linked to the operator's identification information and that the gesture is defined in the operation information; an operation determination unit for determining the content of an operation corresponding to the gesture; and an operation application unit for applying the content of an operation to the industrial machine.

2. The control device according to claim 1, wherein the range information includes a position vector from a predetermined reference position, and the position vector indicates the position of the spatial region.

3. The control device according to claim 1, wherein the range information includes the spatial length in a three-dimensional direction, and the spatial length indicates the size of the spatial region.

4. The control device according to claim 1, wherein the identification information is at least one of the operator's stored information, possessed equipment, and biometric information.

5. The control device according to claim 1, wherein the identification information is a gesture.

6. The control device according to claim 1, wherein the spatial region is associated with an object to be operated.

7. The control device according to claim 1, further comprising a notification unit that notifies the operator that new range information has been defined.

8. An industrial machine comprising any one of the control devices described in claims 1 to 7.

9. A computer-readable storage medium that stores instructions for one or more processors to execute a process that includes: acquiring identification information of an industrial machine operator; acquiring range information defining the range of a virtually provided spatial area near the industrial machine, and operation information defining the content of gesture operations accepted in the spatial area; selecting range information linked to the operator's identification information; acquiring a three-dimensional image from an imaging device; detecting a gesture from the three-dimensional image; determining that the gesture is performed within the range of the spatial area linked to the operator's identification information and that the gesture is defined in the operation information; determining the content of the operation corresponding to the gesture; and applying the content of the operation to the industrial machine.

Images