Operation assistance system

Abstract

Provided is an operation assistance system for assisting operation of an industrial machine by an operator, the operation assistance system comprising a transmission-type display mounted on a head of the operator, and a control device that controls display by the transmission-type display, and the control device having a display unit that displays an image related to an operation of the industrial machine on the transmission-type display, a condition setting unit that sets a display condition for determining that display by the display unit is necessary, a state acquisition unit that acquires at least one state of at least one of the industrial machine and the operator, and a condition determination unit that determines whether the state acquired by the state acquisition unit satisfies the display condition, and causes the display unit to perform display when the display condition is satisfied.

Description

Operation Support System

[0001] The present invention relates to an operation support system.

[0002] Head-mounted terminals including a transmissive display and a camera are commercially available. Supporting operators of industrial machinery using such head-mounted terminals has also been considered. For example, to support the installation of tape feeders that supply carrier tapes holding electronic components to a circuit board in a device that mounts electronic components on a substrate, a system has been proposed in which an identification mark attached to a feeder placement section in an image captured by the camera of the head-mounted terminal is recognized, and information such as the tape feeder to be installed in the feeder placement section and the work procedure is displayed on a transmissive display so as to overlap the feeder placement section (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2018-56252

[0004] The system described in Patent Document 1 displays information when it recognizes an identification mark attached to a feeder placement section, but the information display limits the operator's field of view and may reduce workability. For this reason, a system that can display information on a transmissive display at a more appropriate time is desired.

[0005] An operation assistance system according to one aspect of the present disclosure is an operation assistance system that assists an operator in operating industrial machinery, and includes a transparent display worn on the operator's head, and a control device that controls the display of the transparent display, wherein the control device has a display unit that displays an image related to the operation of the industrial machinery on the transparent display, a condition setting unit that sets display conditions that determine whether display by the display unit is necessary, a status acquisition unit that acquires at least one state of at least the industrial machine and the operator, and a condition determination unit that determines whether the state acquired by the status acquisition unit satisfies the display conditions, and causes the display unit to perform display if the display conditions are satisfied.

[0006] 1 is a block diagram illustrating a configuration of an operation assistance system according to an embodiment of the present disclosure;

[0023] FIG. 2 is a schematic diagram illustrating a state in which the operation assistance system of FIG.

[0007] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Fig. 1 is a block diagram showing a configuration of an operation assistance system 1 according to an embodiment of the present disclosure. Fig. 2 is a schematic diagram showing how the operation assistance system 1 is used.

[0008] The operation assistance system 1 is used to assist an operator H in operating an industrial machine M. The operation assistance system 1 includes a head-mounted terminal 10 and a control device 20. The industrial machine M may be, for example, an NC lathe, and generally includes an operation panel P having a display D and an input device E.

[0009] As shown in FIG. 2 , the head-mounted terminal 10 is worn on the head of an operator H. The head-mounted terminal 10 also has a transparent display 11, a camera 12, a microphone 13, and a distance sensor 14. The head-mounted terminal 10 may have a form such as glasses, goggles, or a helmet. The transparent display 11 is placed in front of the eyes of the operator H and can display an image visible to the operator H, and has a transparent screen through which the other side can be seen. The camera 12 captures an image of the area that the operator H can see through the transparent display 11. The microphone 13 converts sound into an electrical signal. The distance sensor 14 detects the depth of field of the image captured by the camera 12.

[0010] The control device 20 controls the display on the transmissive display 11. As shown in FIG. 2 , the control device 20 may be configured to communicate with the head-mounted terminal 10 via wireless communication, may be incorporated into the head-mounted terminal 10, or may be configured to be carried by the operator H separately from the head-mounted terminal 10. The control device 20 has a memory, a processor, an input / output interface, etc., and may be realized by one or more computer devices that execute appropriate control programs. The control device 20 may be integrated with the control device of the industrial machine M.

[0011] The control device 20 has a display unit 21, a condition setting unit 22, a state acquisition unit 23, an image processing unit 24, a condition determination unit 25, a voice operation unit 26, and a gesture operation unit 27. Note that these components categorize the functions of the control device 20, and do not necessarily have to be clearly distinguishable in terms of physical configuration and program configuration.

[0012] The display unit 21 displays on the transmissive display 11 an image related to the operation of the industrial machine M. The display unit 21 may display on the transmissive display 11 the same information as on the display D of the industrial machine M, or may provide a dedicated display screen. The display unit 21 may also display information guiding the user to perform voice operations, gesture operations, etc., which will be described later.

[0013] The condition setting unit 22 sets display conditions that are determined to be necessary for display by the display unit 21. The condition setting unit 22 may read out display conditions stored in a memory or the like, or may accept input of display conditions via an input device, a communication line, a recording medium, or the like.

[0014] Examples of display conditions include when no operation is being performed on the control panel P of the industrial machine M, when a specific operation (e.g., manual pulse operation) is being performed on the control panel P of the industrial machine, when the operator H is operating the industrial machine M by at least one of voice and gestures, when the operator H is not looking at the control panel P of the industrial machine M, when the operator H is in a specific position (e.g., an internal work position of the industrial machine M or a secondary operation position away from the main control panel such as in front of the control panel of the tool magazine), and when the operator H's hands are full.

[0015] The status acquisition unit 23 acquires at least one status of at least either the industrial machine M or the operator H. That is, the status acquisition unit 23 acquires status values ​​that are parameters of the display conditions set in the condition setting unit 22. Specifically, the status of the industrial machine M or the operator H acquired by the status acquisition unit 23 may include the operation status of the control panel P of the industrial machine M, whether the operator H is operating the industrial machine M by voice, whether the operator H is operating the industrial machine M by gestures, whether the operator H is looking at the display D of the control panel P, whether the operator H is located away from the control panel P, whether the operator H's hands are occupied, and the like. Furthermore, the status acquisition unit 23 may determine whether the operator H is currently operating the control panel P based on the operation history of the control panel P. The status acquisition unit 23 may determine that the operator H is operating the industrial machine M by voice when the voice operation unit 26 is enabled, or may determine that the operator H is operating the industrial machine M by gesture operation when the gesture operation unit 27 is enabled. Furthermore, the status acquisition unit 23 may acquire the status of the industrial machine M or the operator H from the image processing unit 24, which will be described next. Whether the operator H's hands are occupied can be determined by the image processing unit 24. In addition, it can also be determined that the operator H's hands are occupied, for example, after the door opening operation of the machine tool has been detected, or when the operator H is near a work position inside the machine, etc.

[0016] The image processing unit 24 determines at least one state of the industrial machine M and / or the operator H by image processing of the image captured by the camera 12. For example, the image processing unit 24 may be configured to determine that the operator H is in a specific position (e.g., a position where the interior of the industrial machine M can be accessed, a secondary operation position, or a position away from the control panel P) when a predetermined marker K disposed on the industrial machine M or a pre-registered feature point of the industrial machine M is present in the image captured by the camera 12 in a state that satisfies a predetermined criterion. Specifically, the image processing unit 24 may determine that the operator H is near the marker K when the size of a figure drawn by the marker K or the feature point in the image captured by the camera 12 is equal to or larger than a predetermined size. Alternatively, the position of the operator H may be determined by estimating the movement distance of the camera 12 through triangulation using the marker K or the feature point in images continuously captured by the camera 12. In particular, when a marker K or a feature point inside the industrial machine M is present in the image captured by the camera 12, it can be inferred that the operator H is working inside the machine. Note that the detection of the marker K can be performed using well-known methods such as pattern matching and machine learning.

[0017] Furthermore, the image processing unit 24 may be configured to determine that the operator H's hands are occupied when the hand of the operator H holding an object is present in the image captured by the camera 12. The image processing unit 24 may also be configured to determine that the hand of the operator H is holding an object when it is determined that the operator H is wearing gloves. Furthermore, when the area of ​​a region in the image captured by the camera 12 that is brighter than its surroundings is equal to or greater than a threshold, the image processing unit 24 may determine that the region that is brighter than its surroundings is the display D of the operation panel P, that is, determine that the operator H is looking at the display D.

[0018] The condition determination unit 25 may be configured to determine whether the state acquired by the state acquisition unit 23 satisfies a display condition, cause the display unit 21 to perform display when any of the display conditions is satisfied, and cause the display unit 21 to stop display when none of the display conditions is satisfied. The condition determination unit 25 may also be configured to cause the display unit 21 to perform display when multiple states satisfy the display conditions. When the display unit 21 is to perform display on the transmissive display 11, the condition determination unit 25 may input a command signal to the industrial machine M to turn off the power of the display D or to activate a screen saver for the display D.

[0019] The voice operation unit 26 analyzes the voice signal from the microphone 13 to identify the voice instruction of the operator H and inputs an operation signal corresponding to the industrial machine M.

[0020] The gesture operation unit 27 analyzes the image captured by the camera and the depth of field measured by the distance sensor to determine the instruction given by the operator H's gesture, and inputs an operation signal corresponding to the industrial machine M.

[0021] The operation support system 1 having the above configuration can display information on the transmissive display only when necessary by the operator H, depending on the status of the industrial machine M and the operator H, thereby improving the work efficiency of the operator H regarding the industrial machine M.

[0022] The following supplementary notes are further disclosed regarding the above-described embodiment and modified examples. (Supplementary Note 1) An operation support system (1) is an operation support system (1) that supports an operator (H) in operating an industrial machine (M), and includes a transmissive display (11) worn on the head of the operator (H), and a control device (20) that controls the display of the transmissive display (11), and the control device (20) has a display unit (21) that displays an image related to the operation of the industrial machine (M) on the transmissive display (11), a condition setting unit (22) that sets display conditions for determining that display by the display unit (21) is necessary, a status acquisition unit (23) that acquires at least one state of at least one of the industrial machine (M) and the operator (H), and a condition determination unit (25) that determines whether the state acquired by the status acquisition unit (23) satisfies the display conditions and causes the display unit (21) to perform display if the display conditions are satisfied.

[0023] (Supplementary Note 2) In the operation support system (1) of Supplementary Note 1, the state acquired by the state acquisition unit (23) may include an operation status of the control panel (P) of the industrial machine (M).

[0024] (Supplementary Note 3) In the operation assistance system (1) of Supplementary Note 1 or 2, the state acquired by the state acquisition unit (23) may include whether or not the operator (H) is operating the industrial machine (M) by at least one of voice and gesture.

[0025] (Supplementary Note 4) In the operation support system (1) of Supplementary Notes 1 to 3, the state acquired by the state acquisition unit (23) may include the position of the operator (H).

[0026] (Supplementary Note 5) In the operation support system (1) of Supplementary Note 4, the position of the operator (H) may be determined whether or not the operator (H) is located away from the control panel (P) of the industrial machine (M).

[0027] (Supplementary Note 6) In the operation support system (1) of Supplementary Notes 1 to 5, the state acquired by the state acquisition unit (23) may include whether the operator's (H) hands are occupied.

[0028] (Supplementary Note 7) The operation support system (1) of Supplementary Note 4 or 5 may further include a camera (12) attached to the head of the operator (H), and the control device (20) may further include an image processing unit (24) that determines that the operator (H) is at a specific position when a predetermined marker (K) disposed on the industrial machine (M) or a pre-registered feature point of the industrial machine (M) is present in an image captured by the camera (12) in a state that satisfies a predetermined criterion.

[0029] (Supplementary Note 8) The operation support system (1) of Supplementary Note 6 may further include a camera (12) attached to the head of the operator (H), and the control device (20) may further include an image processing unit (24) that determines that the operator's (H's) hands are occupied when the operator's (H's) hands are holding something in the image captured by the camera (12).

[0030] Although the present disclosure has been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible to these embodiments without departing from the gist of the present disclosure or the gist of the present disclosure derived from the content of the claims and their equivalents.

[0031] For example, in the operation assistance system of the above-described embodiment, the states of the industrial machine and the operator acquired by the state acquisition unit are merely examples and may be omitted or replaced. Furthermore, in the operation assistance system of the above-described embodiment, a distance sensor is used to detect the operator's gestures, but other sensors may be used to detect the gestures. Furthermore, to enable the operator to operate the industrial equipment, the head-mounted terminal may be provided with an input device such as a switch or touch sensor that can be operated by the operator, or the industrial equipment may be operated using a wearable input device that can be separately worn by the operator.

[0032] REFERENCE SIGNS LIST 1 Operation support system 10 Head-mounted terminal 11 Transmissive display 12 Camera 13 Microphone 14 Distance sensor 20 Control device 21 Display unit 22 Condition setting unit 23 Status acquisition unit 24 Image processing unit 25 Condition determination unit 26 Voice operation unit 27 Gesture operation unit D Display E Input device M Industrial machine P Control panel

Claims

1. An operation support system that supports an operator in operating industrial machinery, comprising: a transparent display worn on the operator's head; and a control device that controls the display of the transparent display, wherein the control device has: a display unit that displays an image related to the operation of the industrial machinery on the transparent display; a condition setting unit that sets display conditions that determine whether display by the display unit is necessary; a status acquisition unit that acquires at least one state of at least the industrial machinery and the operator; and a condition determination unit that determines whether the state acquired by the status acquisition unit satisfies the display conditions, and causes the display unit to perform display if the display conditions are satisfied.

2. An operation support system according to claim 1, wherein the status acquired by the status acquisition unit includes the operating status of the control panel of the industrial machine.

3. An operation support system according to claim 1 or 2, wherein the status acquired by the status acquisition unit includes whether or not the operator is operating the industrial machine by at least one of voice and gesture.

4. An operation support system according to any one of claims 1 to 3, wherein the status acquired by the status acquisition unit includes the position of the operator.

5. An operation support system according to claim 4, wherein the location of the operator is whether or not the operator is located away from an operation panel of the industrial machine.

6. An operation support system according to any one of claims 1 to 5, wherein the state acquired by the state acquisition unit includes whether the operator's hands are occupied.

7. An operation support system as described in claim 4 or 5, further comprising a camera mounted on the head of the operator, wherein the control device further comprises an image processing unit that determines that the operator is at a specific position when a predetermined marker disposed on the industrial machine or a pre-registered feature point of the industrial machine is present in an image captured by the camera in a state that satisfies predetermined criteria.

8. An operation support system as described in claim 6, further comprising a camera attached to the operator's head, wherein the control device further comprises an image processing unit that determines that the operator's hands are occupied when the operator's hands are holding something in an image captured by the camera.

Images