Operation assistance system

Abstract

An operation assistance system that assists an operator with operation of an industrial machine, the operation assistance system including: a transmissive display that is worn on the head of the operator; and a control device that controls display of the transmissive display, wherein the control device has: a display section that displays an image related to operation of the industrial machine on the transmissive display; a condition setting section that sets a display condition for determining that display by the display section is required; a state acquisition section that acquires at least one state of at least either of the industrial machine and the operator; and a condition determination section that determines whether the state acquired by the state acquisition section satisfies the display condition, and causes the display section to display in the case where the display condition is satisfied.

Description

Technical Field

[0001] This invention relates to an operator assistance system. Background Technology

[0002] Commercially available head-mounted terminals include a transmissive display and a camera. The use of such head-mounted terminals to assist operators of industrial machinery has also been studied. As an example, a proposed solution is as follows: To assist in the installation of a belt feeder for holding electronic components in an apparatus for mounting electronic components onto a substrate, an identification mark attached to the feeder mounting section in the image from the camera of the head-mounted terminal is identified, and information such as the belt feeder to be installed on the feeder mounting section and the operation process is displayed on the transmissive display in a manner that overlaps with the feeder mounting section (see Patent Document 1). Existing technical documents Patent documents

[0003] Patent Document 1: Japanese Patent Application Publication No. 2018-56252 Summary of the Invention The problem the invention aims to solve

[0004] In the system described in Patent Document 1, information is displayed when an identification mark attached to the feeder mounting section is detected. However, this information display may limit the operator's field of vision, reducing operability. Therefore, a system that can display information on a transmissive display at a more appropriate timing is desired. Solution for solving the problem

[0005] One aspect of this disclosure relates to an operator assistance system for operating industrial machinery, comprising: a transmissive display worn on the operator's head; and a control device for controlling the display of the transmissive display, wherein the control device comprises: a display unit for displaying images related to the operation of the industrial machinery on the transmissive display; a condition setting unit for setting display conditions for determining whether the display unit needs to display; a status acquisition unit for acquiring at least one status of at least one of the industrial machinery and the operator; and a condition determination unit for determining whether the status acquired by the status acquisition unit satisfies the display conditions, and, if the display conditions are satisfied, causing the display unit to display. Attached Figure Description

[0006] Figure 1 This is a block diagram illustrating the structure of an operation assistance system according to one embodiment of the present disclosure. Figure 2 It is shown Figure 1 A schematic diagram illustrating the usage of the operation assistance system. Detailed Implementation

[0007] The embodiments of this disclosure will now be described with reference to the accompanying drawings. Figure 1 This is a block diagram illustrating the structure of an operation assistance system 1 according to one embodiment of the present disclosure. Figure 2 This is a schematic diagram illustrating the use of the operation assistance system 1.

[0008] The operator assistance system 1 assists the operator H in operating the industrial machinery M. The operator assistance system 1 includes a head-mounted terminal 10 and a control device 20. The industrial machinery M may be, for example, an NC lathe, and typically has an operator panel P with a display D and an input device E.

[0009] like Figure 2 As shown, the head-mounted terminal 10 is worn on the head of the operator H. The head-mounted terminal 10 includes a transmissive display 11, a camera 12, a microphone 13, and a distance sensor 14. The head-mounted terminal 10 can be, for example, in the form of eyeglasses, goggles, or a helmet. The transmissive display 11 is positioned in front of the operator H's eyes, capable of displaying images that the operator H can visually recognize, and has a screen that is transparent enough to see through to the opposite side. The camera 12 captures images of the area that the operator H can see through the transmissive display 11. The microphone 13 converts sound into electrical signals. The distance sensor 14 detects the depth of field of the image captured by the camera 12.

[0010] The control device 20 controls the display of the transmissive display 11. The control device 20 can be configured as follows: Figure 2 The control device 20 can communicate with the head-mounted terminal 10 wirelessly as shown, or it can be assembled into the head-mounted terminal 10, or it can be configured to be carried separately from the head-mounted terminal 10 by the operator H. The control device 20 has a memory, a processor, input / output interfaces, etc., and can be implemented by one or more computer devices that execute appropriate control programs. The control device 20 can also be integrated with the control device of the industrial machinery M.

[0011] The control device 20 includes a display unit 21, a condition setting unit 22, a status acquisition unit 23, an image processing unit 24, a condition determination unit 25, a voice operation unit 26, and a gesture operation unit 27. Furthermore, these structural elements are derived by classifying the functions of the control device 20, and may not be structural elements that can be clearly distinguished in terms of physical and program structure.

[0012] Display unit 21 displays images related to the operation of industrial machinery M on the transmissive display 11. Display unit 21 can perform the same display as the display D of industrial machinery M on the transmissive display 11, or it can prepare a dedicated display screen. In addition, display unit 21 can also display images for guiding voice operations, gesture operations, etc., as described later.

[0013] The condition setting unit 22 sets the display conditions that are determined to be displayed by the display unit 21. The condition setting unit 22 can read the display conditions stored in the memory or the like, or it can accept the input of display conditions via an input device, communication line, recording medium or the like.

[0014] As examples of display conditions, the following are given: no operation is being performed on the control panel P of industrial machine M; a specific operation (e.g., manual pulse operation) is being performed on the control panel P of industrial machine M; operator H is operating industrial machine M through at least one of voice and gesture; operator H is not looking at the control panel P of industrial machine M; operator H is in a specific position (e.g., a secondary operation position away from the main control panel, such as an in-machine working position of industrial machine M or in front of the tool magazine control panel); operator H's hands are occupied, etc.

[0015] The status acquisition unit 23 acquires at least one status of either the industrial machine M or the operator H. That is, the status acquisition unit 23 acquires the status values ​​of parameters set as display conditions in the condition setting unit 22. Specifically, the status of the industrial machine M or the operator H acquired by the status acquisition unit 23 may include: the operating status of the control panel P of the industrial machine M, whether the operator H is operating the industrial machine M via voice, whether the operator H is operating the industrial machine M via gestures, whether the operator H is viewing the display D of the control panel P, whether the operator H is located away from the control panel P, and whether the operator H's hand is occupied. Furthermore, the status acquisition unit 23 may also determine whether the operator H is currently operating the control panel P based on the operation history of the control panel P. The status acquisition unit 23 may determine that the operator H is operating the industrial machine M via voice when the voice operation unit 26 is activated, or it may determine that the operator H is operating the industrial machine M via gestures when the gesture operation unit 27 is activated. Additionally, the status acquisition unit 23 may also acquire the status of the industrial machine M or the operator H from the image processing unit 24, which will be described next. In addition, the image processing unit 24 can determine whether the operator H's hand is occupied, for example, after the machine door is detected to be open, or when the operator H is near the working position inside the machine.

[0016] The image processing unit 24 determines at least one state of either the industrial machine M or the operator H by processing images captured by the camera 12. For example, the image processing unit 24 may be configured to determine that the operator H is located at a specific position (e.g., a position that allows access to the interior of the industrial machine M, a secondary operating position, or a position away from the control panel P) if a predetermined marker K or pre-registered feature point of the industrial machine M exists in the image captured by the camera 12, satisfying a predetermined reference. Specifically, if the size of the graphic depicted by the marker K or feature point in the image captured by the camera 12 is greater than a predetermined size, it can be determined that the operator H is near the marker K. The position of the operator H can also be determined by estimating the movement distance of the camera 12 using triangulation of the marker K or feature point in images continuously captured by the camera 12. In particular, if a marker K or feature point inside the industrial machine M is present in the image captured by the camera 12, it can be inferred that the operator H is performing in-machine work. Furthermore, the detection of the marker K can be performed using known methods such as pattern matching and machine learning.

[0017] Alternatively, the image processing unit 24 can be configured to determine that the operator H's hand is occupied if the image captured by the camera 12 shows the operator H's hand holding an object. The image processing unit 24 can also be configured to determine that the operator H's hand is holding an object even if the operator H is wearing gloves. Furthermore, the image processing unit 24 can also determine that the area with higher brightness than the surrounding area in the image captured by the camera 12 is the display D of the control panel P if the area is above a threshold, i.e., determine that the operator H is viewing the display D.

[0018] The condition determination unit 25 can be configured to determine whether the state acquired by the state acquisition unit 23 satisfies the display conditions, and if any display condition is satisfied, to cause the display unit 21 to display, and if any display condition is not satisfied, to cause the display unit 21 to stop displaying. Alternatively, the condition determination unit 25 can be configured to cause the display unit 21 to display if multiple states satisfy the display conditions. Furthermore, the condition determination unit 25 can also, when displaying on the transmissive display 11 via the display unit 21, input command signals to the industrial machine M to turn off the power to the display D or to activate the screen saver of the display D.

[0019] The voice operation unit 26 analyzes the voice signal from the microphone 13 to determine the voice-based instruction from the operator H and inputs the operation signal corresponding to the industrial machinery M.

[0020] The gesture control unit 27 analyzes the image captured by the camera and the depth of field measured by the distance sensor, determines the gesture-based instruction of the operator H, and inputs the operation signal corresponding to the industrial machine M.

[0021] The operation assistance system 1 with the above structure can display information on the transmissive display only when needed by the operator H, based on the status of the industrial machine M and the operator H, thereby improving the work efficiency of the operator H related to the industrial machine M.

[0022] The following notes are also disclosed regarding the above-described embodiments and variations. (Postscript 1) The operation assistance system (1) is an operation assistance system (1) that assists the operator (H) in operating industrial machinery (M), and includes: a transmissive display (11) which is worn on the head of the operator (H); and a control device (20) which controls the display of the transmissive display (11), wherein the control device (20) includes: a display unit (21) which displays images related to the operation of industrial machinery (M) on the transmissive display (11); a condition setting unit (22) which sets display conditions for determining whether the display unit (21) needs to display; a status acquisition unit (23) which acquires at least one status of at least one of industrial machinery (M) and operator (H); and a condition determination unit (25) which determines whether the status acquired by the status acquisition unit (23) satisfies the display conditions, and causes the display unit (21) to display if the display conditions are satisfied.

[0023] (Postscript 2) In the operation assistance system (1) in Appendix 1, the status acquired by the status acquisition unit (23) may include the operation status of the control panel (P) of the industrial machinery (M).

[0024] (Note 3) In the operation assistance system (1) in Appendix 1 or 2, the status acquired by the status acquisition unit (23) may include whether the operator (H) operates the industrial machinery (M) by at least one of voice and gesture.

[0025] (Note 4) In the operation assistance system (1) in Appendix 1 to 3, the status acquired by the status acquisition unit (23) may include the position of the operator (H).

[0026] (Note 5) In the operation assistance system (1) in Appendix 4, the position of the operator (H) can also be whether the operator (H) is located at the control panel (P) far away from the industrial machinery (M).

[0027] (Note 6) In the operation assistance system (1) in Appendix 1 to 5, the status acquired by the status acquisition unit (23) may include whether the operator's (H) hand is occupied.

[0028] (Note 7) The operation assistance system (1) in Appendix 4 or 5 may also include a camera (12) worn on the head of the operator (H), and the control device (20) may also include an image processing unit (24) that determines that the operator (H) is located in a specific position when a specified mark (K) provided on the industrial machine (M) or a pre-registered feature point of the industrial machine (M) exists in the image captured by the camera (12) in a state that meets a specified reference.

[0029] (Note 8) The operation assistance system (1) in Appendix 6 may also include a camera (12) worn on the head of the operator (H), and the control device (20) may also include an image processing unit (24) that determines that the operator's (H) hand is occupied when the image captured by the camera (12) shows the operator's (H) hand holding something.

[0030] The present disclosure has been described in detail above, but it is not limited to the various embodiments described above. Various additions, substitutions, modifications, partial deletions, etc., can be made to these embodiments without departing from the spirit of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents.

[0031] As an example, in the operation assistance system of the above embodiment, the states of the industrial machinery and the operator acquired by the state acquisition unit are illustrative and can be omitted or replaced. Furthermore, in the operation assistance system of the above embodiment, a distance sensor is used to detect the operator's gestures, but other sensors can also be used to detect gestures. Additionally, to enable the operator to operate the industrial equipment, input devices such as switches and touch sensors that the operator can operate can be provided on the head-mounted terminal, or the industrial equipment can be operated through a wearable input device that the operator can wear separately. Explanation of reference numerals in the attached figures

[0032] 1. Operational assistance system 10 Head-mounted terminals 11 Transmissive Display 12 cameras 13 microphones 14 Distance Sensor 20 Control devices 21 Display Section 22 Condition Setting Department 23 Status Acquisition Department 24 Image Processing Department 25 Condition Determination Section 26. Voice Operation Department 27 Gesture Operation Unit D monitor E input device M Industrial Machinery P control panel

Claims

1. An operation assistance system that assists an operator in operating industrial machinery, said operation assistance system comprising: A transmissive display, which is worn on the operator's head; and A control device that controls the display of the transmissive display. in, The control device has: The display unit displays images related to the operation of the industrial machinery on the transmissive display. The condition setting unit sets display conditions that require the display unit to display the information. A status acquisition unit acquires at least one status of at least either the industrial machinery or the operator; as well as The condition determination unit determines whether the state obtained by the state acquisition unit satisfies the display condition, and if the display condition is satisfied, causes the display unit to display.

2. The operation assistance system according to claim 1, wherein, The status acquired by the status acquisition unit includes the operating status of the control panel of the industrial machinery.

3. The operation assistance system according to claim 1 or 2, wherein, The status acquired by the status acquisition unit includes whether the operator is operating the industrial machinery through at least one of voice and gesture.

4. The operation assistance system according to any one of claims 1 to 3, wherein, The status acquired by the status acquisition unit includes the operator's position.

5. The operation assistance system according to claim 4, wherein, The operator's position refers to whether the operator is located away from the control panel of the industrial machinery.

6. The operation assistance system according to any one of claims 1 to 5, wherein, The status acquired by the status acquisition unit includes whether the operator's hand is occupied.

7. The operation assistance system according to claim 4 or 5, wherein, It also includes a camera that can be worn on the operator's head. The control device also includes an image processing unit that determines that the operator is located in a specific position when a specified mark provided on the industrial machinery or a pre-registered feature point of the industrial machinery exists in an image captured by the camera in a state that meets a specified reference.

8. The operation assistance system according to claim 6, wherein, It also includes a camera that can be worn on the operator's head. The control device also has an image processing unit that determines that the operator's hand is occupied when the image captured by the camera shows the operator's hand holding something.

Images