Human error warning method based on augmented reality

Abstract

A method for warning human error based on augmented reality includes the following steps: using a smart wearable device to photograph multiple physical operating components of an electronic device to obtain interface depth data; performing spatial position calculation based on the interface depth data to set corresponding multiple covering geometric elements between the physical operating components and the smart wearable device to respectively cover the physical operating components; photographing a hand of a wearer of the smart wearable device to obtain the posture and spatial position of the hand; and judging whether the hand touches one of the covering geometric elements based on the posture and spatial position, and if so, further judging whether the corresponding program corresponding to the physical operating component is correct, and if the corresponding program is incorrect, executing a warning action to avoid human error.

Description

Technical Field

[0001] The present invention relates to a human error early warning method, and in particular to a human error early warning method based on augmented reality. Background Art

[0002] With the development of industrial automation, workers in the manufacturing field mostly input the operating instructions of the operating procedures through the instrument interface of the control panel to set the parameters and status of the production equipment and complete the corresponding manufacturing procedures. This type of operation is common in the machinery, semiconductor, chemical and electronics industries, and even in transportation, military and power industries.

[0003] Faced with global economic competition, the current manufacturing industry has high order variability, its production operations are highly complex, the working hours are long, the mental load of on-site work has increased, and personnel are prone to fatigue or slackness, leading to the so-called human error. Despite the existence of complete standard operating procedures, when actually operating the machine, it is still possible to accidentally touch the instrument button or the operating procedure is wrong. Every mistake made by the operator not only damages the products of the batch, but may also cause production line interruption or equipment failure and damage, all of which require time-consuming restarting, calibration and setting of relevant parameters of the machine. In serious cases, it causes safety issues and leads to casualties. In the semiconductor industry with highly complex processes, preventing similar human errors has become one of the keys to maintaining competitiveness.

[0004] For traditional production processes, operators can complete their tasks in sequence through pre-work education and training, or by reading the step-by-step instructions in the operation manual. However, such methods are relatively passive and are highly affected by human factors, such as ability and experience, physical and mental state, work pressure, and field environment, and cannot accurately monitor the behavior of personnel.

[0005] Therefore, how to develop proactive practices to predict people's behavioral intentions and provide timely warnings and prevention of errors is the problem that this case aims to solve. Summary of the invention

[0006] Therefore, an object of the present invention is to provide a method for warning of human error based on augmented reality, which integrates augmented reality (AR) and hand posture tracking technology to track the position of fingers in space in real time and predict the operator's intention to operate the instrument interface. If it is judged to be an erroneous program step, it will give a warning in advance through feedback stimulation such as images, sounds or vibrations to prevent human operation errors and avoid subsequent production losses or casualties.

[0007] To achieve the above-mentioned purpose, the present invention provides a human error warning method based on augmented reality, which is applied to a smart wearable device and includes the following steps: using the smart wearable device to photograph multiple physical operating devices on an operating panel of an electronic device to obtain interface depth data of these physical operating devices, wherein the smart wearable device is communicatively connected to these physical operating devices; performing spatial position calculation based on the interface depth data, and setting corresponding multiple covering geometric elements between these physical operating devices and the smart wearable device based on the result of the spatial position calculation to respectively cover these physical operating devices; photographing a hand of a wearer of the smart wearable device to obtain the posture and spatial position of the hand; and judging whether the hand touches one of these covering geometric elements based on the posture and spatial position of the hand, and if so, further judging whether a corresponding program of the physical operating device corresponding to the touched covering geometric element is correct, and if the corresponding program is incorrect, executing a warning action.

[0008] Through the above-mentioned embodiments, it is possible to prevent accidental button touches or operating procedure errors for manually operated instrument interfaces. This technology is based on the concept of augmented reality, combined with hand gesture and position tracking. For each relevant panel button, its spatial position and size are first calculated. Then, according to the set safety distance, a virtual wrapping geometric element is attached to the outside of each panel button. The spatial position of the finger (or other hand) is estimated in real time, and the intersection operation is performed with the above-mentioned wrapping geometric element. If there is an intersection between the two, it means that the person is about to touch the corresponding button, and check the given operating procedure to determine its correctness. If necessary, a warning will be issued, or the control panel will be temporarily disabled to prevent errors. This can solve the problems of known technologies.

[0009] In order to make the above contents of the present invention more clearly understood, preferred embodiments are specifically cited below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A block diagram of a system for implementing the augmented reality-based human error warning method of the present invention is shown.

[0011] Figure 2 A flow chart showing the method for early warning of human error based on augmented reality implemented in the present invention.

[0012] Figure 3 A side view showing the operating status of the human error warning method based on augmented reality of the present invention.

[0013] Figure 4 A front view showing the operating status of the human error warning method based on augmented reality of the present invention.

[0014] Figure 5 and Figure 6 A side view showing the operating status of the human error warning method based on augmented reality of the present invention.

[0015] Figure 7 Show the invention Figure 6 A side view of the operating condition of a variation of the present invention.

[0016] S1~S6: Steps

[0017] 10~19: Physical operating device

[0018] 20: Instrument interface

[0019] 25: Controller

[0020] 30-39: Wrapping geometric elements

[0021] 30A: Outer layer

[0022] 30B: Inner layer

[0023] 100: Electronic equipment

[0024] 200: Smart wearable devices

[0025] 300: Hands

[0026] 310: Finger tips

[0027] 320: Finger joints

[0028] 330: Palm

[0029] 340: Wrist DETAILED DESCRIPTION

[0030] This technology refers to the concept of augmented reality. In the real environment seen by the wearer (or operator) of the smart wearable device through the smart wearable device, a virtual model (or wrapping geometry) to prevent errors is added as the calculation basis for the operator to trigger the button (or physical operating device), and the hand posture tracking technology is integrated to instantly and effectively predict the operator's operating intention. The above technology has many advantages. First, by using image recognition and spatial positioning, the position and size of the panel button can be quickly obtained, which is not limited by the form and type of the machine, nor is it affected by the geometric variation of the operator's hand. Second, it can be used with existing augmented reality smart glasses, and the operator's hands do not need to hold additional devices to change the operating procedures or work habits. Third, it can prevent human negligence through different sensory feedback, including visual, auditory and tactile warnings, or temporarily disable the control panel (disable), and prevent operational errors in many ways.

[0031] Figure 1A block diagram showing a system implementing an augmented reality based human error warning method. Figure 2 A flowchart showing the implementation of the augmented reality-based human error warning method. Figure 1 and Figure 2 As shown, a method for warning of human negligence based on augmented reality is applied to a smart wearable device 200. The smart wearable device 200 includes but is not limited to AR glasses or AR helmets, and is communicatively connected to an electronic device 100. The electronic device 100 includes but is not limited to applications in machinery, semiconductors, chemicals and electronics, and even includes industrial fields such as transportation, military and electricity. The electronic device 100 includes a controller 25 and an instrument interface 20. The instrument interface 20 has a plurality of physical operating devices 10 to 19. The smart wearable device 200 is communicatively connected to the physical operating devices 10 to 19 through the controller 25. When the physical operating devices 10 to 19 are physically pressed, switched, or rotated by the operator, the corresponding program can be executed by the controller 25, including but not limited to power on, power off, emergency stop, control of the processing flow of each processing station (not shown), temperature control, gas exhaust and other programs.

[0032] Figure 3 A side view showing the operation of the augmented reality-based human error warning method. Figures 1 to 3 As shown, the human error warning method includes the following steps. First, the wearer wears the smart wearable device 200 and activates its function, and uses the smart wearable device 200 to shoot multiple physical operating devices 10 to 19 (located on the operating panel) of the electronic device 100 to obtain the interface depth data of these physical operating devices 10 to 19, as shown in step S1. It is worth noting that the results shot by the smart wearable device 200 can be directly displayed to the wearer for viewing, so as to achieve the effect of instantly viewing the wearer's field of view.

[0033] Next, the spatial position calculation is performed based on the interface depth data to obtain the spatial position of the physical operating devices 10 to 19, and based on the result of the spatial position calculation, a corresponding plurality of encapsulating geometric elements 30 to 39 are set between these physical operating devices 10 to 19 and the smart wearable device 200 to respectively encapsulate these physical operating devices 10 to 19, as shown in step S2. These encapsulating geometric elements 30 to 39 are equivalent to providing the function of a protective cover, which can be visible to the wearer, that is, the smart wearable device 200 displays these encapsulating geometric elements 30 to 39 for the wearer to see, so that the wearer knows that the early warning protection function has been activated. In another example, the smart wearable device 200 does not display these encapsulating geometric elements 30 to 39 for the wearer to see, so as not to confuse the wearer's field of vision. These encapsulating geometric elements 30 to 39 can be generated by offsetting the surface of these physical operating devices 10 to 19 outward (toward the wearer) by a distance.

[0034] Therefore, when the human negligence warning function is activated, the wearer needs to cooperate with the initialization steps. When the wearer wears a smart wearable device 200 such as augmented reality smart glasses, the wearer faces the instrument interface 20 of the electronic device 100 to be operated, and can stand at a designated position if necessary. At this time, the camera on the smart wearable device 200 can capture image information, and refer to the pre-given model and image of the instrument interface 20 to perform spatial positioning calculations. After the position of the instrument interface model in three-dimensional space is established, each related physical operating device (such as a button) is directed toward the user, and a wrapping geometry element (Wrapping Geometry) of each related button is added according to a given safety distance. These wrapping geometry elements do not need to be displayed when in use, but are combined with the real environment in a virtual way.

[0035] Figure 4 A front view showing the operation of the augmented reality-based human error warning method. Figures 2 to 4 As shown, the wearer's hand 300 moves between the smart wearable device 200 and the instrument interface 20. At this time, the smart wearable device 200 photographs the wearer's hand 300 to obtain the posture and spatial position of the hand 300, as shown in step S3. The wearer can see his hand 300 through the smart wearable device 200. In one example, the smart wearable device 200 instantly photographs the hand 300, instantly tracks the spatial coordinates of the fingertips 310 of the hand 300, and calculates the spatial Euclidean distance (Euclidean Distance) between the fingertips 310 and the enveloping geometric elements 30 to 39 to obtain the relative position relationship (relative spatial relationship) between these physical operating devices 10 to 19 and the hand 300. When one of these spatial Euclidean distances is less than a false touch warning distance, the corresponding physical operating device is disabled. Furthermore, during the operation, the smart wearable device 200 captures the hand 300 in real time, and uses image recognition technology to obtain the posture and spatial position of the hand 300 by real-time tracking the fingertips 310, finger joints 320, palms 330 and wrists 340 of the hand 300. For example, when the physical operating device 11 needs to be pressed, the above mechanism determines that the wearer intends to rotate the physical operating device 11, and a warning action can also be executed.

[0036] The false touch warning distance mentioned above is equivalent to the safety distance. The safety distance determines the time of the error warning. When the safety distance is set larger, the error can be determined at a farther position, so the reaction time is longer, and vice versa. In addition, the wrapping geometric element can have a simple geometric shape, such as a plane, a circular surface or a spherical surface, or a part of a cube, a cylinder or a sphere. It can also refer to the shape of a physical operating device and be generated by the surface of the physical operating device through offset calculation.

[0037] Figure 5 and Figure 6 A side view showing the operation of a human error warning method based on augmented reality, wherein Figure 5 The tip of the middle finger 310 has not yet touched the covering geometric element 31 / 32, and Figure 6 The finger tip 310 in FIG. 3 has touched the enclosing geometric element 31 / 32. Figures 2 to 6 As shown, the smart wearable device 200 determines whether the hand 300 touches one of the covering geometric elements 30 to 39 (referred to as the touched covering geometric element) based on the posture and spatial position of the hand 300 (as shown in step S4). If so, it further determines whether a corresponding program (the program, step or operation to be executed) of the physical operating device 10 / 11 / 12 / 13 / 14 / 15 / 16 / 17 / 18 / 19 corresponding to the touched covering geometric element 30 / 31 / 32 / 33 / 34 / 35 / 36 / 37 / 38 / 39 is correct (as shown in step S5). If the corresponding program is incorrect, a warning action is executed (as shown in step S6). For example, after the electronic device 100 is started, a batch of wafers are being wet-etched. If the etching has not reached a predetermined time (for example, 40 minutes), water washing cannot be performed. If the hand 300 has pressed the encapsulated geometric element 31, the encapsulated geometric element 31 corresponds to the program of the physical operation device 11 is water washing, and the controller 25 of the electronic device 100 notifies the smart wearable device 200 that the wet etching has only been performed for 30 minutes. At this time, the smart wearable device 200 determines that the corresponding program is incorrect, so it performs an early warning action. Therefore, the smart wearable device 200 can determine whether the corresponding program is correct based on the pre-planned operation program in the electronic device 100. In step S5, if the corresponding program is judged to be correct, no early warning action is performed, and the hand 300 will smoothly operate the physical operation device 11, such as pressing the physical operation device 11 to execute the corresponding program.

[0038] The above-mentioned warning action includes one or more of the following: (a) disabling the physical operating device corresponding to the touched enveloping geometric element; and (b) outputting a warning message. Regarding the above-mentioned warning message, the error (Error), warning (Warning) and other prompt information or graphics can be presented in the smart wearable device 200 in a visual manner, and a warning sound can also be issued at the same time to prevent errors in the form of auditory feedback, or to remind the wearer by tactile sense with vibration. Alternatively, the warning action can also include: disabling all of these physical operating devices 10 to 19. Alternatively, the warning action can also include: disabling the physical operating device corresponding to the touched enveloping geometric element and the physical operating devices around it (that is, partial disabling, such as disabling the physical operating device 11 and the nearest physical operating devices 10 and 12 to 14 (the first layer of closer devices) around it, while the other physical operating devices 15 to 19 are not disabled because they are relatively far away and there is no possibility of accidental touch).

[0039] Figure 7 show Figure 6 A side view of the operating conditions of a variation of Figure 7 As shown, one or more wrapping geometric elements can also be set to a layered safety distance. For example, the wrapping geometric element 30 has an outer layer 30A and an inner layer 30B. When the fingertip 310 of the hand 300 only touches the outer layer 30A (for example, 10 cm away from the physical operating device), a first sub-action of the warning action is executed (for example, outputting a warning message). When the fingertip 310 touches the outer layer 30A and the inner layer 30B (for example, 5 cm away from the physical operating device), a second sub-action of the warning action is executed (for example, disabling the corresponding physical operating device 10), so that further warning protection can be obtained.

[0040] Through the above-mentioned embodiments of the present invention, it is possible to prevent accidental button touches or operating procedure errors for manually operated instrument interfaces. This technology is based on the concept of augmented reality, combined with hand gesture and position tracking. For each relevant panel button, its spatial position and size are first calculated. Then, according to the set safety distance, a virtual wrapping geometric element is attached to the outside of each panel button. The spatial position of the finger (or other hand) is estimated in real time, and the intersection operation is performed with the above-mentioned wrapping geometric element. If there is an intersection between the two, it means that the person is about to touch the corresponding button, and check the given operating procedure to determine its correctness. If necessary, a warning will be issued, or the control panel will be temporarily disabled to prevent errors. This can solve the problems of known technologies.

[0041] It is worth noting that all the above examples can be appropriately combined, replaced or modified interactively to provide diverse effects to meet design requirements.

[0042] The specific embodiments proposed in the detailed description of the preferred embodiments are only used to facilitate the explanation of the technical content of the present invention, rather than narrowly limiting the present invention to the above embodiments. Various changes and implementations made without exceeding the spirit and claims of the present invention are within the scope of the present invention.

Claims

1. A human error warning method based on augmented reality, applied to a smart wearable device, characterized in that: include: Using the smart wearable device to photograph multiple physical operating components of an electronic device to obtain interface depth data of the physical operating components, wherein the smart wearable device is communicatively connected to the physical operating components; Performing spatial position calculation according to the interface depth data, and setting corresponding multiple covering geometric elements between the physical operating devices and the smart wearable device to respectively cover the physical operating devices according to the result of the spatial position calculation; A hand of a wearer of the smart wearable device is photographed to obtain the posture and spatial position of the hand; and based on the posture and spatial position of the hand, it is determined whether the hand touches one of the covering geometric elements. If so, it is further determined whether a corresponding program of the physical operating device corresponding to the touched covering geometric element is correct. If the corresponding program is incorrect, an early warning action is executed.

2. The method for early warning of human error based on augmented reality as claimed in claim 1, characterized in that: The smart wearable device determines whether the corresponding program is correct based on the operating program pre-planned in the electronic device.

3. The method for early warning of human error based on augmented reality as claimed in claim 1, characterized in that: The warning action includes: disabling the physical operating device corresponding to the touched covering geometric element.

4. The method for early warning of human error based on augmented reality according to claim 1 or 3, characterized in that: The early warning action includes: outputting a warning message.

5. The method for early warning of human error based on augmented reality as claimed in claim 1, characterized in that: The warning action includes: disabling all of the physical operating devices.

6. The method for early warning of human error based on augmented reality as claimed in claim 1, characterized in that: The smart wearable device does not display the wrapped geometric elements to the wearer.

7. The method for early warning of human error based on augmented reality as claimed in claim 1, characterized in that: The smart wearable device displays the wrapped geometric elements for the wearer to see.

8. The method for early warning of human error based on augmented reality as claimed in claim 1, characterized in that: Each of the enveloping geometric elements has an outer layer and an inner layer. When the hand only touches the outer layer, a first sub-action of the warning action is executed. When the hand touches the outer layer and the inner layer, a second sub-action of the warning action is executed.

9. The method for early warning of human error based on augmented reality as claimed in claim 1, characterized in that: The relative position relationship between the physical operating devices and the hand is obtained by real-time tracking the spatial coordinates of the fingertips of the hand and calculating the spatial Euclidean distance between the fingertips and the enclosing geometric elements.

10. The method for early warning of human error based on augmented reality as claimed in claim 1, characterized in that: The posture and spatial position of the hand are obtained by real-time tracking of the fingertips, finger joints, palm and wrist of the hand.

Images