System and method for detecting working conditions

Abstract

To provide a work state detection system and method capable of more accurately detecting the work state of a work object.SOLUTION: A work state detection system 1 that detects a line-of-sight when a work object 2 is worked, comprises: a line-of-sight reference point setting unit 14 for setting a reference point 51 that tracks the line-of-sight of a worker 3 on the work object; a line-of-sight detection unit 15 for detecting the line-of-sight 31 of the worker; and a line-of-sight tracking unit 16 for calculating a relative coordinate value of a contact point 21 where the worker's line-of-sight contacts the work object from the line-of-sight reference point 51 as line-of-sight trajectory data based on the line-of-sight detected by the line-of-sight detection unit, and storing the calculated line-of-sight trajectory data in a storage unit 17.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a work state detection system and method.

Background Art

[0002] In inspection work, assembly work, processing work, etc., for maintaining and improving work quality, work stability and uniformity are desired. In Patent Document 1, by attaching a gaze tracking device to an operator and tracking the operator's gaze, wiring is automatically identified to prevent incorrect wiring. In Patent Document 2, by tracking the operator's gaze and comparing it with correct gaze tracking data, omission of description in the work instruction manual is detected. Further, in Patent Document 2, the movement of the operator is detected and displayed on a plan view or a three-dimensional view of the work place.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] When tracking the gaze of an operator while the operator is working on a work object, there may be a deviation between the visualized gaze and the actual gaze. Further, when the posture of the work object changes, such as rotating or turning it over, an error occurs in the gaze tracking of the operator.

[0005] If the gaze of the operator cannot be accurately detected under a certain standard, the movement of the gaze of a skilled operator and the movement of the gaze of an unskilled operator cannot be compared, and the work quality cannot be analyzed from the movement of the gaze.

[0006] The present invention has been made in view of the above problems, and its objective is to provide a work state detection system and method that can more accurately detect the work state on an object to be worked on. [Means for solving the problem]

[0007] To solve the above problems, a work state detection system according to one aspect of the present invention is a work state detection system that detects the line of sight when working on a work object, comprising: a line of sight reference point setting unit that sets a reference point for tracking the worker's line of sight on the work object; a line of sight detection unit that detects the worker's line of sight; and a line of sight tracking unit that calculates relative coordinate values ​​from the line of sight reference point of the contact point where the worker's line of sight contacts the work object as line of sight trajectory data based on the line of sight detected by the line of sight detection unit, and stores the calculated line of sight trajectory data in a storage unit. [Effects of the Invention]

[0008] According to the present invention, since the line of sight trajectory data is calculated as the relative coordinate value of the contact point where the worker's line of sight comes into contact with the work object from the line of sight reference point, the influence of positional displacement of the work object can be suppressed, and the line of sight can be detected stably. [Brief explanation of the drawing]

[0009] [Figure 1] A system configuration diagram of the work status detection system according to this embodiment. [Figure 2] Hardware configuration diagram of the work status detection system. [Figure 3] Functional configuration diagram of the external system and MR goggles. [Figure 4] This diagram illustrates how a 3D model is superimposed onto the work object, and the position where the worker's line of sight contacts the work state detection system is detected as a relative coordinate value from a line of sight reference point set on the 3D model. [Figure 5] Flowchart for detecting the work status. [Figure 6] Flowchart for work state analysis processing. [Figure 7]An example of a screen for analyzing the work status. [Figure 8] A modified example of the method for setting the line of sight reference point. [Figure 9] A flowchart illustrating the process for generating a 3D model according to the second embodiment. [Figure 10] Flowchart for detecting the work status. [Figure 11] A system configuration diagram of the work state detection system according to the third embodiment. [Figure 12] Flowchart for setting the line of sight reference point. [Figure 13] A system configuration diagram of the work state detection system according to the fourth embodiment. [Figure 14] A system configuration diagram of the work state detection system according to the fifth embodiment. [Figure 15] A system configuration diagram of the work state detection system according to the sixth embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. The work state detection system 1 described in this embodiment accurately detects the line of sight of the worker 3 when the worker 3 is working on the work object 2, based on a line of sight reference point 51 set within the work object 2. The line of sight reference point 51 may be set within a three-dimensional model of the work object 2 that is virtually superimposed on the work object 2. Alternatively, the shape of the work object 2 may be recognized from image data of the work object 2, and the line of sight reference point 51 may be virtually set within the work object 2.

[0011] In either case, since the line of sight reference point 51 is located within the work object 2 (within the 3D model superimposed on the work object 2), even if the positional relationship between the worker 3 and the work object 2 changes, it does not affect the relationship between the contact point 21 where the worker 3's line of sight 31 touches the surface of the work object 2 and the line of sight reference point 51. Furthermore, even if the position of the worker 3's eyes (height) differs slightly from person to person, the impact on the relationship between the contact point 21 and the line of sight reference point 51 can be minimized.

[0012] Therefore, according to this embodiment, during the operation of the operator 3 on the work object 2, the line of sight 31 of the operator 3 can be accurately detected. It is also possible to compare the movements of the line of sight 31 during the operation of the same operator 3 on the same work object 2 at different dates and times. It is also possible to compare the movement of the line of sight 31 during the operation of an operator 3 who is skilled in the work with the movement of the line of sight 31 during the operation of an operator 3 who is not skilled in the work. Thereby, the tacit knowledge of the skilled worker can be converted into formal knowledge, and educational materials for educating workers who are not used to the work can also be generated.

[0013] The work state detection system 1 of this embodiment is applicable to operations such as inspection work, inspection work, assembly work, processing work, welding work, painting work, etc. The work object 2 is applicable to various objects such as electrical products, automobiles, robots, control panels, storage batteries, etc. The work object 2 may be a material such as ice or a rock. In this case, the operator 3 is a craftsman or an artist, etc., and the movement of the line of sight 31 when the operator � processes the material can be detected.

[0014] Furthermore, in the embodiments described below, not only the movement of the line of sight 31 of the operator 3, but also the movement of the fingertips of the operator 3 or the movement of the head of the operator 3 are detected based on the line of sight reference point 51. By detecting these movements of the fingertips or the head together with the movement of the line of sight, the work state of the operator 3 can be analyzed in more detail.

Example

[0015] The first embodiment will be described with reference to FIGS. 1 to 8. FIG. 1 is a system configuration diagram of the work state detection system 1 according to this embodiment. The work state detection system 1 can be configured using a computer as described below, and as its functional units, it includes a photographed image acquisition unit 11, a three-dimensional model acquisition unit 12, an overlay unit 13, a line of sight reference point setting unit 14, a line of sight detection unit 15, a line of sight trajectory data calculation unit 16, a storage unit 17, and an output unit 18.

[0016] The image acquisition unit 11 has the function of acquiring images visible to the worker 3 from the camera 404 (see Figure 2) installed in the MR goggles 4. The 3D model acquisition unit 12 has the function of acquiring a 3D model of the work object 2. The overlay unit 13 has the function of overlaying the 3D model onto the work object 2 in the image captured by the MR goggles 4. The gaze reference point setting unit 14 has the function of setting a gaze reference point 51 at a predetermined position in the 3D model. In the figure, the 3D model is abbreviated as "3D model". The MR goggles 4 may also be augmented reality (AR) goggles. The MR goggles 4 can be any device that can recognize objects in the real space visible to the worker 3 and overlay a 3D model onto the real space (real world), and its name and type are not specified.

[0017] The image acquisition unit 11, the 3D model acquisition unit 12, and the overlay unit 13 may be implemented within the MR goggles 4. The MR goggles 4 will be described further in Figure 3.

[0018] The gaze detection unit 15 is implemented by the gaze tracking sensor 405 (see Figure 2) provided in the MR goggles 4. Although it is the gaze tracking sensor 405 in terms of hardware, in terms of function it is the gaze detection unit 15. The gaze trajectory data calculation unit 16, which acts as the "gaze tracking unit," calculates the trajectory of the detected gaze and stores it as gaze trajectory data in the storage unit 17.

[0019] Here, the gaze trajectory data calculation unit 16 calculates the position of the point (contact point) 21 where the worker's gaze 31 touches the surface of the work object 2 as a relative coordinate value from the gaze reference point 51. If the gaze tracked by the gaze tracking sensor 405 of the MR goggles 4 does not reach the surface of the work object 2, the gaze tracked by the sensor 505 can be extended in the computer toward the work object 2. The contact point 21 can also be called the intersection point 21 or viewpoint 21 where the gaze 31 intersects the surface of the work object 2. The gaze trajectory data calculation unit 16 calculates gaze trajectory data by arranging the gazes on the surface of the work object 2, i.e., the contact points 21, in chronological order.

[0020] The memory unit 17 stores the gaze trajectory data. The memory unit 17 is composed of a storage device such as a semiconductor memory device, a hard disk drive, or a flash memory device. The output unit 18 outputs the data stored in the memory unit 17 (in this case, gaze trajectory data) to an external system such as the work state analysis system 6.

[0021] The work state analysis system 6 can, for example, analyze changes in the eye-tracking data of a certain worker 3 over a predetermined period of time, or compare the eye-tracking data of a certain worker 3 with the eye-tracking data of other workers for the same work on the same work object 2.

[0022] Figure 2 shows the hardware configuration of the work state detection system 1. The work state detection system 1 includes, for example, a work state detection device 100 and MR goggles 4.

[0023] The work state detection device 100 is configured as a computer and includes, for example, a processor 101, memory 102, communication unit 103, user interface unit 104, and storage 105. In the figure, the user interface is abbreviated as "UI".

[0024] The processor 101 realizes the functional units 11 to 18 described in Figure 1 by reading a predetermined computer program (not shown) stored in the storage 105 into the memory 102 and executing it. There is not limited to one processor 101. The processor 101 may include a specific arithmetic unit that performs dedicated arithmetic processing.

[0025] The communication unit 103 is, for example, a device that communicates with the MR goggles 4 and external systems 5 and 6. It may also be divided into a communication unit that communicates with the MR goggles 4 and a communication unit that communicates with the external systems 5 and 6. Here, however, it is shown as if both were a single communication unit. The user interface unit 104 provides information input and information output functions to the user who uses the work state detection system 1. The user interface device 104 is configured using, for example, a monitor display, a printer, a keyboard, a pointing device, a touch panel (none of which are shown). The user can also use the work state detection system 1 in a virtual space using VR goggles (none of which are shown) or MR goggles.

[0026] The MR goggles 4 are a device that overlays a virtual space onto the real world (real space) visible to the worker 3, and are worn on the worker 3's head. The MR goggles 4 include, for example, a processor 401, memory 402, communication unit 403, camera 404, eye-tracking sensor 405, retinal display 406, speaker 407, and microphone 408.

[0027] The processor 401 reads and executes a predetermined computer program (not shown) stored in the memory 402, thereby realizing functions such as eye-tracking detection and projecting information onto the user's retina. The communication unit 403 is a device that communicates bidirectionally with the work state detection device 100.

[0028] Camera 404 is a device that captures the real world from the worker's line of sight. Camera 404 may also have a distance measuring function capable of detecting depth. Eye-tracking sensor 405 detects the line of sight, for example, by recognizing the movement of the user's eyeballs with the camera. Retinal display 406 provides the worker 3 with a virtual image superimposed on the real world by projecting an image onto the user's retina.

[0029] The speaker 407 and microphone 408 may or may not be built into the MR goggles 4.

[0030] External systems 5 and 6 are external computers connected to the work state detection system 1. One external system 5 is an upstream system that provides 3D model data of the work object 2, such as a CAD system, to the work state detection system 1. The other external system 6 is a downstream system that analyzes the work state of the worker 3, etc., using the gaze trajectory data calculated by the work state detection system 1.

[0031] Figure 3 shows a case where the upstream system 5 is a computer that creates 3D model data of the work object 2, such as a CAD system. The external system 5 includes a 3D model generation unit 52 that generates 3D model data and a 3D model conversion unit 53 that converts the generated 3D model data into data 50 for use with the MR goggles 4. Hereinafter, the 3D model data converted for use with the MR goggles 4 will be referred to as the 3D model 50.

[0032] The MR (Mixed Reality) goggles 4 are a device that overlays virtual space information onto visual information from the real world. The MR goggles 4 recognize space from image data captured by the camera 404. The MR goggles 4 align and overlay a 3D model 50 onto the spatially recognized work object 2.

[0033] Figure 4 shows how a 3D model 50 of the work object 2 is superimposed onto the work object 2 as it is actually visible to worker 3, and how this is tracked by worker 3's line of sight.

[0034] As shown in Figure 4(1), when worker 3 wears MR goggles 4 and looks at the real-world work object 2, the spatial recognition unit 41 of the MR goggles 4 recognizes the work object 2. As shown in Figures 4(1) and (2), the spatial recognition unit 41 of the MR goggles 4 superimposes a 3D model 50 onto the recognized work object 2.

[0035] As shown in Figure 4(3), when worker 3 looks at the work object 2, a point 21 is obtained where worker 3's line of sight 31 contacts the surface of the work object 2 (the surface of the 3D model 50). The positional relationship between this contact point 21 and the line of sight reference point 51 set at the center of the 3D model 50 is calculated as a relative coordinate value. Since the line of sight reference point 51 is set inside the work object 2, the relationship between the work object 2 and the contact point 21 of the line of sight 31 is less affected by the height of worker 3's eyes or changes in the posture of the work object 2.

[0036] Figure 5 is a flowchart of the work state detection process. The work state detection system 1 generates data for a 3D model of the work object 2 and sets the line of sight reference point 51 (S11). Then, the work state detection system 1 converts the generated 3D model data into data for use with the MR goggles 4 and transfers it to the MR goggles 4 (S12).

[0037] When worker 3 views the work object 2 through the MR goggles 4, the camera 404 captures an image of the work object 2, and the captured image data is transmitted in real time to the work state detection device 100 (S13).

[0038] Almost simultaneously, the MR goggles 4 spatially recognize the work object 2 and superimpose the 3D model 50 onto the recognized work object 2 (S14). The work state detection system 1 calculates the positional relationship between the point 21 where the worker's line of sight 31 contacts the work object 2 and the line of sight reference point 51 as relative coordinate values ​​(S15), and stores the calculated relative coordinate data set as line of sight trajectory data in the storage unit 17 (S16).

[0039] Figure 6 is a flowchart showing the process of analyzing the work state using eye-tracking data. This process is performed by the work state analysis system 6. The work state analysis system 6 may also be provided within the work state detection system 1.

[0040] The work state analysis system 6 acquires eye-tracking data (S21) and also acquires image data of the worker's eye level captured by the camera 404 of the MR goggles 4 (S22). The work state analysis system 6 overlays each contact point 21 (viewpoint 21) that constitutes the eye-tracking data onto the work object 2 in the image data (S23). The work state analysis system 6 calculates statistical information of the eye-tracking data (S25).

[0041] The work state analysis system 6 outputs dynamic image data of the work object 2 with the worker's viewpoint 21 plotted on it, dynamic image data of the worker's viewpoint 31 plotted on the 3D model 50, and statistical information in association (S26). Since the line of sight reference point 51 is set on the 3D model 50, the position 21 where the line of sight 31 contacts the surface of the work object 2 is calculated as the point where the surface of the 3D model 50 and the line of sight 31 intersect. Therefore, the work state analysis system 6 can simultaneously output the work object 2 with the viewpoint 21 plotted on it and the 3D model 50 included in the line of sight trajectory data.

[0042] Figure 7 shows an example of the analysis screen G1 output by the work state analysis system 6. The upper part of Figure 7 shows the analysis results of the work state of skilled worker U1. The lower part of Figure 7 shows the work state of inexperienced worker U2.

[0043] The work status analysis screen G1 includes, for example, a work video display unit GP11 that displays a video of the work in progress, a reproduction unit GP12 that reproduces changes in line of sight (changes in viewpoint movement) relative to the 3D model 50, and a statistical information display unit GP13 that displays statistical information.

[0044] In the video of the work in progress, a marker indicating the viewpoint is displayed on the surface of the work object 2. Similarly, a marker indicating the viewpoint is also displayed on the 3D model. In Figure 7, the position and order of the viewpoints are shown with circled numbers. In the statistical information display unit GP13, information on the dwell time (or frequency) of the viewpoints is displayed graphically.

[0045] Comparing the eye-tracking data of skilled worker U1 with that of inexperienced worker U2, skilled worker U1 fixates on fewer points than inexperienced worker U2, indicating that they are not looking at unnecessary areas. In other words, inexperienced worker U2 is unnecessarily fixating on areas that are not highly relevant to the task (see the circled number 2 in the lower right of Figure 7).

[0046] Figure 8 shows an example of the setting position of the line of sight reference point 51. In Figure 1 and other figures, the case where the line of sight reference point 51 is set to the center of the 3D model 50 was described. The center of the 3D model 50 may also be the center of gravity assuming that the 3D model 50 is formed from a uniform material. Alternatively, the line of sight reference point 51 may be set to the center of gravity calculated from the actual structure of the workpiece 2.

[0047] Furthermore, as shown in Figure 8(1), the line of sight reference point 51A may be set on the side of the 3D model 50. As shown in Figure 8(2), the line of sight reference point 51B may be set behind the center of the 3D model 50. As shown in Figure 8(3), the line of sight reference point 51C may be set on the top surface of the 3D model 50. As shown in Figure 8(4), the line of sight reference point 51D may be set on any vertex of the 3D model 50. In this way, the line of sight reference point 51 can be set at any location on the 3D model.

[0048] In this embodiment, since the line of sight reference point 51 is located within the work object 2 (the 3D model 50 superimposed on the work object 2), the relationship between the viewpoint (contact point) 21 and the line of sight reference point 51 is not affected even if the positional relationship between the worker 3 and the work object 2 changes. For this reason, the work state detection system 1 can accurately detect the worker 3's line of sight 31 while the worker 3 is working on the work object 2.

[0049] Furthermore, the work state analysis system 6 can accurately analyze the work state of worker 3 using precisely detected eye-tracking data. The work state analysis system 6 can also convert the tacit knowledge of skilled workers into explicit knowledge and generate training materials for inexperienced workers. In addition, the work state analysis system 6 can output analysis results that are useful, for example, for improving work instruction manuals, work processes, and designs. [Examples]

[0050] Example 2 will be explained using Figures 9 and 10. In the following examples, including this example, the differences from Example 1 will be explained in particular. In this example, a case in which the shape of the workpiece 2 changes as the work process progresses will be explained. Generally, in inspection and quality control processes, the workpiece 2 is a finished product, so its shape does not change significantly. At most, a sticker or the like may be attached to the surface of the workpiece 2. In surface treatment and painting processes, the shape of the workpiece 2 does not change significantly either.

[0051] In contrast, during the machining and assembly processes, the external shape of the workpiece 2 changes significantly as the work progresses. In this case, if a 3D model that corresponds to the shape change of the workpiece 2 is not used, it is not possible to accurately determine the location 21 that the worker's 3 line of sight 31 is looking at. Therefore, in this embodiment, the 3D model used for line of sight detection is updated according to the progress of the work process, that is, according to the change in work instructions.

[0052] Figure 9 is a flowchart showing the process for generating a 3D model. The work state detection system 1 generates 3D model data of the work object 2 for each work instruction (S31) (S32). The work state detection system 1 sets a line-of-sight reference point 51 in the generated 3D model data, converts it into a 3D model 50 for use with the MR goggles 4 (S33), and stores it in the storage unit 17 along with information identifying the work instruction (S34).

[0053] In a work site not shown in the diagram, work instructions such as "Please fix part B on top of part A" are given to worker 3 via a work instruction board or work instruction display. Each time the work instruction changes, that is, as the work process progresses, the shape of the work object 2 changes. However, the shape of the work object 2 changes in a predetermined manner according to the predetermined process. Therefore, the work state detection system 1 can, for example, create the necessary 3D model 50 in advance for each work instruction (for each work process) by coordinating with a CAD system that holds the design data of the work object 2 and a manual management system that manages the work instruction manual.

[0054] Figure 10 is a flowchart of the work status detection process. The work status detection system 1 determines whether the work instruction has changed (S41). If the work instruction has changed (S41: YES), the work status detection system 1 retrieves the 3D model 50 corresponding to the new work instruction from the storage unit 17 (S42) and sets it in the MR goggles 4.

[0055] The camera 404 of the MR goggles 4 photographs the work object 2 (S43), and when the work object 2 is spatially recognized (S44), the point 21 where the worker's line of sight 31 contacts the surface of the work object 2 (the surface of the 3D model 50) is detected, and the positional relationship between that contact point (viewpoint or intersection) 21 and the line of sight reference point 51 is calculated as relative coordinate values ​​(S45), and stored in the storage unit 17 as line of sight trajectory data (S46).

[0056] This embodiment, configured in this way, also produces the same effects as Embodiment 1. Furthermore, in this embodiment, the 3D model 50 is updated in conjunction with the work instructions (in conjunction with the progress of the work process), so the worker's line of sight can be accurately detected even when the shape of the work object 2 changes. [Examples]

[0057] Embodiment 3 will be explained using Figures 11 and 12. In this embodiment, the shape of the work object 2 is recognized by image analysis of image data of the work object 2, and a line of sight reference point 51 is set on the shape-recognized work object 2. In other words, in this embodiment, a 3D model 50 is not used to set the line of sight reference point 51 within the work object 2.

[0058] Figure 11 is a system configuration diagram of the work state detection system 1A. In the work state detection system 1A of this embodiment, image data of the work object 2 is input from the image acquisition unit 11 to the shape recognition unit 71, and the shape of the work object 2 is recognized. The line of sight reference point setting unit 14A sets the line of sight reference point 51 on the recognized work object 2.

[0059] Figure 12 is a flowchart of the line-of-sight reference point setting process. When the work state detection system 1A acquires image data of the work object 2 (S51), it recognizes the shape of the work object 2 (S52).

[0060] The work state detection system 1A determines whether the shape or posture of the work object 2 has changed (S53), and if either the shape or posture of the work object 2 has changed (S53:YES), it sets a line of sight reference point 51 on the work object 2 (S54).

[0061] The work state detection system 1A then converts the data of the 3D model with the line of sight reference point 51 set into data for use with the MR goggles 4 (S55) and stores it in the memory unit 17 (S56). The position of the point 21 where the worker's line of sight 31 contacts the surface of the work object 2 can be determined as the intersection point of the shape-recognized surface of the work object 2 and the line of sight 31.

[0062] This embodiment, configured in this way, achieves the same effects as Embodiment 1. Furthermore, in this embodiment, the line of sight reference point 51 can be set in accordance with changes in the shape and posture of the workpiece 2 without using a 3D model. Therefore, in this embodiment, since the line of sight reference point 51 can be set in conjunction with work instructions (in conjunction with the progress of the work process), the line of sight of the worker 3 can be accurately detected even for work in which the shape of the workpiece 2 changes, such as machining work. [Examples]

[0063] Embodiment 4 will be explained using Figure 13. In this embodiment, not only the movement of the worker's gaze 31 but also the movement of the worker's fingertips is detected, and the gaze trajectory data and fingertip trajectory data are output in association.

[0064] Figure 13 is a system configuration diagram of the work state detection system 1B. Compared to the work state detection system 1 shown in Figure 1, the work state detection system 1B further includes a fingertip trajectory data calculation unit 72.

[0065] The fingertip trajectory data calculation unit 72 detects the fingertips of the worker 3 in the image captured by the camera 404 of the MR goggles 4, calculates their movement trajectory, and stores it in the storage unit 17. The output unit 18 can output the work state analysis system 6 with the work image data, gaze trajectory data, and fingertip trajectory data in association.

[0066] This embodiment, configured in this way, also produces the same effects as Embodiment 1. Furthermore, in this embodiment, not only the movement of the worker's gaze during work but also the movement of their fingertips is detected simultaneously, allowing for a more accurate analysis of the worker's work state. [Examples]

[0067] Embodiment 5 will be explained using Figure 14. In this embodiment, the force applied to the fingertips of the worker 3 is detected by the pressure sensor 73 and stored in the storage unit 17 along with the fingertip trajectory data. Furthermore, in this embodiment, an external camera 74 is used in place of, or in conjunction with, the camera 404 of the MR goggles 4.

[0068] Figure 14 is a system configuration diagram showing the work state detection system 1C of this embodiment. Worker 3 is wearing work gloves with a built-in pressure sensor 73. An external camera 74 is provided on the outside of worker 3, capturing at least the upper body of worker 3 in its field of view.

[0069] The pressure detected by the pressure sensor 73 is acquired by the sensor data acquisition unit 75 and stored in the storage unit 17. Image data (moving image data) captured by the external camera 74 is input to the fingertip trajectory data calculation unit 77, and the movement trajectory of the fingertip in the image is calculated. The calculated fingertip trajectory data is stored in the storage unit 17. When calculating the position of the worker's fingertip using the external camera 74, for example, it is sufficient to calculate the relative coordinate value with respect to the center of the MR goggles 4.

[0070] This embodiment, configured in this way, also produces the same effects as Embodiment 1. Furthermore, in this embodiment, the worker's line of sight and fingertip trajectory, as well as the pressure applied to the worker's hand, can be detected and stored, allowing the work state analysis system 6 to analyze the work state more accurately. [Examples]

[0071] Example 6 will be explained using Figure 15. In this example, in addition to the configuration of Example 5, changes in the position of worker 3's head are also detected.

[0072] Figure 15 is a system configuration diagram showing the work state detection system 1D of this embodiment. In this embodiment, image data of work captured by the external camera 74 is input to the fingertip trajectory data calculation unit 77 and the head trajectory data calculation unit 78.

[0073] The head trajectory data calculation unit 78 calculates the position of the center of the head, for example, using the center of the worker's chest as a reference. Alternatively, the fingertip trajectory data and head trajectory data may be calculated using the gaze reference point 51 as a reference.

[0074] This embodiment, configured in this way, also produces the same effects as Embodiment 1. Furthermore, in this embodiment, the position of the worker's head during work can also be detected, and by correcting the gaze trajectory data based on changes in head position, the trajectory of the worker's gaze 31 can be calculated more accurately. In addition, if the worker 3 is looking at a location other than the work object 2 during work, the direction can be calculated and the object can be identified.

[0075] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations. Moreover, each embodiment can be combined as appropriate, as long as it is not obviously contradictory. [Explanation of Symbols]

[0076] 1,1A, 1B,1C,1D: Work state detection system, 2: Work object, 3: Worker, 4: MR goggles 4, 5: External system, 6: Work state analysis system, 11: Image acquisition unit, 12: 3D model acquisition unit, 13: Overlay unit, 14,14A: Eye-line reference point setting unit, 15: Eye-line detection unit, 16: Eye-line trajectory data calculation unit, 17: Memory unit, 21: Contact point, 31: Eye line, 50: 3D model, 51: Eye-line reference point, 71: Shape recognition unit, fingertip trajectory data calculation unit, 73: Pressure sensor, 74: External camera, 75: Sensor data acquisition unit 75, Work image acquisition unit 76, 77: Fingertip trajectory data calculation unit, 78: Head trajectory data calculation unit

Claims

1. A work state detection system that detects the line of sight when working on an object, A line of sight reference point setting unit sets a line of sight reference point for tracking the worker's gaze on the work object, A gaze detection unit that detects the worker's line of sight, A gaze tracking unit calculates relative coordinate values ​​from the gaze reference point of the point where the worker's gaze contacts the work object as gaze trajectory data based on the gaze detected by the gaze detection unit, and stores the calculated gaze trajectory data in a storage unit. Equipped with, The line of sight reference point setting unit resets the line of sight reference point in accordance with the progress of the work procedure for the work object. Work status detection system.

2. Furthermore, it is equipped with an output section, The output unit outputs the image data of the work being done on the work object, along with the saved eye-tracking data. The work status detection system according to claim 1.

3. The aforementioned line-of-sight reference point setting unit sets the line-of-sight reference point on the three-dimensional model of the work object, which is displayed superimposed on the real-world work object. The work state detection system according to claim 2.

4. The line-of-sight reference point setting unit recognizes the shape of the work object based on image data of the work object and sets the line-of-sight reference point on the shape-recognized work object. The work state detection system according to claim 2.

5. Furthermore, it is equipped with a fingertip trajectory detection unit that detects the trajectory of the worker's fingertips, The fingertip trajectory detection unit calculates the relative coordinate values ​​of the fingertip position from the line of sight reference point as fingertip trajectory data, and stores the calculated fingertip trajectory data in the storage unit. The work state detection system according to claim 4.

6. Furthermore, it includes a head trajectory detection unit for detecting the trajectory of the worker's head, The head trajectory detection unit calculates the relative coordinate values ​​of the head from the line of sight reference point as head trajectory data, and stores the calculated head trajectory data in the storage unit. The work state detection system according to claim 5.

7. The line-of-sight reference point setting unit sets the line-of-sight reference point to the center of the three-dimensional model. The work status detection system according to claim 3.

8. The line-of-sight reference point setting unit sets the line-of-sight reference point to the center of the work object. The work state detection system according to claim 4.

9. The aforementioned gaze detection unit is an augmented reality eyewear that captures the direction of the worker's gaze and overlays a virtual space onto the captured real world to provide to the worker, and is equipped with a function to detect gaze. The work status detection system according to claim 1.

10. A method for detecting the working state of an object using a working state detection system, The aforementioned work state detection system is: A line-of-sight reference point is set on the work object to track the worker's gaze, and further, the line-of-sight reference point is reset according to the progress of the work procedure on the work object. The worker's line of sight is detected, Based on the detected line of sight, the relative coordinate values ​​of the contact point where the worker's line of sight contacts the work object, from the line of sight reference point, are calculated as line of sight trajectory data. The calculated eye-tracking data is saved to the memory unit. Method for detecting the working state.

Images