Work training support system and work training support method

Abstract

To provide a technique capable of efficiently increasing the proficiency of a trainee.SOLUTION: A work training support system includes: a detector that detects the movement of a tool during a first period in which an instructor performs work using the tool; a generator that generates, using a result of detecting the movement of the tool during the first period, a first image representing the movement of the tool during the first period; and a display that is a transmission type display worn on a head of a trainee and displays the first image in a field of view of the trainee.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a work training support system and a work training support method.

Background Art

[0002] There is known a technique of estimating how a skilled worker would act in response to an event occurring during work using a model generated by machine learning (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Since it is not easy to create a manual that can accurately convey the skills of a skilled worker to a trainer, trial and error is required to improve the proficiency of the trainer to the same level as that of a skilled worker. Therefore, a technique that can efficiently improve the proficiency of a trainer is desired.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to the first aspect of the present disclosure, a work training support system is provided. The work training support system includes a detection unit that detects the movement of a tool during a first period, which is a period in which an instructor performs work using the tool, a generation unit that generates a first video representing the movement of the tool during the first period using the detection result of the movement of the tool during the first period, and a transmissive display unit that is worn on the head of a trainee and displays the first video in the field of view of the trainee. This type of work training support system allows trainees to perform tasks while viewing both the scene seen through the display unit and the first image displayed on the unit. Therefore, it is possible to efficiently improve the trainee's skill level. (2) In the above-described work training support system, the generation unit may generate the first video representing the movement of the tool during the first period and the trajectory of the movement of the tool during the first period. With this type of work training support system, trainees can perform tasks while referring to the trajectory shown in the first video. (3) In the above-described work training support system, the detection unit detects the movement of the tool during a second period, which is the period during which the trainee performs the work using the tool; the generation unit generates a second video representing the movement of the tool during the second period using the detection results of the movement of the tool during the second period; and the display unit may display the first video and the second video superimposed on each other after the second period. With this type of work training support system, trainees can compare the movements of the tool performed by the instructor with the movements of the tool performed by the trainee, using the first and second images displayed on the display unit. (4) In the above-described work training support system, the generation unit may generate a first video representing the movement of the tool during the first period and the trajectory of the tool's movement during the first period, wherein at least one of the thickness and color of each part of the trajectory is represented according to a physical quantity relating to the tool's movement, and a second video representing the movement of the tool during the second period and the trajectory of the tool's movement during the second period, wherein at least one of the thickness and color of each part of the trajectory is made different according to a physical quantity relating to the tool's movement. This type of work training support system makes it easy for trainees to recognize the difference between the tool movements performed by the instructor and those performed by the trainee. (5) In the above-described work training support system, the detection unit detects the movement of the tool during a second period, which is the period during which the trainee performs the work using the tool; the generation unit generates a third image in which the physical quantities relating to the movement of the tool during the first period and the physical quantities relating to the movement of the tool during the second period are shown side by side, using the detection results of the movement of the tool during the first period and the detection results of the movement of the tool during the second period; and the display unit may display the first image and the third image during the second period. According to this type of work training support system, trainees can perform tasks while referring to the physical quantities related to the movement of the tool during the first period and the physical quantities related to the movement of the tool during the second period, which are shown in the third video. (6) A second embodiment of the present disclosure provides a work training support method. This work training support method detects the movement of a tool during a first period, which is the period during which an instructor performs work using the tool; generates a first video representing the movement of the tool during the first period using the detection results of the tool's movement during the first period; and displays the first video in the trainee's field of view using a transparent display unit attached to the trainee's head during a second period, which is the period during which a trainee performs work using the tool. This type of work training support method allows trainees to perform tasks while viewing both the scene seen through the display unit and the first image displayed on the unit. Therefore, it is possible to efficiently improve the trainee's skill level. This disclosure can also be implemented in various forms other than work training support systems and work training support methods. For example, it can be implemented in the form of a work training support program. [Brief explanation of the drawing]

[0007] [Figure 1] A schematic diagram illustrating the configuration of a work training support system. [Figure 2] A flowchart illustrating the content of work training support methods. [Figure 3] An explanatory diagram showing the first image displayed on the display device. [Figure 4] An explanatory diagram showing the display of the first and second images on a display device. [Figure 5] An explanatory diagram showing the detailed trajectory of the tools in the first and second videos. [Modes for carrying out the invention]

[0008] A. First Embodiment: Figure 1 is a schematic diagram illustrating the configuration of the work training support system 10 in the first embodiment. The work training support system 10 is used for training in work using tool TL. In this embodiment, tool TL is a seal gun that sprays sealant, and the work is a sealant application operation in which an operator applies sealant to a workpiece WK using the seal gun. In the following description, an operator who demonstrates the work will be called an instructor, and an operator who receives training in the work will be called a trainee. When there is no particular distinction between instructor and trainee, they will simply be called operators.

[0009] The work training support system 10 includes a camera 100, a sensor 150, an information processing device 200, and a display device 300. The camera 100, sensor 150, and display device 300 are connected to the information processing device 200 by wired or wireless communication.

[0010] Camera 100 and sensor 150 detect the movement of the tool TL during operation. Camera 100 is fixed in a position that allows it to capture the work in progress. Camera 100 detects the position and orientation of the tool TL from its movement using motion capture technology. In this embodiment, camera 100 detects the position and orientation of the tool TL by reading a marker MK fixed to the tool TL. Sensor 150 is fixed to the tool TL. Sensor 150 detects predetermined movements of the tool TL other than its position and orientation. In this embodiment, the tool TL, a seal gun, comprises a lever and a nozzle, and sprays sealant from the nozzle at a spray volume corresponding to the amount the lever is squeezed. Sensor 150 includes a position sensor for detecting the position of the lever, in other words, the amount the lever is squeezed, and a pressure sensor for detecting the pressure applied to the nozzle when the nozzle is pressed against the workpiece WK. The detection results regarding the movement of the tool TL are transmitted from camera 100 and sensor 150 to information processing device 200. Camera 100 and sensor 150 are sometimes referred to as the detection unit.

[0011] The information processing device 200 is a computer comprising a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, memory 202, and input / output interface 203 are connected via the internal bus 204 to enable bidirectional communication. The memory 202 stores a work training support program PG and 3D CAD data DT of the tool TL. By executing the work training support program PG, the processor 201 functions as a detection control unit 210, a generation unit 220, and a display control unit 230. The detection control unit 210 acquires detection results of the movement of the tool TL during work from the camera 100 and sensor 150. The generation unit 220 uses the 3D CAD data DT and the detection results of the movement of the tool TL to generate an image that reproduces the movement of the tool TL during work using computer graphics technology. The display control unit 230 displays the image generated by the generation unit 220 on the display device 300.

[0012] The display device 300 is worn on the trainee's head. The display device 300 is a transparent head-mounted display that uses augmented reality technology to overlay images of the virtual space onto the real-world scene. Specifically, the display device 300 displays images generated by the generation unit 220 within the trainee's field of view. In this embodiment, the display device 300 is equipped with a camera and various sensors, and performs alignment between the real space and the virtual space by detecting a reference point object in the real space. The display device 300 is sometimes referred to as the display unit.

[0013] Figure 2 is a flowchart showing the contents of the work training support method using the work training support system 10. Figure 3 is an explanatory diagram showing the display of the first video M1 on the display device 300. Figure 4 is an explanatory diagram showing the display of the display device 300 showing both the first video M1 and the second video M2. When the work training support method is started, in step S110, the instructor performs the work using the tool TL. In the following description, the period during which the instructor performs the work using the tool TL is referred to as the first period. The detection control unit 210 uses the camera 100 and the sensor 150 to detect the movement of the tool TL during the first period. The detection control unit 210 stores the detection results of the movement of the tool TL during the first period in the memory 202.

[0014] In step S120, the generation unit 220 uses the 3D CAD data DT stored in the memory 202 and the detection results of the tool TL movement during the first period to generate a first video M1 representing the movement of the tool TL during the first period, that is, the movement of the tool TL due to the instructor's operation. In this embodiment, the first video M1 shows not only the movement of the tool TL during the first period, but also the trajectory L1 of the tool TL's movement during the first period. Specifically, the trajectory L1 of the tool TL's movement is the trajectory of the nozzle tip's movement.

[0015] In step S130, the trainee performs an operation using the tool TL while wearing the display device 300. The content of the operation performed by the trainee in step S130 is the same as the content of the operation performed by the instructor in step S110. In the following description, the period during which the trainee performs an operation using the tool TL is referred to as the second period. The display control unit 230 causes the display device 300 to display the first video M1 from the start to the end of the second period. As shown in FIG. 3, to the trainee, the first video M1 displayed on the display device 300 overlaps with the scene of the real space visible through the display device 300. Therefore, the trainee can perform the operation while viewing the scene of the real space and the first video M1. By moving the tool TL in the real space so as to follow the movement of the tool TL represented in the first video M1, the trainee can imitate the actions of the instructor.

[0016] In step S140, the detection control unit 210 uses the camera 100 and sensor 150 to detect the movement of the tool TL during the second period. The processing in step S140 is performed concurrently with the processing in step S130. The detection control unit 210 stores the detection results of the movement of the tool TL during the second period in the memory 202. The generation unit 220 generates a third video M3 that shows the measured values ​​of physical quantities measured using the camera 100 and sensor 150 during the second period and the measured values ​​of physical quantities measured using the camera 100 and sensor 150 during the first period side by side. The display control unit 230 displays the third video M3 in addition to the first video M1 on the display device 300 from the start to the end of the second period. The measured values ​​during the second period are shown in the third video M3 in real time. As shown in Figure 3, in this embodiment, the third video M3 shows the nozzle pressing pressure, the lever grip amount, the movement speed of the tool TL, and the rotation angle of the tool TL. The nozzle pressing pressure is measured using a pressure sensor, the lever grip amount is measured using a position sensor, and the tool TL movement speed and rotation angle are measured using camera 100. The third video M3 further displays the pass / fail judgment result of the trainee's work. The generation unit 220 determines the pass / fail judgment result of the trainee's work and displays the pass / fail judgment result in the third video M3. For example, the generation unit 220 determines that the work is a pass if the difference between the measured value during the first period and the measured value during the second period is within a predetermined range, and determines that it is a fail if the difference between the measured value during the first period and the measured value during the second period is not within a predetermined range.

[0017] In step S150, the generation unit 220 generates a second video M2 representing the movement of the tool TL during the second period, that is, the movement of the tool TL by the operation of the trainer, using the 3D CAD data DT stored in the memory 202 and the detection result of the movement of the tool TL during the second period. In the present embodiment, in addition to the movement of the tool TL during the second period, the trajectory L2 of the movement of the tool TL during the second period is represented in the second video M2. Specifically, the trajectory L2 of the movement of the tool TL is the trajectory of the movement of the nozzle tip. In order to make it easier to distinguish between the first video M1 and the second video M2, the generation unit 220 represents the color of the tool TL and the color of the trajectory L2 in the second video M2 with colors different from the color of the tool TL and the color of the trajectory L1 in the first video M1.

[0018] In step S160, the display control unit 230 causes the display device 300 to display the first video M1 and the second video M2 superimposed on each other after the second period. Displaying the first video M1 and the second video M2 superimposed on each other includes not only displaying the first video M1 and the second video M2 at the same position but also displaying the first video M1 and the second video M2 adjacent to each other. In the present embodiment, the display control unit 230 causes the display device 300 to display a third video M3 in addition to the first video M1 and the second video M2. As shown in FIG. 4, the trainer can analyze the difference between the movement of the tool TL by the instructor and the movement of the tool TL by the trainer using the first video M1, the second video M2, and the third video M3 displayed on the display device 300. Then, this work training support method ends.

[0019] Figure 5 is an explanatory diagram showing the details of the tool TL trajectories L1 and L2 in the first video M1 and the second video M2. In this embodiment, the thickness and color of each part of the trajectories L1 and L2 are represented by thickness and color corresponding to the magnitude of the measurement value of the sensor 150. Specifically, the thickness of each part of the trajectories L1 and L2 is represented by thickness corresponding to the amount of lever grip, and the greater the amount of lever grip, the thicker the trajectories L1 and L2 become. Furthermore, spherical markers MB are shown on the trajectories L1 and L2 at the point where the lever gripping operation begins and the point where the lever gripping operation ends, and cubic markers MC are shown at the point where the amount of lever grip changes abruptly. The color of each part of the trajectories L1 and L2 is represented by color corresponding to the magnitude of the nozzle pressing pressure, and the greater the nozzle pressing pressure, the lower the brightness of the color. That is, the greater the nozzle pressing pressure, the darker the color of the trajectories L1 and L2 becomes.

[0020] In the example shown in Figure 5, the instructor prevents the sealant from overflowing from the end by releasing the lever abruptly just before the end of the sealant application area and then continuing to press the nozzle down to the end. In contrast, the trainee starts releasing the lever earlier than the instructor, but completely releases the lever at the end, making it easier for the sealant to overflow from the end. The differences between the instructor's actions and the trainee's actions described above are represented by the differences in the thickness and color of the trajectories L1 and L2, and the differences in the positions of the markers MB and MC displayed along with the trajectories L1 and L2. Therefore, the trainee can easily recognize the differences between the instructor's actions and the trainee's actions by comparing the trajectories L1 and L2 with the markers MB and MC.

[0021] As described above, the work training support system 10 in this embodiment displays the first video M1 on the display device 300 during the second period. This allows the trainee to perform work training while viewing both the real-world scene visible through the display device 300 and the first video M1 displayed on the display device 300. Since the first video M1 shows the movement of the tool TL by the instructor, the trainee can experience the instructor's work method by moving the tool TL in the same way as the instructor. Therefore, the trainee's skill level can be efficiently increased.

[0022] Furthermore, in this embodiment, the first video M1 shows the trajectory L1 of the tool TL's movement during the first period, so the trainee can perform the work while referring to the trajectory L1 shown in the first video M1.

[0023] Furthermore, in this embodiment, the third video M3 displays both the real-time measurement values ​​measured using the camera 100 and sensor 150 during the second period and the measurement values ​​measured using the camera 100 and sensor 150 during the first period side by side. Therefore, trainees can perform their tasks while referring to the measurement values ​​from the second period and the measurement values ​​from the first period displayed in the third video M3.

[0024] Furthermore, in this embodiment, after the trainee has finished their work, the first video M1 and the second video M2 are displayed on the display device 300 superimposed on each other. Therefore, by comparing the first video M1 and the second video M2 displayed on the display device 300, the trainee can analyze the difference between the movement of the tool TL by the instructor and the movement of the tool TL by the trainee.

[0025] Furthermore, in this embodiment, the first video M1 shows the trajectory L1 of the tool TL's movement during the first period, and the second video M2 shows the trajectory L2 of the tool TL's movement during the second period. In addition, the thickness and color of each part of the trajectories L1 and L2 are represented by thickness and color corresponding to the magnitude of the measured value measured using the sensor 150. Therefore, the trainee can easily recognize the difference between the movement of the tool TL by the instructor and the movement of the tool TL by the trainee.

[0026] B. Other embodiments: (B1) In the first embodiment of the work training support system 10 described above, the work training support system 10 is equipped with a sensor 150. In contrast, the work training support system 10 does not need to be equipped with a sensor 150.

[0027] (B2) In the work training support system 10 of the first embodiment described above, the first video M1 shows the trajectory L1 of the tool TL during the first period, and the second video M2 shows the trajectory L2 of the tool TL during the second period. In contrast, the first video M1 does not have to show the trajectory L1 of the tool TL during the first period, and the second video M2 does not have to show the trajectory L2 of the tool TL during the second period.

[0028] (B3) In the work training support system 10 of the first embodiment described above, the thickness of each part of the trajectory L1 and L2 is represented by a thickness corresponding to the measured value measured using the sensor 150, and the color of each part of the trajectory L1 and L2 is represented by a color corresponding to the measured value measured using the sensor 150. In contrast, at least one of the thickness and color of each part of the trajectory L1 and L2 may be constant regardless of the measured value measured using the sensor 150.

[0029] (B4) In the work training support system 10 of the first embodiment described above, the third video M3 is displayed on the display device 300 during the second period. In contrast, the third video M3 does not have to be displayed on the display device 300 during the second period.

[0030] (B5) In the work training support system 10 of the first embodiment described above, the processes from step S140 to step S160 of the work training support method do not need to be executed.

[0031] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]

[0032] 10...Work training support system, 100...Camera, 150...Sensor, 200...Information processing device, 201...Processor, 202...Memory, 203...Input / output interface, 204...Internal bus, 210...Detection control unit, 220...Generation unit, 230...Display control unit, 300...Display device, DT...3D CAD data, L1...Trajectory, L2...Trajectory, M1...First image, M2...Second image, M3...Third image, MB...Marker, MC...Marker, MK...Marker, PG...Work training support program, TL...Tool, WK...Work

Claims

1. A work training support system, A detection unit that detects the movement of the tool, including its position and orientation, during a first period in which the supervisor performs work using the tool, A generation unit generates a first video representing the movement of the tool during the first period and the trajectory of the tool's movement during the first period, using the detection results of the tool's movement during the first period. A transparent display unit worn on the head of a trainee, the display unit which displays the first image within the trainee's field of vision, Equipped with, The aforementioned tool is a seal gun comprising a lever and a nozzle, which sprays a sealant from the nozzle in a quantity corresponding to the amount the lever is squeezed. The aforementioned operation is a sealant application operation in which the sealant is applied to the workpiece using the seal gun, The detection unit detects, in addition to the position and orientation of the tool during the first period, the amount the lever is gripped during the first period and the pressure applied to the nozzle during the first period. The trajectory of the movement of the tool is the trajectory of the tip of the nozzle. The generating unit is The thickness of the trajectory in the first video is varied according to the amount the lever is gripped. The color of the trajectory in the first image is changed according to the pressure applied from the workpiece to the nozzle when the nozzle is pressed against the workpiece. A work training support system that adds markers to the trajectory in the first video, representing at least one of the following points: the point where the operation of gripping the lever begins, the point where the operation of gripping the lever ends, and the point where the amount of grip on the lever changes by a predetermined amount or more in a predetermined time.

2. A work training support system according to claim 1, The detection unit detects the movement of the tool, including its position and orientation, during a second period, which is the period during which the trainee performs the task using the tool. The generation unit generates a second video representing the movement of the tool during the second period, using the detection results of the tool's movement during the second period. The display unit is a work training support system that displays the first image and the second image superimposed on each other after the second period.

3. A work training support system according to claim 2, The detection unit detects, in addition to the position and orientation of the tool during the second period, the amount the lever is gripped during the second period and the pressure applied to the nozzle during the second period. The generating unit is A second video is generated that further includes the trajectory of the tool's movement during the second period. The thickness of the trajectory in the second image is varied according to the amount the lever is gripped. The color of the trajectory in the second image is changed according to the pressure applied from the workpiece to the nozzle when the nozzle is pressed against the workpiece. A work training support system that adds markers to the trajectory in the second video, representing at least one of the following points: the point where the operation of gripping the lever begins, the point where the operation of gripping the lever ends, and the point where the amount of grip on the lever changes by a predetermined amount or more in a predetermined time.

4. A work training support system according to claim 1, The detection unit detects the movement of the tool during the second period, which is the period during which the trainee performs the work using the tool. The generation unit uses the detection results of the tool's movement during the first period and the detection results of the tool's movement during the second period to generate a third image in which the physical quantities relating to the tool's movement during the first period and the physical quantities relating to the tool's movement during the second period are shown side by side. The display unit is a work training support system that displays the first video and the third video during the second period.

5. A method for supporting work training, The movement of the tool, including its position and orientation, is detected during the first period, which is the period during which the supervisor performs work using the tool. Using the detection results of the tool's movement during the first period, a first video is generated that represents the tool's movement during the first period and the trajectory of the tool's movement during the first period. During the second period, which is the period in which the trainee performs the work using the tool, the first image is displayed in the trainee's field of view using a transparent display unit attached to the trainee's head. The aforementioned tool is a seal gun comprising a lever and a nozzle, which sprays a sealant from the nozzle in a quantity corresponding to the amount the lever is squeezed. The aforementioned operation is a sealant application operation in which the sealant is applied to the workpiece using the seal gun, In detecting the movement of the tool, in addition to the position and orientation of the tool during the first period, the amount the lever is gripped during the first period and the pressure applied to the nozzle during the first period are detected. The trajectory of the movement of the tool is the trajectory of the tip of the nozzle. In generating the first video, The thickness of the trajectory in the first video is varied according to the amount the lever is gripped. The color of the trajectory in the first image is changed according to the pressure applied from the workpiece to the nozzle when the nozzle is pressed against the workpiece. A work training support method, comprising adding markers to the trajectory in the first video, representing at least one of the following points: the point where the operation of gripping the lever begins, the point where the operation of gripping the lever ends, and the point where the amount of grip on the lever changes by a predetermined amount or more in a predetermined time.

Images