Work skill assistance method, device, and program

Abstract

A work skill assistance method comprises: a step (001) for acquiring videos of a plurality of workers; a step (002) for analyzing changes in each part of each worker for each time on the basis of the video to generate time-series attitude position data; a step (003) for adjusting one time axis on the basis of the attitude position data of a first worker and the attitude position data of a second worker so that the attitude of both workers and the position of each part approximate each other; a step (004) for dividing the respective video for each unit work by using the adjustment of the time axis; a step (005) for comparing and evaluating the motion of the first worker and the motion of the second worker for each unit work of each; and a step (011) for reproducing and displaying the respective video after the time axis adjustment in parallel together with the result of the comparison evaluation.

Description

Work mastery support method, device, and program

[0001] The present invention relates to a technique for assisting a worker in acquiring proficiency in a repetitive task that the worker performs.

[0002] For example, when a beginner or inexperienced worker is engaged in a new task on a factory assembly line, if the worker can objectively compare his or her own movements with those of others (in most cases, those who are skilled in the task), the worker can easily recognize the differences between the two movements and efficiently progress through the training process.It is also convenient for those who instruct and educate inexperienced workers to be able to objectively compare the movements of multiple people.

[0003] Patent Document 1 discloses a technology for checking a running form, which is a repetitive movement, by filming a person running and analyzing the movements (positional changes) of the runner's left and right feet and hands through image processing of the video data, and displaying the result as a form confirmation screen on a display unit. In this technology, for a movement pattern in which the left and right feet and hands move back and forth, a movement with an average period is used as a reference movement pattern, and a movement that differs from this is used as a comparison movement pattern, and the two can be displayed side by side or superimposed. Furthermore, it is described that if there is a difference in period between the two, the display times of the two can be matched, for example, by interpolating or thinning frames of one of the videos during display.

[0004] However, in the running form targeted by Patent Document 1, for example, the time required for one reciprocating movement of the right hand is not different from the time required for one reciprocating movement of the left hand, and the same is true for the left and right feet. Therefore, comparison is possible simply by matching the overall period or display time of a single movement pattern. In contrast, in a worker's work that consists of a series of several unit tasks, such as picking up a nut from a parts box, grasping a tool, and tightening the nut onto an object, there is variation in the speed or speed of each unit task compared to others (e.g., an expert). Therefore, even if the overall display time is matched between the images of an expert and an unexperienced worker, the start and end timing of each unit task will not necessarily match, making it difficult to compare the two.

[0005] Japanese Patent Application Laid-Open No. 2017-229081

[0006] This invention is a method for assisting worker proficiency in work that includes a series of unit tasks and that supports the proficiency of workers, the method comprising: acquiring video of each of a plurality of workers performing the task; analyzing changes in each part of each worker's body over time based on each of the video; generating time-series posture and position data for each worker; adjusting the time axis of at least one of the posture and position data of the first worker and the second worker based on the posture and position data of the first worker and the posture and position data of the second worker so that the postures and positions of the two workers are similar to each other; adjusting the time axis of at least one of the video of the first worker and the video of the second worker in accordance with this time axis adjustment; dividing each video into unit tasks; comparing and evaluating the movements of the first worker and the second worker for each unit task; and playing back and displaying the time axis-adjusted videos in parallel together with the results of this comparison and evaluation.

[0007] In this way, by dividing the working movements of at least two workers into unit tasks and reproducing and displaying them in parallel, it becomes easy to compare them, and an unskilled worker who is in the process of becoming proficient can efficiently learn how his or her movements differ from those of other workers (for example, skilled workers).

[0008] 1 is a functional block diagram of a task proficiency assistance device according to an embodiment; a flowchart showing the processing flow of an embodiment; an explanatory diagram showing video of tasks performed by a first worker and a second worker in real time for each unit task; an explanatory diagram of video of tasks performed by a first and second worker after time-series registration processing; an explanatory diagram in which video of tasks performed by a second worker is divided into unit tasks using division position labels of the first worker; a first explanatory diagram showing an example of playback and display of video of the first and second workers; a second explanatory diagram showing an example of playback and display of video of the first and second workers; and a diagram showing an example of a display screen.

[0009] An example of the application of the present invention to task mastery support for workers engaged in assembly work on a factory assembly line is described below. The term "task" as used in this invention refers to repetitive movements performed on an assembly line or the like, and in particular refers to a task that includes a series of multiple unit tasks, rather than a simple repetitive motion such as "running." The task may be performed outdoors. The example task performed by a worker described below is the use of nuts to attach a small component, such as a bracket, to a stud bolt on a large component moving down the assembly line. The task in this example consists of a series of five unit tasks: "Picking up the bracket and nut," "Placing the bracket on the stud bolt and temporarily tightening the nut by hand," "Holding the tool," "Tightening the nut with the tool," and "Marking the nut after tightening." For example, the entire task takes approximately one to two minutes.

[0010] One embodiment aims to promote the second worker's proficiency or confirm the second worker's level of proficiency by comparatively displaying video of a first worker who serves as a reference and is proficient in the task, and video of a second worker who is in the process of becoming proficient.

[0011] 1 is a functional block diagram showing the configuration of a work proficiency assistance device according to one embodiment. The work proficiency assistance device is mainly composed of an information processing device 100, which is a so-called computer system, and an output unit 001, which is composed of various types of displays for reproducing and displaying video. The output unit 001 may also include audio output.

[0012] The information processing device 100 includes a program storage unit 120 that performs information processing according to a predetermined program, and a data storage unit 110 that stores and saves various data. The information processing device 100 is configured from a computer system installed in, for example, a factory building, but a part of the configuration may be installed externally as a cloud server or the like.

[0013] Although not shown, a camera (preferably a 3D camera) is provided at a position corresponding to each worker on the assembly line to capture video of the worker performing the work. For example, the camera is installed in front of the worker (on the assembly line side) and adjusted so that it captures at least the upper body, including both hands and face. Video data of each worker captured during work is sent to the information processing device 100 via an appropriate network, including wireless communication, and stored in the data storage unit 110.

[0014] The video of the workers acquired during work includes video of multiple workers, including at least reference video of a skilled worker corresponding to the first worker working, and video of a person in the process of becoming skilled corresponding to the second worker. The video data also includes RGB and monochrome information, depth information, etc. Furthermore, the reference video of at least one worker who will become the first worker is annotated in advance with labels indicating division positions for dividing the series of videos into five unit tasks, and is stored in the data storage unit 110 as annotation video data including the labels.

[0015] The program storage unit 120 performs motion analysis and comparison based on the video of the second worker who is in the process of becoming proficient and the annotation video data of the first worker who is proficient. The results are displayed on the output unit 001.

[0016] The program storage unit 120 includes a posture / position data acquisition unit 121 , a time-series position adjustment unit 122 , a work movement division unit 123 , a work movement comparison / evaluation unit 124 , and a playback image processing unit 125 .

[0017] The posture / position data acquisition unit 121 analyzes the video data of the second worker, who is the target for comparison, and identifies the worker's posture and the positions of each part of the body, such as the arms, by extracting points such as the worker's neck, shoulders, elbows, and wrists. The unit then analyzes changes in the worker's posture and the positions of each part of the body over time to generate time-series posture / position data. In one embodiment, for the video data of the first worker, which serves as the reference, the posture / position data acquisition unit 121 generates time-series posture / position data in parallel with the generation of annotation data. This posture / position data is stored in the data storage unit 110. Using this posture / position data, a skeletonized image of the worker's movements, such as that shown in FIG. 8 (described later), can be obtained.

[0018] The time series registration unit 122 adjusts the time axis of at least one of the posture and position data of the first worker and the second worker (in a preferred embodiment, the time series posture and position data of the target second worker) based on the time series posture and position data of the first worker as a reference and the time series posture and position data of the second worker as a comparison target, which are prepared in advance, so that the postures and positions of both workers are similar to each other. For example, by using an algorithm capable of time series matching such as dynamic time warping, the time axis of the posture and position data of the second worker is adjusted (in other words, registration in the time direction) so that the distance between features representing work movements between the two videos (more specifically, the individual frames constituting the videos) is minimized (this is called time series registration processing).

[0019] By dynamically changing the time axis of the time-series posture and position data of the second worker in this way, a certain movement (change in posture and position) of the second worker and a certain movement (change in posture and position) of the first worker that are similar to each other appear to occur at the same timing. In other words, if we consider it in units of frames that make up a video, the time axis of the posture and position data of the second worker is dynamically adjusted (i.e., so that it becomes partially longer or shorter) so that two frames that are most similar to each other (have the smallest feature distance) but that are at different times in real time appear to occur at the same timing.

[0020] In response to this adjustment of the time axis, the work action division unit 123 divides the video of the first worker and the video of the second worker after the time series alignment process into unit tasks. Specifically, since labels that separate each unit task are annotated in advance for the video of the first worker, the video of the second worker can be divided into unit tasks by adding similar labels to the time series positions corresponding to each label after the time series alignment process. Note that this division process may be performed on the original video in which the workers are realistically captured, but in one embodiment, it is performed on time series posture and position data that shows the workers in a skeletonized form.

[0021] The work movement comparison and evaluation unit 124 compares and evaluates the movements of the first worker and the second worker for each unit task. That is, for each divided unit task, the posture and position data of both workers arranged in the same time series by the time series alignment process described above is compared, and the discrepancy between the two is evaluated using an appropriate evaluation index. For example, the comparison and evaluation is performed using one or more appropriate evaluation indexes such as the distance between the movement positions, the distance between the feature values ​​obtained by analyzing the movements, the degree of abnormality in the work, the level of proficiency in the work, etc.

[0022] The playback image processing unit 125 generates image data to be displayed on the output unit 001, including the results of the comparative evaluation by the work movement comparison and evaluation unit 124. FIG. 8 shows an example of a display generated by the playback image processing unit 125 and displayed on the screen of the output unit 001. A skeletonized image 11 of the first worker after time series alignment processing is displayed on the left half of the screen, and a skeletonized image 12 of the second worker is displayed on the right half of the screen. In one example, videos showing the movements of both workers are displayed in a synchronized manner through time series alignment processing. Then, portions that diverge as a result of the comparative evaluation are appropriately highlighted by changing the color, increasing the brightness, encircling them with a circle, or the like. In the illustrated example, highlighting circles 13 and 14 are added to the hands of both workers. For example, highlighting may be achieved by flashing the entire screen of the second worker or temporarily changing the color during the period when the movements diverge. The time (elapsed time) from the start of the work, which is set to 0, may also be displayed on the screen of the output unit 001.

[0023] Next, the above-mentioned time-series positioning process and division process for each unit task will be further explained with reference to the explanatory diagrams of FIGS.

[0024] FIG. 3 is an explanatory diagram showing the length of video footage of actual work by a first worker (worker A in the figure) and a second worker (worker B in the figure) in the form of a bar graph with time on the horizontal axis. As mentioned above, the example work includes five unit tasks a, b, c, d, and e. While FIG. 3 separates the video into separate unit tasks for the sake of explanation, the actual video footage does not have separate unit task segments. The time required for each unit task (a, b, c, d, and e) is usually different. Furthermore, the time required for each unit task by the first worker is often different from the time required for each unit task by the second worker. Generally, the second worker, who is less skilled, takes longer. Furthermore, the extent to which the second worker takes longer than the first worker, who is more skilled, varies for each unit task. Furthermore, there may be workers who are poor at unit task c and workers who are poor at unit task d.

[0025] For this reason, even if a series of videos covering the entire work were played back and displayed side by side, it would be impossible to properly compare the two. Also, even if the playback speed of one of the videos were adjusted so that the start and end of the entire work coincided, there would be discrepancies along the way, making it impossible to properly compare the two.

[0026] FIG. 4 is an explanatory diagram of the videos (more specifically, the time-series posture and position data) of the first and second workers that have been subjected to time-series registration processing by the time-series registration unit 122. As described above, an algorithm capable of time-series matching, such as dynamic time warping, is used to adjust the time axis of the second worker's posture and position data so that the distance between the features representing the work actions between the two videos (more specifically, the individual frames constituting the videos) is minimized. In other words, the time axis of the second worker's time-series posture and position data is locally stretched or shrunk, so that frames representing the same actions exist on the same timeline. Note that for the sake of explanation, FIG. 4 shows the posture and position data of the second worker (worker B) divided into unit tasks. However, in reality, there are no unit task divisions. Meanwhile, as described above, the posture and position data of the first worker is annotated with labels for dividing the data into unit tasks.

[0027] 5 is an explanatory diagram illustrating the division of the second worker's video (specifically, posture and position data) into unit tasks by the task motion division unit 123 after time series registration processing. As shown in FIG. 4, the time series registration processing makes it appear as if frames showing the same motion exist on the same timeline, so the second worker's posture and position data can be divided using annotation data (labels) indicating the boundaries of each unit task of the first worker. At this stage in FIG. 5, the start and end timing of each unit task of the first worker and the start and end timing of each unit task of the second worker (when time is dynamically expanded or contracted) are consistent, and furthermore, within each unit task, the timing of specific movements, such as picking up a nut, are aligned with each other.

[0028] 6 is an explanatory diagram showing, in a bar graph with time on the horizontal axis, the playback period of the playback image of the first worker and the playback period of the playback image of the second worker, which are ultimately played back and displayed in parallel on the output unit 001. Each playback image is a skeletal image based on posture and position data, as shown in FIG. 8. In this example, since the actual required time for the second worker is longer, the image of the second worker is played back at a speed equivalent to real time. The playback image of the first worker is played back in slow motion so that the start and end timing of each unit task a, b, c, d, and e coincides with the playback image of the second worker.

[0029] In one example, the replay image processing unit 125 generates a replay image so that the posture and position data of both images, which have been time-series aligned as shown in Fig. 5, are replayed and displayed at an appropriate speed, thereby enabling the replay shown in Fig. 6.

[0030] In this example, when a second worker views the playback screen to practice, the second worker's (the worker's) movements and the movements of the first worker are played back in sync, allowing the second worker to compare and observe the movements during the unit work. For example, when the second worker picks up a nut with his right hand, the first worker, who serves as the reference, will also be picking up a nut with his right hand, allowing the second worker to compare and learn from the movement patterns, such as the orientation of the hands and the direction of hand movement.

[0031] Alternatively, the video of the second worker may be played back and displayed in real time, and the time axis of the video of the first worker may be dynamically adjusted so that the movements are synchronized with the video of the second worker. This also makes it possible to play back the video as shown in Figure 6.

[0032] 2 is a flowchart showing the processing flow of the above-mentioned task proficiency assistance device, which will be described below. First, in step 001, video data of a first worker, who serves as a reference, and video data of a second worker, who is an unskilled worker, are read from the data storage unit 110. In one embodiment, as described above, the video data of the first worker already includes labels indicating division positions as annotation data. Video data of three or more workers may be simultaneously imported.

[0033] In step 002, as described above, the video data read in step 001 is subjected to an analysis of changes in posture and position to generate time-series posture and position data. Here, processing is performed on both the video data of the first worker and the video data of the second worker, and the posture and position data of the first worker includes a label indicating the division position.

[0034] In step 003, as explained for the time series registration unit 122, the time series registration of the two pieces of attitude / position data is performed using an algorithm capable of time series matching such as dynamic time warping.

[0035] In the next step 004, as explained for the work action division unit 123, a label indicating the division position in the posture / position data of the first worker is used to assign a label indicating the division position to the posture / position data of the second worker that has been time-series aligned with each other, and the posture / position data of the second worker is divided into unit tasks.

[0036] Then, in step 005, the movements of the first worker and the second worker are compared and evaluated for each unit task. That is, for each divided unit task, the posture and position data of both workers that are arranged in the same time series by time series registration processing is compared, and the deviation between the two is evaluated using an appropriate evaluation index.

[0037] In step 006, using the comparison and evaluation results obtained in step 005, processing of the reproduced image is started to determine the portion to be highlighted in the output unit 001. The processing from step 007 onwards is repeatedly executed for each time (time after time-series alignment) from the initial start point of the video.

[0038] In step 007, it is determined whether the two images at a certain time are different enough to be highlighted based on the comparison and evaluation results. If the result is NO, the process returns to step 007 via step 010, and the determination of the two images at the next time is repeated.

[0039] If the determination in step 007 is YES, the process proceeds to step 008, where the portion where the movements are deviating is highlighted as described above. At the same time, the time (elapsed time) may be displayed.

[0040] In the next step 009, it is determined whether the processing has been completed to the end of the series of operations. If the result is NO, the process returns to step 007 via step 010, and the processing is repeated until the end. If the processing has been completed to the end, the process proceeds from step 009 to step 011, and the image is reproduced and displayed on the output unit 001.

[0041] Next, FIG. 7 shows a second example of a replay image generated by the replay image processing unit 125. Similar to FIG. 6, FIG. 7 is an explanatory diagram showing the replay period of the replay image of the first worker and the replay period of the replay image of the second worker as a bar graph with time on the horizontal axis. Each replay image is a skeleton image shown in FIG. 8 based on posture and position data. In this second example, for each unit task, the relatively long image of the second worker is replayed at a speed equivalent to real time. On the other hand, the relatively short image of the first worker is replayed at a speed equivalent to real time so that the start of each unit task is synchronized with the second worker. Since the image of the first worker is shorter for each unit task, the image of the first worker finishes replay display earlier for each unit task and waits.

[0042] When comparing inexperienced workers, some may be faster than others for each unit task. In such cases, the playback display of the other workers will wait until the video of the worker with the longest time for each unit task has finished.

[0043] According to this second embodiment, when a second worker views the playback screen to improve his / her own proficiency, he / she can learn how much slower his / her own movements are compared to the movements of the first worker who serves as a reference, where the delay occurs, etc. Since the start times of the unit tasks of the two workers are displayed together, it is easy to compare the two.

[0044] Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment and various modifications are possible.

[0045] For example, in the above embodiment, the video of the first worker is described as having been previously labeled as annotation data indicating the division position for each unit task, but the division process may be performed in other ways. For example, it is possible to acquire videos of multiple workers and determine the division positions based on the characteristic transition patterns of their movements in a series of unit tasks. Furthermore, division may be triggered by external signals acquired in connection with the execution of unit tasks, such as signals from sensors that detect the position of hands, signals acquired with the operation of tools, or signals such as vibrations generated in a specific process.

[0046] It is also possible to compare three or more workers, with any one of them being the first worker.

Claims

1. A work proficiency support method that assists workers in mastering their tasks, which include a series of unit tasks performed in sequence, We acquire video footage of multiple workers, including a first worker who is a skilled reference and a second worker who is in the process of learning the skill, performing their respective tasks. Based on each video, the changes in each part of each worker's body are analyzed time by time, and time-series posture and position data is generated for each worker. The posture and position data of the first worker is annotated with labels indicating the division point for each unit of work, and saved as reference data. Based on the posture and position data of the first worker and the posture and position data of the second worker, the time axis of at least one of the posture and position data of the first worker and the posture and position data of the second worker is adjusted so that the postures and positions of each part of both workers approximate each other. In response to this adjustment of the time axis, the posture and position data of the second worker is divided for each unit of work, along with the time-series position of the above label in the posture and position data of the first worker. For each unit task, the actions of the first worker and the actions of the second worker are compared and evaluated. Along with the results of this comparative evaluation, the time-axis adjusted versions of each video will be played and displayed in parallel. Methods for supporting work proficiency.

2. The video of the second worker is played back and displayed in real time, and the time axis of the video of the first worker is dynamically adjusted and played back so that the movements are synchronized with the second worker's. The method for supporting work proficiency according to claim 1.

3. (delete)

4. (delete)

5. (delete)

6. When two pieces of video footage, one of the first worker and the other of the second worker, are divided into the same unit of work, the video with the relatively longer real-time duration is played back at a speed equivalent to real-time, while the other video is played back in slow motion. The method for supporting work proficiency according to claim 1.

7. Between the video of the first worker and the video of the second worker, which are divided into the same unit of work, the video with the relatively longer real-time duration is played back and displayed at a speed equivalent to real time. The other video simultaneously begins playback and display at a speed equivalent to real time, and at the same time, it finishes playback and displays first and waits. The method for supporting work proficiency according to claim 1.

8. This applies to three or more workers, and any worker is selected as the first worker. The method for supporting work proficiency according to claim 1.

9. Comparative evaluation is performed using one or more evaluation indicators, such as the distance of the movement position, the distance of the feature quantities analyzed from the movement, the degree of abnormality of the work, and the degree of proficiency in the work. The method for supporting work proficiency according to claim 1.

10. When playing back and displaying each video in parallel after adjusting the time axis, highlight the parts of the worker's body or the time periods where there is a large discrepancy between the actions of the first worker and the actions of the second worker. The method for supporting work proficiency according to claim 1.

11. A work proficiency support device that assists workers in mastering their tasks, which include a series of unit tasks performed by the worker, Based on video footage of multiple workers performing tasks, including a first worker who serves as a skilled reference and a second worker who is in the process of learning, the posture and position data acquisition unit analyzes the changes in each part of each worker's body over time and generates time-series posture and position data for each worker. A data storage unit that annotates the posture and position data of the first worker with labels indicating the division position for each unit of work, and stores it as reference data. A time-series alignment unit adjusts the time axis of at least one of the posture and position data of the first worker and the second worker so that the postures and positions of each part of both workers approximate each other, based on the posture and position data of the first worker and the posture and position data of the second worker. In response to this adjustment of the time axis, the work motion division unit divides the posture and position data of the second worker according to the time-series position of the above label in the posture and position data of the first worker, for each unit of work. A work motion comparison and evaluation unit compares and evaluates the actions of the first worker and the actions of the second worker for each unit of work, Along with the results of this comparative evaluation, the output section displays each video in parallel after time axis adjustment. A work proficiency support device equipped with the following.

12. A work proficiency support program that causes a computer system to execute the work proficiency support method described in claim 1.