Motion analysis device, motion analysis method, and motion analysis program

Abstract

The motion analysis device (100) includes a matching unit (130) that extracts, as a third similar trajectory, a trajectory of a target body part of a target worker from a trajectory indicated by reference trajectory information (190) indicating the trajectory of a body part corresponding to the target body part when the target worker is performing the work to be performed, which is a trajectory of a target body part of a target worker in a subsequent period that is a period from a target time that is a time corresponding to a target frame included in an input video showing the target worker performing work to a time later than the target time, and which is a trajectory derived based on the input video.

Description

Technical Field

[0001] The present disclosure relates to an operation analysis device, an operation analysis method, and an operation analysis program.

Background Art

[0002] In the industrial field, the measurement of the cycle time of workers and the analysis of work contents were manually performed. Patent Document 1 proposes a method for automating the measurement of the cycle time of workers and the analysis of work contents in order to reduce the load of the measurement of the cycle time of workers and the analysis of work contents. In this method, the time-series transition of the posture information obtained from the video is defined as an operation trajectory, and the work content is specified by comparing it with a previously learned trajectory. In this method, by using the posture information based on the data acquired by the camera and the sensor, no burden is imposed on the worker who is the analysis target.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, in order to analyze the working time and working procedure of a line worker in a factory, the trajectory of joints before and after the analysis target time including the analysis target time is matched with the trajectory of joints in the registered steady work, thereby detecting the working time and working procedure. However, in actual work, operations different from those in steady work (such as work omission or non-steady work) may occur. When an operation different from that in steady work occurs, since the trajectory of the joints of the worker is different from the registered trajectory, there is a possibility of erroneously matching the trajectory of the joints. However, Patent Document 1 does not disclose a technique for reducing the possibility of erroneously matching the trajectory of joints. This disclosure aims to reduce the possibility of incorrect matching of the target body trajectory in a technology that matches the trajectory of a target body part of a target worker with the trajectory of a target body part during routine work. [Means for solving the problem]

[0005] The motion analysis device related to this disclosure is A matching unit performs a third matching process, which extracts trajectories similar to a third target trajectory, which is a trajectory derived based on the input video, from the trajectories indicated by reference trajectory information that shows the trajectory of the part corresponding to the target part when the target worker is performing the work to be performed, during a subsequent period from a target time, which is the time corresponding to the target frame contained in the input video showing the target worker performing the work, to a time in the future than the target time. It is equipped with. [Effects of the Invention]

[0006] According to this disclosure, the matching unit performs matching for a period after the target frame. Therefore, according to this disclosure, even if an action different from the action in a normal operation occurs in the period before and after the target frame, matching can be performed appropriately if a normal operation is performed in the latter half of that period. Accordingly, according to this disclosure, in a technology that matches the trajectory of a target part of a target worker with the trajectory of a target part in a normal operation, the possibility of incorrectly matching the trajectory of the target part can be reduced. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram showing an example configuration of the motion analysis device 100 according to Embodiment 1. [Figure 2] A diagram illustrating the processing of the motion analysis device 100 according to Embodiment 1. [Figure 3] A diagram illustrating the processing of the matching unit 130 according to Embodiment 1. [Figure 4] A diagram showing an example of the hardware configuration of the motion analysis device 100 according to Embodiment 1. [Figure 5] A flowchart illustrating the operation of the motion analysis device 100 according to Embodiment 1. [Figure 6] This figure illustrates the effects of the motion analysis device 100 according to Embodiment 1, where (a) is a figure illustrating matching in the preceding and succeeding periods, and (b) is a figure illustrating matching in the preceding period. [Figure 7] This figure shows an example of the hardware configuration of a motion analysis device 100 according to a modified example of Embodiment 1. [Figure 8] A flowchart illustrating the operation of the motion analysis device 100 according to Embodiment 2. [Figure 9] A flowchart illustrating the operation of the motion analysis device 100 according to Embodiment 3. [Modes for carrying out the invention]

[0008] In the description and drawings of the embodiments, the same elements and corresponding elements are denoted by the same reference numeral. The descriptions of elements denoted by the same reference numeral are omitted or simplified as appropriate. The arrows in the figures mainly indicate the flow of data or processing. Also, "part" may be interpreted as "circuit," "process," "procedure," "processing," or "circuitry" as appropriate.

[0009] Embodiment 1. This embodiment will now be described in detail with reference to the drawings.

[0010] ***Explanation of the structure*** Figure 1 shows an example configuration of the motion analysis device 100 according to this embodiment. As shown in this figure, the motion analysis device 100 includes a skeleton extraction unit 110, a trajectory extraction unit 120, a matching unit 130, and a similarity comparison unit 140. The motion analysis device 100 also stores reference trajectory information 190.

[0011] The skeleton extraction unit 110 acquires input video from an external source, extracts the skeleton of the target worker in each frame included in the acquired input video, and generates skeleton information indicating the extracted skeleton in each frame. In this case, as a specific example, the skeleton extraction unit 110 identifies the joint positions of the target worker in each frame included in the acquired input video, and generates information indicating the joint positions identified in each frame as skeleton information. The input video is video output by a camera, showing the target worker performing work. The video is typically a moving image. The target worker is expected to be performing routine work. The input video may also be information output by a 3D (3-Dimensions) sensor, showing the movements of the target worker. The data acquired by the 3D sensor is sometimes called video. The information showing the posture of the target worker acquired by the 3D sensor at each time point is sometimes called a frame.

[0012] The trajectory extraction unit 120 extracts three trajectory patterns based on the target frame from the skeletal information generated by the skeletal extraction unit 110 for the target part of the target worker, and generates trajectory information showing each extracted pattern. The target part is, in specific examples, a joint or a part of the skeleton. The target frame is a frame selected from the frames included in the input video. The three trajectory patterns for the target part, based on the target frame, consist of the trajectory of the target part in the period before and after the target frame, the trajectory of the target part in the period prior to the target frame, and the trajectory of the target part in the period after the target frame. The period before and after the target frame is a continuous period from a time before the target time to a time in the future of the target time. The period prior to the target frame is a continuous period from a time before the target time to the target time. The period after the target frame is a continuous period from the target time to a time in the future of the target time. The target frame is the frame that recognizes the work content of the target worker. The target time is the time corresponding to the target frame, and specifically, it is the time when the target frame was captured by the camera, or the time when the sensor observed the posture of the target worker. The start and end times of the before and after periods may be determined in any way. The start time of the period prior to the target time may be determined in any way. The end time of the period after the target time may be determined in any way. The first half of the trajectory of the target body in the preceding and succeeding periods may be considered the trajectory of the target body in the preceding period, and the second half of the trajectory of the target body in the succeeding period may be considered the trajectory of the target body in the succeeding period. Alternatively, the trajectory obtained by combining the trajectory of the target body in the preceding period and the trajectory of the target body in the succeeding period may be considered the trajectory of the target body in the preceding and succeeding periods. The trajectory of the target part in the previous period is also called the first target trajectory. The trajectory of the target part in the preceding and succeeding periods is also called the second target trajectory. The trajectory of the target part in the subsequent period is also called the third target trajectory. The first target trajectory is the trajectory of the target part in the previous period, and is a trajectory derived based on the input video. The second target trajectory is the trajectory of the target part in the preceding and succeeding periods, and is a trajectory derived based on the input video. The third target trajectory is the trajectory of the target part, and is a trajectory derived based on the input video. The trajectory extraction unit 120 extracts the first target trajectory, the second target trajectory, and the third target trajectory from the skeletal information generated by the skeletal extraction unit 110. Note that there may be multiple target parts. If there are multiple target parts, the trajectory extraction unit 120 extracts three trajectory patterns from the skeletal information for each target part.

[0013] The matching unit 130 performs a first matching process, a second matching process, and a third matching process, respectively. The first matching process extracts trajectories similar to the first target trajectory from the trajectories indicated by the reference trajectory information 190 as first similar trajectories. The second matching process extracts trajectories similar to the second target trajectory from the trajectories indicated by the reference trajectory information 190 as second similar trajectories. The third matching process extracts trajectories similar to the third target trajectory from the trajectories indicated by the reference trajectory information 190 as third similar trajectories. Specifically, the matching unit 130 finds similar intervals corresponding to the trajectory of each pattern by matching the trajectory of each pattern indicated by the trajectory information generated by the trajectory extraction unit 120 with the trajectory indicated by the reference trajectory information 190. At this time, the matching unit 130 calculates the similarity for each pattern's trajectory corresponding to the similar interval. The matching process is a process of extracting parts from the trajectory indicated by the reference trajectory information 190 that are similar to or correspond to the trajectory of each pattern indicated by the trajectory information. The similar interval is a continuous interval indicated by the reference trajectory information 190, and is an interval corresponding to a certain time range. In the similar interval corresponding to the trajectory of the target pattern, the similarity between the trajectory of the target pattern and the trajectory indicated by the reference trajectory information 190 is relatively high. The similarity can be calculated in any way. In addition, if work is missed or non-routine work occurs during the target period, the similar interval corresponding to the target period may be an interval corresponding to the work that the target worker should have been doing. Further, based on the reference trajectory information 190, the matching unit 130 identifies the frame identifier corresponding to the target frame in each similar section. The frame identifier corresponding to the target frame is a frame identifier identified based on the matching result and is the frame identifier corresponding to the frame corresponding to the target frame among the frames corresponding to each time point of the trajectory indicated by the reference trajectory information 190. The frame identifier is an identifier for identifying the frame corresponding to each time point of the trajectory indicated by the reference trajectory information 190, and as a specific example, it is a frame number. The frame corresponding to each time point of the trajectory indicated by the reference trajectory information 190 is, as a specific example, the frame that is the origin of the trajectory of the target part at each time point. Based on the result of matching using the frame corresponding to a time before the time corresponding to the target frame and based on the reference trajectory information 190 or the information indicating the work content to be performed by the target worker, the matching unit 130 estimates the work content to be performed by the target worker at the time corresponding to the target frame, estimates the position of the similar section based on the estimated work content, and may perform matching considering the estimated result.

[0014] The similarity comparison unit 140 outputs information corresponding to the work content corresponding to the estimated work trajectory. The estimated work trajectory is the trajectory corresponding to the highest similarity among the similarities between the first target trajectory and the first similar trajectory, the similarity between the second target trajectory and the second similar trajectory, and the similarity between the third target trajectory and the third similar trajectory, and is the trajectory indicated by the reference trajectory information 190. As a specific example, the similarity comparison unit 140 compares the similarities corresponding to the trajectories of each pattern obtained by the matching unit 130 and outputs the frame identifier in the similar section corresponding to the trajectory of the pattern with the highest corresponding similarity, which is the frame identifier identified by the matching unit 130.

[0015] The reference trajectory information 190 is information indicating the trajectory of the part corresponding to the target part when the operator is performing steady work, and is information indicating the frame identifier corresponding to each time point of the trajectory of the target part. Steady work is the work that the operator should perform. The part corresponding to the target part is, as a specific example, the same part as the target part or a part equivalent to the target part. The trajectory of the target part indicated by the reference trajectory information 190 may indicate the variation of the relative position of the part other than the target part or the target part with respect to the work target, or may indicate the range of the position of the target part at each time point. The information indicating the frame identifier corresponds to the information corresponding to the work content corresponding to each time point of the trajectory indicated by the reference trajectory information 190. The reference trajectory information 190 may be time-series data of skeleton information for a plurality of cycles acquired in advance. A cycle is a unit indicating the work for one cycle of the work performed periodically. Also, the reference trajectory information 190 may be time-series data obtained by summarizing the skeleton information for a plurality of cycles into one cycle, such as by obtaining the average value of the skeleton information for a plurality of cycles, or may be a model obtained by machine learning the trajectory of the target part using the skeleton information for a plurality of cycles.

[0016] FIG. 2 is a diagram schematically explaining the processing of the motion analysis device 100. In the example shown in FIG. 2, it is assumed that the target joint is extracted as the target part of the target operator from the input video by the skeleton extraction unit 110, and then the trajectory of the target joint is extracted by the trajectory extraction unit 120. Here, the input trajectory information is the trajectory information generated by the trajectory extraction unit 120 and is information indicating the trajectory of the target joint of the target operator. Also, in this example, the reference trajectory information 190 is information indicating the trajectory of the target joint, which is the target part of the operator shown in the reference video. It is assumed that the reference trajectory information 190 is generated by extracting the target joint of the operator shown in the reference video and extracting the trajectory of the extracted target joint. Here, the reference video is the same as the input video. In the reference trajectory information 190 according to this example, the trajectory of the target joint is divided into a plurality of sections according to the corresponding work content, and labels are assigned to each section. In this example, the matching unit 130 matches the trajectory of the target joint indicated by the input trajectory information to the trajectory of the section corresponding to label 2 among the trajectories of the target joint indicated by the reference trajectory information 190. The matching unit 130 also identifies the frame identifier of the frame included in the reference video, which corresponds to the target frame, based on the joint position corresponding to the target frame and the joint position indicated by the reference trajectory information 190. In this example, the work content corresponding to label 2 was identified as the work content corresponding to the target frame.

[0017] Figure 3 is a schematic diagram illustrating the processing of the matching unit 130. In this example, the input trajectory information and the reference trajectory information 190 are each represented by graphs. The input trajectory information in this example is information showing the trajectory of the target part in the period before and after the target frame. The horizontal axis of the graph shows each frame included in the input video or reference video in the order of the shooting time corresponding to each frame, and the vertical axis of the graph shows the coordinates of the position of the target part. The circles in the graph indicate the position of the target part in each frame included in the input video or reference video. The solid lines in each graph are lines drawn by interpolating the position of the target part in each frame. The dashed lines in the graph corresponding to the reference trajectory information 190 indicate the shape of the graph corresponding to the input trajectory information. In this example, the position of the target part is represented in one dimension, but the position of the target part may be represented in two or three dimensions. In this example, the matching unit 130 performs matching based on the shape of the graph and the application of the coordinates of the target part, as shown by the dashed line in the graph corresponding to the reference trajectory information 190. Subsequently, the matching unit 130 obtains a frame identifier corresponding to the target frame based on the matching result.

[0018] Figure 4 shows an example of the hardware configuration of the motion analysis device 100 according to this embodiment. The motion analysis device 100 consists of a computer. The motion analysis device 100 may consist of multiple computers.

[0019] As shown in this figure, the motion analysis device 100 is a computer equipped with hardware such as a processor 11, a storage device 12, a communication device 13, and an input / output interface 14. These hardware components are appropriately connected via signal lines 19.

[0020] The processor 11 is an integrated circuit (IC) that performs arithmetic operations and controls the hardware of the computer. Specific examples of the processor 11 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The motion analysis device 100 may be equipped with multiple processors that replace the processor 11. The multiple processors share the role of the processor 11.

[0021] The memory device 12 consists, as a specific example, of a volatile memory device and a non-volatile memory device. A specific example of a volatile memory device is RAM (Random Access Memory). Non-volatile storage devices include, specifically, ROM (Read Only Memory), HDD (Hard Disk Drive), and flash memory.

[0022] The communication device 13 consists of a receiver and a transmitter. A specific example of the communication device 13 is a communication chip or a NIC (Network Interface Card).

[0023] The input / output interface 14 is a port to which input and output devices are connected. A specific example of the input / output interface 14 is a USB (Universal Serial Bus) terminal. Specific examples of input devices include a keyboard and a mouse. A specific example of an output device is a display.

[0024] Each part of the motion analysis device 100 may use the communication device 13 and the input / output interface 14 as appropriate when communicating with other devices.

[0025] The memory device 12 stores the motion analysis program. The motion analysis program is a program that enables the computer to implement the functions of each part of the motion analysis device 100. The motion analysis program is executed by the processor 11. The functions of each part of the motion analysis device 100 are implemented by software.

[0026] The data used when executing the motion analysis program, and the data obtained by executing the motion analysis program, are appropriately stored in the storage device 12. Each part of the motion analysis device 100 utilizes the storage device 12 as appropriate. Note that the terms "data" and "information" may have the same meaning. The storage device 12 may be independent of the computer.

[0027] The motion analysis program may be recorded on a computer-readable non-volatile recording medium. Examples of non-volatile recording media include optical discs or flash memory. The motion analysis program may also be provided as a program product.

[0028] ***Explanation of operation*** The operating procedure of the motion analysis device 100 corresponds to the motion analysis method. Furthermore, the program that implements the operation of the motion analysis device 100 corresponds to the motion analysis program.

[0029] Figure 5 is a flowchart illustrating an example of the operation of the motion analysis device 100. The operation of the motion analysis device 100 will be explained using Figure 5.

[0030] (Step S101) External video input is received by the skeletal extraction unit 110. Here, the input video is assumed to show the target worker.

[0031] (Step S102) The skeleton extraction unit 110 extracts the skeleton of the target worker in each frame included in the input video and generates skeleton information that shows the skeleton extracted in each frame.

[0032] (Step S103) The trajectory extraction unit 120 selects a target frame and extracts three trajectory patterns for the target part from the skeletal information generated by the skeletal extraction unit 110, based on the selected target frame. Subsequently, the trajectory extraction unit 120 generates trajectory information showing each of the extracted trajectory patterns.

[0033] (Step S104) The matching unit 130 finds similar intervals corresponding to each pattern's trajectory by matching the trajectory of each pattern indicated by the trajectory information generated by the trajectory extraction unit 120 with the trajectory indicated by the reference trajectory information 190. At this time, the matching unit 130 calculates the degree of similarity for each pattern's trajectory corresponding to the similar interval. The matching unit 130 also identifies the frame identifier corresponding to the target frame in each similar interval based on the reference trajectory information 190.

[0034] (Step S105) The similarity comparison unit 140 compares the similarity of the trajectories of each pattern obtained by the matching unit 130 and outputs the frame identifier corresponding to the trajectory of the pattern with the highest similarity.

[0035] ***Explanation of the effects of Embodiment 1*** According to the conventional technology, matching is performed for the period before and after the target frame. However, when work is missed or non-routine work occurs during the period before and after the target frame, in many cases, there is no trajectory in the reference trajectory information 190 that is similar to the trajectory shown by the input trajectory information during the period before and after the target frame. Therefore, as shown in Figure 6(a), it is not possible to properly match the trajectory shown by the input trajectory information with the trajectory shown by the reference trajectory information 190. Consequently, according to the conventional technology, it is not possible to identify the frame identifier corresponding to the target frame with relatively high accuracy, and therefore it is not possible to identify the work corresponding to the target frame with relatively high accuracy.

[0036] On the other hand, according to this embodiment, matching is performed with the trajectory shown in the reference trajectory information 190 using three trajectory patterns based on the target frame. Here, even if a task is missed or an irregular task occurs, at least one of the trajectories in the period immediately before the task is missed or an irregular task occurs, and the trajectory immediately after the completion of the task is missed or an irregular task occurs, is usually considered to be similar to a part of the trajectory shown in the reference trajectory information 190. Therefore, according to this embodiment, even if a task is missed or an irregular task occurs, at least one of the three trajectory patterns can usually be appropriately matched with a part of the trajectory shown in the reference trajectory information 190. As a specific example, according to this embodiment, even if it is not possible to properly perform matching in the preceding and succeeding periods as shown in Figure 6(a), if it is possible to properly perform matching in the preceding period as shown in Figure 6(b), the frame identifier corresponding to the target frame can be identified with relatively high accuracy. Here, the preceding period is the period during which the target worker was performing routine work. That is, according to this embodiment, even if an action different from routine work occurs in the latter half of the preceding and succeeding periods, if matching is performed using the trajectory in the preceding period, the similarity in the matching result will be higher because the trajectory in the latter half of the preceding and succeeding periods will not be used. Similarly, according to this embodiment, if an action different from routine work occurs in the first half of the preceding and succeeding periods, if matching is performed using the trajectory in the subsequent period, the similarity in the matching result will be higher. Therefore, according to this embodiment, even if work is missed or non-routine work occurs, the work corresponding to the target frame can be identified with relatively high accuracy, and thus the accuracy of the work being performed by the worker who should be performing routine work can be analyzed with relatively high accuracy.

[0037] ***Other configurations*** <Example 1> In this modified example, the matching unit 130, when performing matching with respect to the target frame, uses either an absolute coordinate system or a relative coordinate system as the coordinate system of the trajectory indicated by the input trajectory information, instead of using a coordinate system with the coordinates corresponding to the target frame as the origin. In particular, the matching unit 130 uses either a relative coordinate system or an absolute coordinate system as the coordinate system of the third target trajectory. Furthermore, the matching unit 130 saves matching results as appropriate, in particular saving matching results for subsequent periods. When matching results for periods overlapping with the target period are saved, the matching unit 130 reuses the saved matching results when performing matching for the target period. The target period is typically a period prior to or immediately before or after the target frame. As a specific example, when matching results for the same period as a prior period based on the target frame are saved, the matching unit 130 reuses the saved matching results as matching results for that prior period. In this modified example, since matching is performed without converting the target frame into a trajectory with the target frame as the origin, the matching results for subsequent periods can be reused when performing matching for previous periods and preceding / following periods. Therefore, in step S103, the trajectory extraction unit 120 may extract only the trajectory for subsequent periods instead of extracting three trajectory patterns. Also, in step S104, when performing matching for previous periods and preceding / following periods, the matching unit 130 may reuse the matching results for subsequent periods corresponding to times earlier than the time corresponding to the target frame.

[0038] According to this modified version, since the coordinate system of the trajectory indicated by the input trajectory information is either an absolute coordinate system or a relative coordinate system, the matching results for a period after the target frame can be reused in matching for periods other than the subsequent period. Therefore, according to this modified version, the matching results for a period after the target frame can be reused in matching for times later than the time corresponding to the target frame. Specifically, the matching results for the first subsequent period based on the first target frame can be reused in matching for the second prior period and the second preceding and succeeding periods based on the second target frame. Here, the second target frame is a frame that corresponds to a time later than the time the first target frame was acquired. There is an overlapping period between the first subsequent period and the second prior period, and there is also an overlapping period between the first subsequent period and the second preceding and succeeding periods.

[0039] <Modification 2> Figure 7 shows an example of the hardware configuration of the motion analysis device 100 according to this modified example. The motion analysis device 100 includes a processing circuit 18 instead of a processor 11, or a processor 11 and a memory device 12. The processing circuit 18 is hardware that implements at least some of the components of the motion analysis device 100. The processing circuit 18 may be dedicated hardware, or it may be a processor that executes the program stored in the memory device 12.

[0040] When the processing circuit 18 is dedicated hardware, specific examples of the processing circuit 18 include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The motion analysis device 100 may include multiple processing circuits that replace the processing circuit 18. The multiple processing circuits share the role of the processing circuit 18.

[0041] In the motion analysis device 100, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0042] The processing circuit 18 can be implemented, in specific examples, by hardware, software, firmware, or a combination thereof. The processor 11, the memory device 12, and the processing circuit 18 are collectively referred to as the "processing circuitry." In other words, the functions of each functional component of the motion analysis device 100 are realized by the processing circuitry. The motion analysis device 100 according to other embodiments may also have the same configuration as this modified example.

[0043] Embodiment 2. The following will explain the differences from the previously described embodiment, primarily with reference to the drawings.

[0044] ***Explanation of the structure*** The configuration of the motion analysis device 100 according to this embodiment is the same as the configuration of the motion analysis device 100 according to Embodiment 1. The matching unit 130 in this embodiment performs a first matching process, and if the similarity between the first target trajectory and the first similar trajectory is lower than the first threshold, it performs the second matching process and the third matching process, respectively. The first threshold can be determined in any way. Specifically, when the matching unit 130 performs matching, it first performs matching in the previous period. Furthermore, if the similarity in the previous period is sufficiently high, the matching unit 130 does not perform matching in the preceding and succeeding periods, or in the subsequent period.

[0045] ***Explanation of operation*** Figure 8 is a flowchart illustrating an example of the operation of the motion analysis device 100. The operation of the motion analysis device 100 will be explained using Figure 8. In this flowchart, each period is a period based on the target frame.

[0046] (Step S201) The trajectory extraction unit 120 extracts the trajectory of the target part from the skeletal information during the previous period.

[0047] (Step S202) The matching unit 130 performs matching for the previous period using the trajectory extracted by the trajectory extraction unit 120 and the reference trajectory information 190.

[0048] (Step S203) If the similarity corresponding to the matching result in step S202 is lower than the first threshold, the motion analyzer 100 proceeds to step S204. Otherwise, the motion analyzer 100 terminates the processing of this flowchart. Furthermore, it is preferable to set the first threshold to a reasonably high value in order to prevent misjudgments.

[0049] (Step S204) The trajectory extraction unit 120 extracts the trajectories of the target part from the skeletal information for each of the preceding and succeeding periods and the succeeding period.

[0050] (Step S205) The matching unit 130 uses the trajectory extracted by the trajectory extraction unit 120 in step S204 and the reference trajectory information 190 to perform matching between the preceding and succeeding periods, and between the succeeding and succeeding periods, respectively.

[0051] (Step S206) The similarity comparison unit 140 outputs a frame identifier corresponding to the highest similarity among the similarity corresponding to the previous period, the similarity corresponding to the preceding and succeeding periods, and the similarity corresponding to the subsequent period.

[0052] ***Explanation of the effects of Embodiment 2*** As described above, according to this embodiment, since matching is performed only for the previous period when the similarity corresponding to the previous period is sufficiently high, the total number of matching operations can be reduced. Therefore, according to this embodiment, the matching process can be sped up.

[0053] Embodiment 3. The following will explain the differences from the previously described embodiment, primarily with reference to the drawings.

[0054] ***Explanation of the structure*** The configuration of the motion analysis device 100 according to this embodiment is the same as the configuration of the motion analysis device 100 according to Embodiment 1. In Embodiment 2, matching is first performed for the previous period. If the similarity to the previous period is low, it is considered that there were omissions or non-routine tasks during that period. Therefore, when the target frame is considered to be a frame acquired within the target period, matching is performed only for the subsequent period based on the target frame. The target period is the period during which omissions or non-routine tasks occurred in the work of the target worker. Furthermore, the similarity between a target frame and subsequent periods, based on the target frame at the point when the time corresponding to the target frame has left the target period, is higher than the similarity between a target frame and subsequent periods based on the target frame within the target period. Therefore, if the similarity between a target frame and subsequent periods becomes sufficiently high after the target period has entered, the system reverts to performing only matching within the previous period. Note that the target frames are advanced sequentially to the next frame.

[0055] In this embodiment, the matching unit 130 performs a third matching process if, after performing the first matching process, the similarity between the first target trajectory and the first similar trajectory is lower than the second threshold. Furthermore, if, after performing the third matching process, the matching unit 130 performs the first matching process if the similarity between the third target trajectory and the third similar trajectory is lower than the third threshold. The second and third thresholds may be defined in any way.

[0056] In this embodiment, there is basically only one matching result, so the similarity comparison unit 140 according to this embodiment outputs a frame identifier corresponding to the matching result.

[0057] ***Explanation of operation*** Figure 9 is a flowchart illustrating an example of the operation of the motion analysis device 100. The operation of the motion analysis device 100 will be explained using Figure 9. In this flowchart, each period is a period based on the target frame. The target frame is advanced sequentially to frames corresponding to future times, and the processing in this flowchart is executed each time the target frame is advanced.

[0058] (Step S301) If the matching period is a previous period, the motion analyzer 100 proceeds to step S302. Otherwise, the motion analyzer 100 proceeds to step S306.

[0059] (Step S302) This step is the same as step S201.

[0060] (Step S303) This step is the same as step S202.

[0061] (Step S304) If the similarity in the matching result of step S303 is lower than the second threshold, the motion analyzer 100 proceeds to step S305. Otherwise, the motion analyzer 100 terminates the processing of this flowchart.

[0062] (Step S305) The motion analysis device 100 changes the matching target period to the subsequent period. Furthermore, if the similarity corresponding to a subsequent period is lower than the similarity corresponding to a previous period, the motion analysis device 100 does not need to change the matching target period to the subsequent period.

[0063] (Step S306) The trajectory extraction unit 120 extracts the trajectory of the target body part from the skeletal information for the period thereafter.

[0064] (Step S307) The matching unit 130 will perform matching during the period thereafter.

[0065] (Step S308) If the similarity in the matching result in step S307 is lower than the third threshold, the motion analyzer 100 proceeds to step S309. Otherwise, the motion analyzer 100 terminates the processing of this flowchart.

[0066] (Step S309) The motion analysis device 100 changes the matching target period to a previous period. Furthermore, if the similarity corresponding to the previous period is lower than the similarity corresponding to the subsequent period, the motion analysis device 100 does not need to change the matching target period to the previous period.

[0067] ***Explanation of the effects of Embodiment 3*** As described above, according to this embodiment, since matching is not performed in the preceding and succeeding periods, the total number of matching operations can be reduced. Therefore, according to this embodiment, the matching process can be accelerated.

[0068] ***Other Embodiments*** The aforementioned embodiments can be freely combined, any component of each embodiment can be modified, or any component can be omitted in each embodiment. Furthermore, the embodiments are not limited to those shown in Embodiments 1 to 3, and various modifications can be made as needed. The procedures described using flowcharts, etc., may be modified as appropriate. [Explanation of symbols]

[0069] 11 Processor, 12 Storage device, 13 Communication device, 14 Input / Output interface, 18 Processing circuit, 19 Signal line, 100 Motion analysis device, 110 Skeleton extraction unit, 120 Trajectory extraction unit, 130 Matching unit, 140 Similarity comparison unit, 190 Reference trajectory information.

Claims

1. a third matching process for extracting, as a third similar trajectory, a trajectory of a target body part of a target worker in a subsequent period from a target time, which is a time corresponding to a target frame included in an input video showing a target worker performing work, to a time in the future than the target time, said trajectory being similar to a third target trajectory, which is a trajectory derived based on the input video, from a trajectory indicated by reference trajectory information indicating a trajectory of a body part corresponding to the target body part when the target worker is performing the work to be performed; a second matching process for extracting, as a second similar trajectory, a trajectory of the target part in a period from a time earlier than the target time to a time later than the target time, the trajectory being similar to a second target trajectory that is a trajectory derived based on the input video, from the trajectory indicated by the reference trajectory information; A matching unit that performs the above steps; a similarity comparison unit that outputs information corresponding to a work content corresponding to an estimated work trajectory, which is a trajectory corresponding to the highest similarity among the similarity between the second object trajectory and the second similar trajectory and the similarity between the third object trajectory and the third similar trajectory, and is a trajectory indicated by the reference trajectory information; A motion analysis device comprising: The reference trajectory information includes information corresponding to the task content corresponding to each point in time of the trajectory indicated by the reference trajectory information.

2. the matching unit performs a first matching process to extract, as a first similar trajectory, a trajectory of the target part in a previous period that is a period from a time earlier than the target time to the target time and that is similar to a first target trajectory that is a trajectory derived based on the input video, from the trajectory indicated by the reference trajectory information; 2. The motion analysis device of claim 1, wherein the estimated work trajectory is a trajectory corresponding to the highest similarity among the similarity between the first object trajectory and the first similar trajectory, the similarity between the second object trajectory and the second similar trajectory, and the similarity between the third object trajectory and the third similar trajectory.

3. The motion analysis device further comprises: a skeleton extraction unit that extracts a skeleton of the target worker in each frame included in the input video and generates skeleton information indicating the skeleton extracted in each frame; a trajectory extraction unit that extracts each of the first object trajectory, the second object trajectory, and the third object trajectory from the generated skeleton information; The motion analysis device according to claim 2 .

4. The motion analysis device according to claim 2 or 3, wherein the matching unit performs the first matching process, and when a similarity between the first target trajectory and the first similar trajectory is lower than a first threshold, performs each of the second matching process and the third matching process.

5. The matching unit is performing the third matching process when a similarity between the first object trajectory and the first similar trajectory is lower than a second threshold value in the first matching process; The motion analysis device according to claim 2 or 3, wherein when the third matching process is performed, the first matching process is performed if a similarity between the third target trajectory and the third similar trajectory is lower than a third threshold value.

6. The motion analysis device according to claim 1 , wherein the matching unit uses a relative coordinate system or an absolute coordinate system as a coordinate system of the third object trajectory.

7. a third matching process is performed by the computer to extract, as a third similar trajectory, a trajectory of a target body part of a target worker in a subsequent period from a target time, which is a time corresponding to a target frame included in an input video showing a target worker performing work, to a time in the future than the target time, the trajectory being a third target trajectory derived based on the input video, from a trajectory indicated by reference trajectory information indicating a trajectory of a body part corresponding to the target body part when the target worker is performing the work to be performed; the computer performs a second matching process to extract, as a second similar trajectory, a trajectory of the target part during a period from a time earlier than the target time to a time later than the target time, the trajectory being similar to a second target trajectory that is a trajectory derived based on the input video, from the trajectory indicated by the reference trajectory information; a motion analysis method in which the computer outputs information corresponding to a task content corresponding to an estimated task trajectory, which is a trajectory corresponding to the highest similarity among a similarity between the second object trajectory and the second similar trajectory and a similarity between the third object trajectory and the third similar trajectory, and which is a trajectory indicated by the reference trajectory information, A motion analysis method, wherein the reference trajectory information includes information corresponding to task content corresponding to each point in time of the trajectory indicated by the reference trajectory information.

8. a third matching process for extracting, as a third similar trajectory, a trajectory of a target body part of a target worker in a subsequent period from a target time, which is a time corresponding to a target frame included in an input video showing a target worker performing work, to a time in the future than the target time, said trajectory being similar to a third target trajectory, which is a trajectory derived based on the input video, from a trajectory indicated by reference trajectory information indicating a trajectory of a body part corresponding to the target body part when the target worker is performing the work to be performed; a second matching process for extracting, as a second similar trajectory, a trajectory of the target part during a period from a time earlier than the target time to a time later than the target time, the trajectory being similar to a second target trajectory that is a trajectory derived based on the input video, from the trajectory indicated by the reference trajectory information; a similarity comparison process for outputting information corresponding to a work content corresponding to an estimated work trajectory, which is a trajectory corresponding to the highest similarity among the similarity between the second object trajectory and the second similar trajectory and the similarity between the third object trajectory and the third similar trajectory, and which is a trajectory indicated by the reference trajectory information; A motion analysis program for causing a motion analysis device that is a computer to execute the above, A motion analysis program, wherein the reference trajectory information includes information corresponding to the work content corresponding to each point in time of the trajectory indicated by the reference trajectory information.

Images