Behavioral analysis system and behavioral analysis method

Abstract

To provide an action analysis system and an action analysis method that analyzes start time, a speed and a degree of perfection of an action of a target person who moves following a sample.SOLUTION: An action analysis system 100 includes: a sample video input unit which receives input of a sample video; a target video input unit which receives input of target person video; an evaluation timing identifying unit 18 which identifies evaluation timing by selecting a target person image frame for evaluating action of a target person from among a plurality of image frames included in the target person video; and a similarity evaluation unit 20 which evaluates a degree of similarity by comparing the selected target person image frame with a sample image frame included in the sample video and corresponding to the target person image frame. The evaluation timing identifying unit 18 selects a target person image frame on the basis of an image frame relating to a predetermined action and included in the target person video, and an image frame related to the predetermined action and included in the sample video.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an action analysis system and an action analysis method.

Background Art

[0002] The amount of in-home exercise has been increasing, and services for remotely monitoring and supporting exercise instead of in-person are being considered.

[0003] Patent Document 1 discloses an action similarity evaluation device that extracts skeletal information of each action from a model video and an imitation video, and evaluates the similarity of each action based on the similarity of the skeletal information at the synchronization time, enabling accurate action similarity evaluation regardless of differences in body shape, clothing, or height between the model of the model video and the user.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the action similarity evaluation device described in Patent Document1, there is room for improvement in that the method for determining the synchronization time, the evaluation method when the speed of each action of the imitation is different from that of the model, etc. are not clear.

[0006] An object of the present disclosure is to analyze the start time, speed, completion degree, etc. of the actions of a subject who performs actions according to a model.

Means for Solving the Problems

[0007] The behavioral analysis system of this disclosure comprises: a sample video input unit for inputting sample video; a subject video input unit for inputting subject video; an evaluation timing identification unit for identifying an evaluation timing by selecting a subject image frame for evaluating the subject's actions from a plurality of image frames contained in the subject video; and a similarity evaluation unit for comparing the selected subject image frame with a sample image frame contained in the sample video that corresponds to the subject image frame, and evaluating the similarity. The evaluation timing identification unit selects a subject image frame based on an image frame related to a predetermined action contained in the subject video and an image frame related to a predetermined action contained in the sample video.

[0008] The behavioral analysis method of this disclosure includes a sample video input unit that inputs a sample video, a subject video input unit that inputs a subject video, an evaluation timing identification unit that identifies an evaluation timing by selecting a subject image frame from multiple image frames included in the subject video to evaluate the subject's actions, a similarity evaluation unit that compares the selected subject image frame with a sample image frame included in the sample video that corresponds to the subject image frame and evaluates the similarity, and the evaluation timing identification unit selects a subject image frame based on an image frame related to a predetermined action included in the subject video and an image frame related to a predetermined action included in the sample video. [Effects of the Invention]

[0009] According to this disclosure, it is possible to analyze the start time, speed, and degree of completion of the actions of a subject performing actions according to a sample. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing the behavioral analysis system of Example 1. [Figure 2] Figure 1 is a flowchart illustrating the process for determining the evaluation timing in the behavioral analysis system. [Figure 3]Figure 1 is a flowchart showing the process in the evaluation operation identification unit 22. [Figure 4] This is a flowchart that outlines the specific process of Example 1. [Figure 5] This is a diagram showing the behavioral analysis system of Example 2. [Figure 6] This is a diagram showing the behavioral analysis system of Example 3. [Figure 7] This is a diagram showing the main components of the behavioral analysis system of Example 4. [Figure 8] This is a diagram showing the main components of the behavioral analysis system of Example 5. [Figure 9] This is a diagram showing the main components of the behavioral analysis system of Example 6. [Figure 10] This diagram shows the overall configuration of the behavioral analysis system related to this disclosure. [Modes for carrying out the invention]

[0011] The following describes an example using drawings. [Examples]

[0012] Figure 1 is a diagram showing the configuration of the behavioral analysis system of Example 1.

[0013] The behavioral analysis system 100 shown in this figure includes a sample video input unit 10 for inputting sample videos, a subject video input unit 12 for inputting videos of subjects (subject videos), a specific person selection unit 14, a posture extraction unit 16, an evaluation timing identification unit 18, a similarity evaluation unit 20, an evaluation action identification unit 22, and a similarity overall determination unit 24.

[0014] The specific person selection unit 14 receives data of the sample video input into the sample video input unit 10 and the video of the subject input into the subject video input unit 12, and identifies the subject as the person corresponding to the sample.

[0015] The posture extraction unit 16 extracts the posture of the person identified by the specific person selection unit 14. Specifically, it extracts the person's skeletal information.

[0016] The evaluation timing specifying unit 18 receives the posture of a specific person extracted by the posture extraction unit 16 and the data of the reference video from the reference video input unit 10, and uses these data to associate the actions of the specific person with the actions of the reference. In this case, even if the actions of the person are significantly different from the actions of the reference, or if the timing of the person's actions is significantly deviated from the reference, they are specified as actions to be evaluated.

[0017] In other words, the evaluation timing specifying unit 18 selects an image frame (target person image frame) for evaluating the actions of the target person from a plurality of image frames included in the video of the target person. In this case, for a plurality of image frames in which the action of the target person is a certain scene, a scene with a similar action is specified as an image frame to be evaluated (reference image frame) from the reference video. Here, an image frame refers to the entire area of each image constituting a video (moving picture), and it can also be said that it is an image of one frame constituting the video. Also, selecting an image frame is the same as determining the evaluation timing. Here, the evaluation timing refers to the timing for evaluating the actions of the target person.

[0018] Note that in the evaluation timing specifying unit 18, the comparison between the actions of the target person and the actions of the reference may be performed using the dynamic time warping (DTW) method. By using DTW, it is possible to determine the identity of the actions even when the actions of the target person are slower or faster than the actions of the reference. Also, regarding the determination of the identity of the actions, in addition to DTW, artificial intelligence (AI) may be used. Furthermore, machine learning may be used, and among machine learning, deep learning may be used. Here, the fact that the action is slow corresponds to a large number of image frames in which the same action is captured. Also, the fact that the action is fast corresponds to a small number of image frames in which the same action is captured. Furthermore, when the action is not performed, the search timing result may sometimes become a fixed value.

[0019] The similarity evaluation unit 20 calculates and evaluates the similarity between the operation identified by the evaluation timing identification unit 18 and the sample. The similarity evaluation unit 20 may also apply DTW or AI deep learning. Furthermore, methods for comparing image features can also be used.

[0020] Here, similarity can be quantified by comparing the subject's actions with the example's actions in a single image frame, or by comparing time-series data of the subject's actions and the example's actions, which are composed of multiple image frames. Furthermore, similarity can be used to evaluate the actions of the entire body, or to evaluate the actions of individual body parts.

[0021] The evaluation action identification unit 22 identifies, from the posture extracted by the posture extraction unit 16, actions (parts of the sample actions) that are performed in the sample but not in the action targeted by the similarity evaluation unit 20 (the action of the subject), and identifies the cause (the reason why the action was not performed in the action of the subject). In addition, a similarity threshold can be set in advance, and if the similarity is below the threshold, it can be determined that the movement was not performed properly. Once a poorly performed action is identified, the movement can be identified using the skeletal information of the person extracted from the posture extraction unit 16. If the similarity is above the threshold, it can be determined that the movement was performed properly.

[0022] The Similarity Comprehensive Determination Unit 24 uses the data obtained from the Similarity Evaluation Unit 20 and the Evaluation Action Identification Unit 22 to comprehensively determine the similarity (overall similarity) between the action in question and the sample. It is possible to calculate the similarity of all actions using the similarity of the same action. Furthermore, if there are multiple people, it is also possible to calculate the similarity of all actions for each person.

[0023] Furthermore, it is desirable that the results obtained by the evaluation timing identification unit 18, the similarity evaluation unit 20, the evaluation operation identification unit 22, the overall similarity determination unit 24, etc., can be viewed by the user on an output unit (not shown) such as a PC, mobile terminal, or website. Here, PC refers to a personal computer.

[0024] Figure 2 is a flowchart showing the process for determining the evaluation timing in the behavioral analysis system shown in Figure 1.

[0025] In Figure 2, the process involves reading the video data of the subject (step S11), extracting the region of a specific person (step S12), and then determining the presence of that person (step S13). If the person is present, their posture is extracted (step S14). If the person is not present, the process returns to step S11. Here, the video data specifically refers to the data of multiple image frames contained in the video.

[0026] Furthermore, the sample video data is also read (step S21). Then, similar to step S12, the region of a specific person is extracted (step S22). Next, the presence of that person is determined (step S23). If the person is present, their posture is extracted (step S24). On the other hand, if the person is not present, the process returns to step S21.

[0027] Next, using the postures extracted in steps S14 and S24, a posture to be compared is searched for (step S31). Then, it is determined whether the obtained posture is the closest (step S32). The evaluation timing is determined from the posture determined to be the closest in step S32 (step S33).

[0028] Figure 3 is a flowchart showing the process in the evaluation operation identification unit 22 of Figure 1.

[0029] Figure 3 shows an example of extracting the pose of a movement (step S41). In this case, the positions of the joints of the person's head, hands, arms, legs, etc., are extracted.

[0030] Next, the shape of the posture is transformed (step S42). Specifically, the positions of joints, etc., extracted in step S41 are represented as points, and operations such as (a) scaling, (b) linear transformation, (c) affine transformation, and (d) projective transformation are performed on the figure formed by these points.

[0031] Next, a comparison of posture is performed (step S43). In this case, the positions of the joints of the person's head, hands, arms, legs, etc. are compared with the sample.

[0032] Next, posture analysis is performed (step S44). In this case, data such as the center of gravity and arm angle (e.g., 120 degrees) are obtained.

[0033] Figure 4 is a flowchart illustrating the specific process of this embodiment.

[0034] This diagram shows an overview of the specific circumstances of the operation and the analysis.

[0035] First, the user performs exercises such as Tai Chi or yoga according to a sample video and films their own movements using a mobile device or similar. The filmed video (video of the subject) is then transferred to the analysis PC (analysis unit) of the behavioral analysis system. The analysis unit may be a computing device such as a server, in addition to a PC.

[0036] Next, the analysis PC analyzes the performance of the exercise and outputs the analysis results. The analysis results are sent to the user's mobile device or other device. The analysis results may be displayed in numerical form, tables, graphs, etc., or they may point out areas that the user should correct to improve the performance of the exercise.

[0037] Through the process described above, users can learn about the level of their movement and repeat training while paying attention to how their own movements can closely resemble the example. [Examples]

[0038] Figure 5 is a diagram showing the behavioral analysis system of Example 2.

[0039] The behavioral analysis system 500 shown in this figure includes a sample video input unit 50 for inputting sample videos, multiple subject video input units 51, 52, and 53 for inputting videos of multiple subjects, a multiple person selection unit 54, a multiple person posture extraction unit 56, an evaluation timing identification unit 58, a similarity evaluation unit 60, an evaluation action identification unit 62, and a similarity overall determination unit 64.

[0040] The multiple-person selection unit 54 receives sample video and video data of multiple subjects, and identifies multiple individuals corresponding to the sample.

[0041] The multi-person posture extraction unit 56 extracts the postures of multiple people identified by the multi-person selection unit 54.

[0042] The evaluation timing identification unit 58 receives the posture data of multiple people extracted by the multiple-person posture extraction unit 56 and the sample video data, and uses this data to associate the actions of the multiple people with the actions of the sample. In this case, even if the actions of the multiple people differ significantly from the sample, or if the timing of the actions differs significantly from the sample, the actions are identified as actions to be evaluated.

[0043] The similarity evaluation unit 60 calculates and evaluates the similarity between the operation identified by the evaluation timing identification unit 58 and the sample.

[0044] The evaluation motion identification unit 62 identifies, from the poses extracted by the multi-person pose extraction unit 56, any motions that are performed in the sample but not in the motion targeted by the similarity evaluation unit 60 (parts of the sample motion), and identifies the cause. Furthermore, by calculating or setting a similarity threshold in advance, it is possible to determine that the movement is not being performed correctly if the similarity is below the threshold. If a poorly performed action is identified, the skeletal information of the person extracted from the multi-person pose extraction unit 56 can be used to identify the action.

[0045] The similarity comprehensive determination unit 64 uses the data obtained from the similarity evaluation unit 60 and the evaluation operation identification unit 62 to comprehensively determine the similarity between the operation and the sample. [Examples]

[0046] Figure 6 is a diagram showing the behavioral analysis system of Example 3.

[0047] The difference between the behavioral analysis system 600 shown in this figure and the behavioral analysis system 100 in Figure 1 is the addition of a playback frame control unit 61.

[0048] The playback frame control unit 61 controls the playback of the sample video to the user at a slow speed. By providing the playback frame control unit 61, the user's understanding of the sample's operation can be improved.

[0049] Furthermore, the evaluation timing specification unit 18 can adjust the sample speed if the speed of the sample video and the speed of the input video are different. For example, if the input is a slow motion, the frame change detected by the evaluation timing specification unit 18 will be slow. This degree of change is then input to the playback frame control unit 61. The playback frame control unit 61 can use the input degree of change to lower the playback speed of the sample. On the other hand, if the input is a fast motion, the playback speed of the sample can be increased using the same degree of change. [Examples]

[0050] Figure 7 is a diagram showing the main components of the behavioral analysis system of Example 4.

[0051] The difference between the behavioral analysis system 700 shown in this figure and the behavioral analysis system 100 shown in Figure 1 is the addition of a behavioral speed extraction unit 71.

[0052] The action speed extraction unit 71 determines the speed of the movements in the sample video. By providing the action speed extraction unit 71, it becomes easier to evaluate the difference in speed between the sample movements and the user's movements. [Examples]

[0053] Figure 8 is a diagram showing the main components of the behavioral analysis system of Example 5.

[0054] The difference between the behavior analysis system 800 shown in this figure and the behavior analysis system 700 in Figure 7 is that a normal behavior database 81, an abnormal behavior database 82, and a sample video generation unit 83 have been added.

[0055] The normal behavior database 81 stores data on behaviors similar to the sample. The abnormal behavior database 82 stores data on behaviors different from the sample. The sample video generation unit 83 edits the sample video to suit each individual user.

[0056] This configuration allows for the display of actions that individual users need to correct, tailored to each user's specific needs.

[0057] Furthermore, normal behavior is considered to be a familiar action that the user has practiced sufficiently with. On the other hand, abnormal behavior is likely to be a movement that the user has not practiced sufficiently with and is unfamiliar with. In such cases, normal behavior and abnormal behavior can be distinguished using the speed extracted. Each behavior database can be updated normally in accordance with the user's practice.

[0058] Furthermore, based on a database of normal and abnormal behaviors, it's possible to practice unfamiliar exercises more frequently. [Examples]

[0059] Figure 9 is a diagram showing the main components of the behavioral analysis system of Example 6.

[0060] The difference between the behavioral analysis system 900 shown in this figure and the behavioral analysis system 100 shown in Figure 1 is the addition of an effectiveness database 91.

[0061] The effectiveness database 91 stores data on the relationship between user practice and its effects. Specifically, it stores data on the degree to which a subject's video differs from a sample video, and the relationship between this and the effects that contribute to the user's health, etc.

[0062] This configuration allows for a clear demonstration of the effectiveness of the practice to the user.

[0063] The effects of the behavioral analysis system and methodology described herein are summarized below.

[0064] According to this disclosure, it is possible to analyze individual differences in the movements of subjects performing actions according to a model, thereby obtaining appropriate evaluations tailored to the movements of each subject.

[0065] Furthermore, the evaluation can be performed regardless of the start time of the subject's actions or the duration of those actions.

[0066] Furthermore, it can be evaluated accurately regardless of the speed of operation.

[0067] Furthermore, even if an operational error occurs, it can be easily detected and corrected.

[0068] Furthermore, the behavioral analysis system and behavioral analysis method disclosed herein can be applied to various actions and can be used in rehabilitation, school education, skills training, and when an untrained robot learns actions using a model robot.

[0069] Examples 1 to 6 describe the case where the user is elderly, but the behavioral analysis system and behavioral analysis method disclosed herein can also be applied to other usage scenarios.

[0070] Examples include educational settings, game-like experiences (such as multiple people participating in a game-like format and evaluating similarity), work training (such as using expert workers' movements as example videos in a factory), exercise with robots (the robot can act as an evaluation server, judging the user's movements and adjusting the robot's own movement speed. It can also record unfamiliar actions and have the robot perform them along with the user during playback), and rehabilitation settings. In educational settings, for example, in dance practice, adjusting the speed, such as playing difficult parts slowly, can improve proficiency.

[0071] The overall structure of the behavioral analysis system related to this disclosure will be explained below using diagrams.

[0072] Figure 10 is a diagram showing the overall configuration of the behavioral analysis system related to this disclosure.

[0073] In this diagram, the movements of user 1001 are recorded on smartphone 1002 and the video is transferred to cloud 1003 (1005). When the video is transferred from cloud 1003 to the motion evaluation server 1004 (1005), the motion evaluation server 1004 evaluates the movements and transfers the evaluation results to cloud 1003 (1005). User 1001 can check the evaluation results of their own movements by receiving the information from cloud 1003 on smartphone 1002 (1006).

[0074] Here, it is also possible to process video data from user 1001 on the cloud 1003. The results can then be sent directly from the cloud 1003 to user 1001's smartphone 1002.

[0075] Furthermore, if the smartphone 1002 has processing capabilities, the video can be processed on the smartphone 1002 and the results can be viewed directly.

[0076] In this example, video transmission 1005 and result transmission 1006 are performed using only user 1001's smartphone 1002. However, information about user 1001's actions can also be transmitted and shared with, for example, medical professionals such as doctors or training gym coaches. In this way, user 1001 can receive objective advice. [Explanation of symbols]

[0077] 10, 50: Sample video input unit, 12, 51, 52, 53: Target person video input unit, 14: Specific person selection unit, 16: Posture extraction unit, 18: Evaluation timing identification unit, 20: Similarity evaluation unit, 22: Evaluation action identification unit, 24: Similarity overall judgment unit, 54: Multiple person selection unit, 56: Multiple person posture extraction unit, 61: Playback frame control unit, 71: Action speed extraction unit, 81: Normal behavior database, 82: Abnormal behavior database, 83: Sample video generation unit, 91: Effect database, 100, 500: Action analysis system.

Claims

1. A sample video input section for inputting sample videos, A subject video input unit for inputting subject video, An evaluation timing determination unit that determines the evaluation timing by selecting a subject image frame for evaluating the subject's actions from multiple image frames included in the subject video, A similarity evaluation unit compares the selected subject image frame with a sample image frame included in the sample video that corresponds to the subject image frame, and evaluates the degree of similarity. A person identification unit receives the data of the sample video and the subject video, and identifies the subject as the person corresponding to the sample. A posture extraction unit that extracts the posture of the person identified by the person identification unit, The system includes an evaluation action identification unit that, from the posture extracted by the posture extraction unit, identifies a part of the sample action that is performed in the sample but is not performed in the action of the target person targeted by the similarity evaluation unit, and identifies the reason why the action is not performed in the action of the target person or the action that could not be performed, The evaluation timing identification unit is an action analysis system that selects the subject image frame based on an image frame relating to a predetermined action that is included in the subject video and an image frame relating to the predetermined action that is included in the sample video.

2. The behavioral analysis system according to claim 1, further comprising an output unit that outputs results including the similarity score.

3. The behavioral analysis system according to claim 1, wherein the evaluation timing identification unit selects the subject image frame by a method using dynamic time stretching or artificial intelligence.

4. The behavioral analysis system according to claim 1, further comprising a comprehensive similarity determination unit that determines the overall similarity between the subject's actions and the sample actions using data obtained from the similarity evaluation unit and the evaluation action identification unit.

5. The sample video input unit receives a sample video, The subject video input unit inputs the subject's video, The evaluation timing determination unit determines the evaluation timing by selecting a subject image frame for evaluating the subject's actions from multiple image frames included in the subject video. The similarity evaluation unit compares the selected subject image frame with the sample image frame included in the sample video that corresponds to the subject image frame, and evaluates the similarity. The designated person selection unit receives the sample video and the target person video data, identifies the target person as the person corresponding to the sample, The posture extraction unit extracts the posture of the person identified by the specific person selection unit, The evaluation action identification unit identifies, from the posture extracted by the posture extraction unit, a part of the sample action that is performed in the sample but is not performed in the action of the subject targeted by the similarity evaluation unit, and identifies the reason why the action is not performed in the action of the subject or the action that could not be performed. The evaluation timing identification unit selects the subject image frame based on an image frame relating to a predetermined action that is included in the subject video and an image frame relating to the predetermined action that is included in the sample video.

6. The behavioral analysis method according to claim 5, wherein the output unit outputs results including the similarity score.

7. The behavioral analysis method according to claim 5, wherein the evaluation timing identification unit selects the subject image frame by a method using dynamic time stretching or artificial intelligence.

8. The behavioral analysis method according to claim 5, wherein the similarity comprehensive determination unit determines the overall similarity between the subject's behavior and the sample's behavior using the data obtained by the similarity evaluation unit and the evaluation behavior identification unit.

Images