A human motion assessment method based on motion state information

Abstract

The present invention relates to the technical field of computer vision image processing, and in particular to a method for evaluating human motion based on motion state information, comprising: obtaining a video of a tester to be tested; performing skeleton analysis and posture analysis on video frames of the video to be tested to generate a corresponding human key point coordinate diagram; selecting multiple optimal video frames based on the human key point coordinate diagram in combination with corresponding motion evaluation standards; calculating motion state information of the video to be tested based on multiple optimal video frames; comparing the motion state information of the video to be tested with motion standard information of a template video to generate a corresponding motion evaluation result. The human motion evaluation method in the present invention can effectively reflect motion continuity and motion stability, thereby improving the accuracy of motion evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image processing, and in particular to a method for evaluating human motion based on motion state information. Background Art

[0002] With the widespread application of Internet big data information technology, the application scenarios of human behavior detection and recognition technology based on visual images are increasing. By analyzing the movement characteristics of human expressions, postures, etc., it can provide rich identification feature information for the detection and prediction of people's behavior in public places or specific activity spaces, which is an important part of human activity big data information.

[0003] For example, in the fields of sports competition and health screening, it is necessary to identify human movements in order to evaluate the standard conditions of the movements. Movement evaluation needs to be based on movement evaluation standards and movement evaluation decision-making basis. In the early days, the commonly used evaluation method was that the evaluator observed the test subject's movements with his eyes, and manually compared the test subject's movements with the standard movement patterns to give scores; at the same time, the evaluator used a handheld camera to extract the video, and then took a screenshot of the video and saved it for reference. The above existing methods not only waste manpower and material resources, but also the evaluation results are not objective and accurate due to subjective reasons during human judgment.

[0004] With the development of computer technology, methods for evaluating actions based on videos have emerged in the prior art. For example, the Chinese patent with publication number CN108205654B discloses "A video-based action detection method and device", which includes: detecting limb key points in the video frame of the video to be detected; obtaining corresponding behavioral features based on the limb key points; performing action completion detection based on the behavioral features and the global action evaluation model to obtain a global detection result, wherein the global action evaluation model is used to detect the completion of human body actions in the video relative to the standard action.

[0005] The motion detection method in the above-mentioned existing scheme obtains behavioral features through the limb key points in the video frame, and then performs motion evaluation based on the behavioral features and the global motion evaluation model. Motion evaluation generally compares each video frame in the video to be tested with the corresponding video frame in the template video and calculates the similarity, and then judges the completion of the action based on the similarity. This is a static motion evaluation. However, for some difficult actions, it is necessary to evaluate the continuity and stability of the action. However, the existing static evaluation method is difficult to effectively reflect the continuity and stability of the tester's action, resulting in low accuracy of motion evaluation. Therefore, how to design a human motion evaluation method that can effectively reflect the continuity and stability of the action is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is: how to provide a human motion evaluation method based on motion state information so as to effectively reflect the motion continuity and motion stability, thereby improving the accuracy of motion evaluation.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A human motion assessment method based on motion state information, comprising:

[0009] S1: Obtain the test video of the tester;

[0010] S2: Perform skeleton analysis and posture analysis on the video frames of the video to be tested, and generate a corresponding human body key point coordinate map;

[0011] S3: Select the corresponding multi-frame optimal video frames based on the human body key point coordinate diagram and the corresponding action evaluation criteria;

[0012] S4: Calculating motion state information of the video to be tested based on multiple optimal video frames;

[0013] S5: Compare the motion state information of the video to be tested with the motion standard information of the template video to generate a corresponding motion evaluation result.

[0014] Preferably, in step S3, when selecting multiple optimal video frames, first, the angle information between each key point is calculated based on the human body key point coordinate map; then, the recommended key points in the human body key point coordinate map are determined based on the angle information between the key points; finally, the corresponding multiple optimal video frames are selected based on the recommended key points in the human body key point coordinate map and the action evaluation criteria.

[0015] Preferably, the angle information between the key points is a counterclockwise rotation angle;

[0016] Calculate the counterclockwise rotation angle by the following steps:

[0017] S301: Obtain key point coordinates A, B, C for calculating the counterclockwise rotation angle;

[0018] S302: Calculate the corresponding key point vector based on the key point coordinates A, B, and C and the key point vector

[0019] S303: The key point vector Rotate counterclockwise until it is aligned with the key point vector coincide; then the key point vector Rotate to keypoint vector The angle is taken as the corresponding counterclockwise rotation angle.

[0020] Preferably, the recommended key points are determined by the following steps:

[0021] S311: Calculate the variance of each counterclockwise rotation angle in the human body key point coordinate diagram;

[0022] S312: Calculating corresponding motion information ratios based on variances of each counterclockwise rotation angle;

[0023] S313: Select the key point corresponding to the counterclockwise rotation angle with the largest motion information ratio as the recommended key point.

[0024] Preferably, the variance of the counterclockwise rotation angle is calculated by the following formula:

[0025]

[0026] The motion information ratio of the counterclockwise rotation angle is calculated by the following formula:

[0027]

[0028] In the above formula: 2 represents the variance of the counterclockwise rotation angle; R represents the counterclockwise rotation angle; u r represents the mean value of the counterclockwise rotation angle in the human body key point coordinate diagram; N represents the number of counterclockwise rotation angles in the human body key point coordinate diagram; I n Represents the motion information ratio of the nth counterclockwise rotation angle in the human body key point coordinate diagram; It represents the variance of the nth counterclockwise rotation angle in the human body key point coordinate diagram; e represents a natural constant.

[0029] Preferably, in step S4, the motion state information of the video to be tested includes limb motion speed, limb movement stability and limb movement trend.

[0030] Preferably, the limb movement speed is calculated by the following formula:

[0031]

[0032] In the above formula: V(N1, N2) represents the movement speed of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1; R1 and R2 respectively represent the counterclockwise rotation angles between the key points of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1; Fps represents the frame rate of the video to be tested.

[0033] Preferably, the stability of limb movements is determined by the following formula:

[0034]

[0035] In the above formula: S2 Indicates the stability characteristic value of the corresponding finger; when judging the stability of limb movements, S 2 Compare with the set stability threshold Y: If S 2 ≥Y, it means the limb movement is stable, otherwise, the limb movement is unstable; R={R1,R2,…R n} represents the counterclockwise rotation angle between the key points of the corresponding limbs in multiple optimal video frames, where n represents the number of optimal video frames; represents the mean of the counterclockwise rotation angles in R.

[0036] Preferably, the limb movement trend is determined by the following formula:

[0037] θ = R2 - R1;

[0038] In the above formula: θ represents the angle difference between the counterclockwise rotation angles R1 and R2 of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1. When judging the limb movement trend: if θ>0, it means that the limb rotates counterclockwise, if θ<0, it means that the limb rotates clockwise.

[0039] Preferably, in step S5, the motion standard information includes a motion speed standard range and a standard motion trend;

[0040] When the limb movement speed is within the standard range of movement speed, the limb movement trend corresponds to the standard movement trend and the limb movement is stable, the movement assessment result is passed; otherwise, the movement assessment fails.

[0041] Compared with the prior art, the human body motion assessment method of the present invention has the following beneficial effects:

[0042] 1. The present invention can effectively reflect the continuity and stability of the tester's movements by selecting multiple optimal video frames and calculating the motion state information of the video to be tested, and then generate a motion evaluation result by comparing the motion state information of the video to be tested with the motion standard information of the template video, which can ensure the accuracy of human motion evaluation.

[0043] 2. The present invention generates a human body key point coordinate diagram through skeleton analysis and posture analysis, and then selects multiple optimal video frames in combination with action evaluation standards, so that the selected multiple optimal video frames can be associated with the corresponding action evaluation standards, which can ensure the accuracy of the selection of the optimal video frames, thereby further ensuring the accuracy and effect of the action evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0045] Figure 1It is the logic block diagram of the human motion evaluation method;

[0046] Figure 2 Schematic diagram of ten counterclockwise rotation angles on the human body;

[0047] Figure 3 It is a schematic diagram of the limb angle between the right upper arm and the right forearm;

[0048] Figure 4 Schematic diagram of the counterclockwise rotation angle between the right upper arm and the right forearm. DETAILED DESCRIPTION

[0049] The following is a further detailed description through specific implementation methods:

[0050] Example:

[0051] First, the meaning of action evaluation criteria and action evaluation decision basis are explained.

[0052] Action assessment criteria: refers to the items that need to be assessed during action assessment.

[0053] Taking squat as an example, the movement evaluation criteria include: 1) whether the test rod is directly above the head; 2) whether the torso and calf are parallel or perpendicular to the ground; 3) whether the thigh is below the horizontal line when squatting; 4) whether the knees remain in the same direction as the feet.

[0054] Action evaluation decision basis: refers to the scoring criteria for action evaluation.

[0055] Taking squat as an example, the decision-making basis for action evaluation includes: 1) the test rod is directly above the head, the torso is parallel to the calf or perpendicular to the ground, the thighs are below the horizontal line when squatting, and the knees are kept in the same direction as the feet, and 3 points are scored; 2) Failure to complete the required action or placing a wooden board under the heels to complete the required action will score 2 points; 3) Failure to complete the required action even with a template under the heels will score 1 point; 4) Pain in any part of the body during the test will score 0 points.

[0056] Based on the above description, this embodiment discloses a human motion assessment method based on motion state information.

[0057] like Figure 1 As shown, the human motion evaluation method based on motion state information includes:

[0058] S1: Obtain the test video of the tester;

[0059] S2: Perform skeleton analysis and posture analysis on the video frames of the video to be tested, and generate a corresponding human body key point coordinate map; in this embodiment, the AlphaPose model proposed by Shanghai Jiao Tong University is used to perform skeleton analysis and posture analysis on the video frames of the video to be tested.

[0060] S3: Select the corresponding multi-frame optimal video frames based on the human body key point coordinate diagram and the corresponding action evaluation criteria;

[0061] S4: Calculating motion state information of the video to be tested based on multiple optimal video frames;

[0062] S5: Compare the motion state information of the video to be tested with the motion standard information of the template video to generate a corresponding motion evaluation result.

[0063] In the present invention, by selecting multiple optimal video frames and calculating the motion state information of the video to be tested, the motion continuity and motion stability of the tester can be effectively reflected, and then the motion evaluation result is generated by comparing the motion state information of the video to be tested with the motion standard information of the template video, which can ensure the accuracy of human motion evaluation. At the same time, the present invention generates a human key point coordinate diagram by skeleton analysis and posture analysis, and then selects multiple optimal video frames in combination with the motion evaluation standard, so that the selected multiple optimal video frames can be associated with the corresponding motion evaluation standard, which can ensure the accuracy of the selection of the optimal video frame, thereby further ensuring the accuracy and effect of the motion evaluation.

[0064] During the specific implementation process, when selecting the optimal multi-frame video frames, first, the angle information between each key point is calculated based on the human body key point coordinate map; then, the recommended key points in the human body key point coordinate map are determined based on the angle information between the key points; finally, the corresponding multi-frame optimal video frames are selected based on the recommended key points in the human body key point coordinate map and the action evaluation criteria.

[0065] Specifically, the angle information between key points is the counterclockwise rotation angle; combined with Figure 2 As shown, the types of counterclockwise rotation angles include the angle between the left forearm and the left upper arm, the angle between the left upper arm and the left shoulder, the angle between the left upper arm and the torso, the angle between the torso and the left thigh, the angle between the left thigh and the left calf, the angle between the right upper arm and the right forearm, the angle between the right shoulder and the right upper arm, the angle between the torso and the right upper arm, the angle between the torso and the right thigh, and the angle between the right thigh and the right calf.

[0066] Calculate the counterclockwise rotation angle by the following steps:

[0067] S301: Obtain key point coordinates A, B, C for calculating the counterclockwise rotation angle;

[0068] S302: Calculate the corresponding key point vector based on the key point coordinates A, B, and C and the key point vector

[0069] S303: The key point vector Rotate counterclockwise until it is aligned with the key point vector coincide; then the key point vector Rotate to keypoint vector The angle is taken as the corresponding counterclockwise rotation angle.

[0070] During the specific implementation process, the recommended key points are determined through the following steps:

[0071] S311: Calculate the variance of each counterclockwise rotation angle in the human body key point coordinate diagram;

[0072] S312: Calculating corresponding motion information ratios based on variances of each counterclockwise rotation angle;

[0073] S313: Select the key point corresponding to the counterclockwise rotation angle with the largest motion information ratio as the recommended key point.

[0074] In the specific implementation process, the variance of the counterclockwise rotation angle is calculated by the following formula:

[0075]

[0076] The motion information ratio of the counterclockwise rotation angle is calculated by the following formula:

[0077]

[0078] In the above formula: 2 represents the variance of the counterclockwise rotation angle; R represents the counterclockwise rotation angle; u r represents the mean value of the counterclockwise rotation angle in the human body key point coordinate diagram; N represents the number of counterclockwise rotation angles in the human body key point coordinate diagram; I n Represents the motion information ratio of the nth counterclockwise rotation angle in the human body key point coordinate diagram; It represents the variance of the nth counterclockwise rotation angle in the human body key point coordinate diagram; e represents a natural constant.

[0079] In the present invention, by calculating the angle information and selecting the recommended key points based on the angle information, and finally selecting multiple optimal video frames based on the recommended key points and the action evaluation criteria, the accuracy of the selection of the optimal video frames can be guaranteed, and the selected multiple optimal video frames can be associated with the corresponding action evaluation criteria, thereby ensuring the accuracy of the selection of the optimal video frames. At the same time, the present invention selects the recommended key points with large movement amplitude to participate in the selection of the optimal video frames. On the one hand, the recommended key points can accurately reflect the limbs that perform the main movement, thereby ensuring the accuracy of the action evaluation; on the other hand, the key points corresponding to the limbs with small movement amplitude or no movement are not involved in the calculation, which can reduce the amount of calculation for the action evaluation.

[0080] In addition, skeleton analysis and posture analysis of video frames obtain two-dimensional postures. The key points in the human body key point coordinate diagram are actually the projection of the real posture on the two-dimensional plane, which makes it difficult for simple limb angles to accurately represent the limb characteristics of an action. Figure 3 As shown in the figure, when the right arm is bent in front of the chest and bent at the side of the body, the limb angle between the right upper arm and the right forearm is the same. From the data, the limb characteristics of the right arm movement are the same, because the limb angle between the right upper arm and the right forearm is the same, but in fact, there are great differences between the two movements.

[0081] To this end, the present invention adds direction information, i.e., rotation direction, based on the limb angle, so that the generated counterclockwise rotation angle has both angle information and direction information (e.g. Figure 4 As shown in FIG. 1 , the problem of partial posture information loss when the real posture is projected onto a two-dimensional plane can be solved, and the characteristics of the action limbs can be accurately represented, thereby ensuring the accuracy of human motion evaluation. At the same time, the ten counterclockwise rotation angles designed by the present invention can basically cover the important action limb characteristics of the human posture, thereby further ensuring the effect of human motion evaluation.

[0082] In the specific implementation process, the motion state information of the video to be tested includes the limb movement speed, limb movement stability and limb movement trend. The motion standard information includes the standard range of movement speed and the standard movement trend; when the limb movement speed is within the standard range of movement speed, the limb movement trend corresponds to the standard movement trend and the limb movement is stable, the motion evaluation result is passed; otherwise, the motion evaluation fails.

[0083] Limb movement speed is calculated using the following formula:

[0084]

[0085] In the above formula: V(N1, N2) represents the movement speed of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1; R1 and R2 respectively represent the counterclockwise rotation angles between the key points of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1; Fps represents the frame rate of the video to be tested.

[0086] In the specific implementation process, the stability of limb movements is determined by the following formula:

[0087]

[0088] In the above formula: S 2 Indicates the stability characteristic value of the corresponding finger; when judging the stability of limb movements, S 2 Compare with the set stability threshold Y: If S 2≥Y, it means the limb movement is stable, otherwise, the limb movement is unstable; R={R1,R2,…R n} represents the counterclockwise rotation angle between the key points of the corresponding limbs in multiple optimal video frames, where n represents the number of optimal video frames; represents the mean of the counterclockwise rotation angles in R.

[0089] In the specific implementation process, the limb movement trend is determined by the following formula:

[0090] θ = R2 - R1;

[0091] In the above formula: θ represents the angle difference between the counterclockwise rotation angles R1 and R2 of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1. When judging the limb movement trend: if θ>0, it means that the limb rotates counterclockwise, if θ<0, it means that the limb rotates clockwise.

[0092] In the present invention, the motion state information of the tester can be well reflected through the limb movement speed, limb movement stability and limb movement trend, and thus the tester's movement continuity and stability can be effectively reflected. Subsequently, the motion state information of the video to be tested and the motion standard information of the template video are compared to generate the motion evaluation result, thereby ensuring the accuracy of the human body motion evaluation. At the same time, the calculation formula and method of the present invention can well calculate the limb movement speed, limb movement stability and limb movement trend, thereby better reflecting the tester's motion state information.

[0093] Specifically, when generating a human body key point coordinate map, firstly, the corresponding video frame is input into a pre-trained posture estimation model, and the corresponding heat map is output; then, the key point coordinates are calculated by the heat map to obtain the corresponding human body key point coordinate map;

[0094] When training the posture estimation model, a posture data set for training is obtained; then the labels pre-marked on the training graph in the posture data set are converted into corresponding heat map labels to obtain corresponding label heat maps; finally, the posture estimation model is trained based on the label heat maps;

[0095] When generating a label heat map, first set the size of the label heat map (W h ×H h ) to generate a size W h ×H h The all-zero matrix is ​​then used to calculate the heat distribution of the pre-labeled labels on the label heat map to generate the corresponding label heat map;

[0096]

[0097] When calculating the coordinates of the key points, get the size Wh ×H h The heat map is then reduced to 1*W h *H h ; Then calculate the maximum heat value index of the corresponding key point in the heat map by the following formula; Finally, calculate the coordinates of the corresponding key point by combining the index corresponding to the maximum heat value in the heat map with the size of the heat map. Specifically, divide the index by the corresponding W h , the quotient is of size W h ×H h The number of rows of key points is x, and the remainder is of size W h ×H h The number of columns of the key points y, that is, the coordinates of the key points (x, y);

[0098]

[0099] In the above formula: g represents the heat value; x0 and y0 represent the real coordinates of the pre-annotated label; x and y represent the coordinates of the label in the label heat map; σ represents the standard deviation, which is 2 or 3; e represents a natural constant; i and j represent the index of the one-dimensional heat map respectively; x i 、x j represents the heat value corresponding to index i and j; β represents the calibration coefficient.

[0100] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes can be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims. At the same time, common sense such as the well-known specific structures and characteristics in the embodiments are not described in detail here. Finally, the scope of protection claimed by the present invention shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A human motion assessment method based on motion state information, characterized in that: include: S1: Obtain the test video of the tester; S2: Perform skeleton analysis and posture analysis on the video frames of the video to be tested, and generate a corresponding human body key point coordinate map; S3: Select the corresponding multi-frame optimal video frames based on the human body key point coordinate diagram and the corresponding action evaluation criteria; S4: Calculating motion state information of the video to be tested based on multiple optimal video frames; In step S4, the motion state information of the video to be tested includes limb motion speed, limb motion stability and limb motion trend; Limb movement speed is calculated using the following formula: In the above formula: V(N1, N2) represents the movement speed of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1; R1 and R2 represent the counterclockwise rotation angle between the key points of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1 respectively; Fps represents the frame rate of the video to be tested; The stability of limb movements is determined by the following formula: In the above formula: S 2 Indicates the stability characteristic value of the corresponding finger; when judging the stability of limb movements, S 2 Compare with the set stability threshold Y: If S 2 ≥Y, it means the limb movement is stable, otherwise, the limb movement is unstable; R={R1,R2,…R n } represents the counterclockwise rotation angle between the key points of the corresponding limbs in multiple optimal video frames, where n represents the number of optimal video frames; represents the mean value of the counterclockwise rotation angle in R; The limb movement trend is determined by the following formula: θ = R2 - R1; In the above formula: θ represents the angle difference between the counterclockwise rotation angles R1 and R2 of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1. When judging the movement trend of the limb: if θ>0, it means that the limb rotates counterclockwise, and if θ<0, it means that the limb rotates clockwise; S5: Compare the motion state information of the video to be tested with the motion standard information of the template video to generate a corresponding motion evaluation result.

2. The human motion assessment method based on motion state information as claimed in claim 1, characterized in that: In step S3, when selecting multiple optimal video frames, first calculate the angle information between each key point based on the human body key point coordinate map; then determine the recommended key points in the human body key point coordinate map based on the angle information between the key points; finally, select the corresponding multiple optimal video frames based on the recommended key points in the human body key point coordinate map and the action evaluation criteria.

3. The human motion assessment method based on motion state information as claimed in claim 2, characterized in that: The angle information between key points is the counterclockwise rotation angle; Calculate the counterclockwise rotation angle by the following steps: S301: Obtain key point coordinates A, B, C for calculating the counterclockwise rotation angle; S302: Calculate the corresponding key point vector based on the key point coordinates A, B, and C and the key point vector S303: The key point vector Rotate counterclockwise until it is aligned with the key point vector coincide; then the key point vector Rotate to keypoint vector The angle is taken as the corresponding counterclockwise rotation angle.

4. The human motion assessment method based on motion state information as claimed in claim 3, characterized in that: Determine the recommended key points through the following steps: S311: Calculate the variance of each counterclockwise rotation angle in the human body key point coordinate diagram; S312: Calculating corresponding motion information ratios based on variances of each counterclockwise rotation angle; S313: Select the key point corresponding to the counterclockwise rotation angle with the largest motion information ratio as the recommended key point.

5. The human motion assessment method based on motion state information as claimed in claim 4, characterized in that: The variance of the counterclockwise rotation angle is calculated using the following formula: The motion information ratio of the counterclockwise rotation angle is calculated by the following formula: In the above formula: 2 represents the variance of the counterclockwise rotation angle; R represents the counterclockwise rotation angle; u r represents the mean value of the counterclockwise rotation angle in the human body key point coordinate diagram; N represents the number of counterclockwise rotation angles in the human body key point coordinate diagram; I n Represents the motion information ratio of the nth counterclockwise rotation angle in the human body key point coordinate diagram; It represents the variance of the nth counterclockwise rotation angle in the human body key point coordinate diagram; e represents a natural constant.

6. The human motion assessment method based on motion state information according to claim 1, characterized in that: In step S5, the motion standard information includes a motion speed standard range and a standard motion trend; When the limb movement speed is within the standard range, the limb movement trend corresponds to the standard movement trend, and the limb movement is stable, the movement assessment result is passed; Otherwise, the action evaluation fails.

Images