An action evaluation method based on optimal frame selection

Abstract

The present invention relates to the technical field of computer vision image processing, and in particular to an action evaluation method based on optimal frame selection, comprising: obtaining a video to be tested of a tester; performing skeleton analysis and posture analysis on the video frame of the video to be tested, and generating a corresponding human key point coordinate diagram; selecting a corresponding optimal video frame based on the human key point coordinate diagram in combination with a corresponding action evaluation standard; selecting a corresponding optimal template video frame from a template video based on the optimal video frame; calculating the similarity between the optimal video frame and the optimal template video frame, and generating a corresponding action evaluation result based on the calculation result of the similarity. The action evaluation method in the present invention can reduce the evaluation cost and improve the evaluation efficiency, thereby improving the effect of action evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image processing, and in particular to an action evaluation method based on optimal frame selection. Background Art

[0002] With the widespread application of Internet big data information technology, the application scenarios of human behavior detection and recognition technology based on visual images are increasing. By analyzing the movement characteristics of human expressions, postures, etc., it can provide rich identification feature information for the detection and prediction of people's behavior in public places or specific activity spaces, which is an important part of human activity big data information.

[0003] For example, in the fields of sports competition and health screening, it is necessary to identify human body movements in order to evaluate the standard conditions of the movements. Movement evaluation needs to be based on movement evaluation standards and movement evaluation decision-making basis. In the early days, the commonly used evaluation method was that the evaluator observed the test subject's movements with his eyes, and manually compared the test subject's movements with the standard movement patterns to give scores; at the same time, the evaluator used a handheld camera to extract the video, and then took a screenshot of the video and saved it for reference. The above existing methods not only waste manpower and material resources, but also the evaluation results are not objective and accurate due to subjective reasons during human judgment.

[0004] With the development of computer technology, methods for evaluating actions based on videos have emerged in the prior art. For example, the Chinese patent with publication number CN108205654B discloses "A video-based action detection method and device", which includes: detecting limb key points in the video frame of the video to be detected; obtaining corresponding behavioral features based on the limb key points; performing action completion detection based on the behavioral features and the global action evaluation model to obtain a global detection result, wherein the global action evaluation model is used to detect the completion of human body actions in the video relative to the standard action.

[0005] The motion detection method in the above-mentioned existing scheme obtains behavioral features through the key points of the limbs in the video frame, and then performs motion evaluation based on the behavioral features and the global motion evaluation model. Motion evaluation generally compares each video frame in the video to be tested with the corresponding video frame in the template video and calculates the similarity, and then judges the completion of the action based on the similarity. However, for simple actions, the motion evaluation can be completed by comparing one or more video frames. For example, when evaluating shoulder flexibility, the motion evaluation can be achieved by selecting the video frame corresponding to the minimum distance between the two fists at the back and comparing it with the template video frame. However, the existing motion evaluation method needs to compare a large number of video frames in the video to be tested with the corresponding video frames of the template video, so that a lot of "useless" calculations are done during the motion evaluation, resulting in a waste of computing resources, increasing the cost of motion evaluation, and reducing the evaluation efficiency. Therefore, how to design a motion evaluation method that can reduce the evaluation cost and improve the evaluation efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is: how to provide a motion evaluation method based on optimal frame selection so as to reduce the cost of motion evaluation and improve the efficiency of motion evaluation, thereby improving the effect of motion evaluation.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] An action evaluation method based on optimal frame selection includes the following steps:

[0009] S1: Obtain the test video of the tester;

[0010] S2: Perform skeleton analysis and posture analysis on the video frames of the video to be tested, and generate a corresponding human body key point coordinate map;

[0011] S3: Select the corresponding optimal video frame based on the human body key point coordinate diagram combined with the corresponding action evaluation criteria;

[0012] S4: selecting a corresponding optimal template video frame from the template video based on the optimal video frame;

[0013] S5: Calculate the similarity between the optimal video frame and the optimal template video frame, and generate a corresponding action evaluation result based on the calculation result of the similarity.

[0014] Preferably, in step S2, after the human body key point coordinate map is generated, median filtering is performed on the key points in the human body key point coordinate map.

[0015] Preferably, in step S3, when selecting the optimal video frame, the angle information, distance information and position relationship information between each key point are calculated based on the human body key point coordinate diagram, and then the corresponding optimal video frame is selected based on the angle information, distance information or position relationship information combined with the action evaluation standard.

[0016] Preferably, the angle information between the key points is a counterclockwise rotation angle;

[0017] Calculate the counterclockwise rotation angle by the following steps:

[0018] S301: Obtain key point coordinates A, B, C for calculating the counterclockwise rotation angle;

[0019] S302: Calculate the corresponding key point vector based on the key point coordinates A, B, and C and the key point vector

[0020] S303: The key point vector Rotate counterclockwise until it is aligned with the key point vector coincide; then the key point vector Rotate to keypoint vector The angle is taken as the corresponding counterclockwise rotation angle.

[0021] Preferably, in step S4, when selecting the optimal template video frame, the corresponding key point feature values ​​are calculated based on the human body key point coordinate map of the optimal video frame, and then the video frame corresponding to the preset feature value and the key point feature value is matched from the template video as the optimal template video frame of the optimal video frame.

[0022] Preferably, the key point feature values ​​are calculated based on the counterclockwise rotation angle between the key points;

[0023] The calculation formula of the key point eigenvalue is as follows:

[0024]

[0025] in,

[0026] In the above formula: I represents the key point feature value of the optimal video frame; represents the variance of the nth counterclockwise rotation angle in the optimal video frame; R represents the counterclockwise rotation angle; u r represents the mean of all counterclockwise rotation angles in the optimal video frame; N represents the number of counterclockwise rotation angles in the optimal video frame; λ represents the set adjustment parameter, and its initial value is 0.1.

[0027] Preferably, in step S5, after the similarity is calculated, the value of the adjustment parameter λ is adjusted based on the calculation result of the similarity.

[0028] Preferably, in step S5, the similarity between the optimal video frame and the optimal template video frame is expressed by calculating the similarity of the counterclockwise rotation angle between the optimal video frame and the optimal template video frame.

[0029] Preferably, the similarity of the counterclockwise rotation angle between the optimal video frame and the optimal template video frame is calculated by the following steps:

[0030] S501: Determine the weight value of each counterclockwise rotation angle based on the variance of each counterclockwise rotation angle in the optimal video frame;

[0031] S502: Based on each counterclockwise rotation angle in the optimal video frame and its weight value and the corresponding counterclockwise rotation angle in the optimal template video frame, the corresponding similarity is calculated in combination with the following formula:

[0032]

[0033] in,

[0034] In the above formula: S represents similarity; R k represents the kth counterclockwise rotation angle in the optimal template video frame; R′ K represents the kth counterclockwise rotation angle in the optimal video frame; w k represents the weight value of the kth counterclockwise rotation angle in the optimal video frame; n represents the number of counterclockwise rotation angles in the optimal video frame and the optimal template video frame; represents the variance of the kth counterclockwise rotation angle in the optimal video frame; e represents a natural constant.

[0035] Preferably, in step S501, the larger the variance of the counterclockwise rotation angle is, the larger the corresponding weight value is, and the sum of the weight values ​​of each counterclockwise rotation angle is 1.

[0036] Compared with the prior art, the action evaluation method in the present invention has the following beneficial effects:

[0037] 1. The present invention avoids the problem of "useless" calculations during action evaluation by selecting the optimal video frame and matching the optimal template video frame to calculate the similarity, which can avoid the waste of computing resources. In addition, this method greatly reduces the amount of calculation during action evaluation, thereby reducing the cost of action evaluation and improving the efficiency of action evaluation.

[0038] 2. The present invention generates a human body key point coordinate diagram by means of skeleton analysis and posture analysis, and then selects the optimal video frame and matches the optimal template video frame in combination with the action evaluation standard, so that the selection of the optimal video frame and the optimal template video frame can be associated with the corresponding action evaluation standard, which can ensure the accuracy of the selection of the optimal video frame and the optimal template video frame, thereby ensuring the accuracy and effect of the action evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0040] Figure 1 It is the logic block diagram of the action evaluation method;

[0041] Figure 2 Schematic diagram of ten counterclockwise rotation angles on the human body;

[0042] Figure 3 It is a schematic diagram of the limb angle between the right upper arm and the right forearm;

[0043] Figure 4 Schematic diagram of the counterclockwise rotation angle between the right upper arm and the right forearm. DETAILED DESCRIPTION

[0044] The following is a further detailed description through specific implementation methods:

[0045] Example:

[0046] First, the meaning of action evaluation criteria and action evaluation decision basis are explained.

[0047] Action assessment criteria: refers to the items that need to be assessed during action assessment.

[0048] Taking squat as an example, the movement evaluation criteria include: 1) whether the test rod is directly above the head; 2) whether the torso and calf are parallel or perpendicular to the ground; 3) whether the thigh is below the horizontal line when squatting; 4) whether the knees remain in the same direction as the feet.

[0049] Action evaluation decision basis: refers to the scoring criteria for action evaluation.

[0050] Taking squat as an example, the decision-making basis for action evaluation includes: 1) the test rod is directly above the head, the torso is parallel to the calf or perpendicular to the ground, the thighs are below the horizontal line when squatting, and the knees are kept in the same direction as the feet, and 3 points are scored; 2) Failure to complete the required action or placing a wooden board under the heels to complete the required action will score 2 points; 3) Failure to complete the required action even with a template under the heels will score 1 point; 4) Pain in any part of the body during the test will score 0 points.

[0051] Based on the above description, this embodiment discloses an action evaluation method based on optimal frame selection.

[0052] like Figure 1 As shown, the action evaluation method based on optimal frame selection includes the following steps:

[0053] S1: Obtain the test video of the tester;

[0054] S2: Perform skeleton analysis and posture analysis on the video frames of the video to be tested, and generate a corresponding human body key point coordinate map; in this embodiment, the AlphaPose model proposed by Shanghai Jiao Tong University is used to perform skeleton analysis and posture analysis on the video frames of the video to be tested.

[0055] S3: Select the corresponding optimal video frame based on the human body key point coordinate diagram and the corresponding action evaluation standard; the optimal video frame can be one frame or multiple frames.

[0056] S4: selecting a corresponding optimal template video frame from the template video based on the optimal video frame;

[0057] S5: Calculate the similarity between the optimal video frame and the optimal template video frame, and generate a corresponding action evaluation result based on the calculation result of the similarity.

[0058] In the present invention, by selecting the optimal video frame and matching the optimal template video frame to calculate the similarity, the problem of "useless" calculation during action evaluation is avoided, the waste of computing resources can be avoided, and this method greatly reduces the amount of calculation during action evaluation, thereby reducing the cost of action evaluation and improving the efficiency of action evaluation. At the same time, the present invention generates a human body key point coordinate diagram by means of skeleton analysis and posture analysis, and then selects the optimal video frame and matches the optimal template video frame in combination with the action evaluation standard, so that the selection of the optimal video frame and the optimal template video frame can be associated with the corresponding action evaluation standard, and the selection accuracy of the optimal video frame and the optimal template video frame can be guaranteed, thereby ensuring the accuracy and effect of the action evaluation.

[0059] In the specific implementation process, after the human body key point coordinate map is generated, the key points in the human body key point coordinate map are subjected to median filtering. The median filtering is used to remove the jitter generated during the skeleton analysis and posture analysis process to ensure the clarity of the key points in the human body key point coordinate map.

[0060] During the specific implementation process, when selecting the optimal video frame, the angle information, distance information and position relationship information between each key point are calculated based on the human body key point coordinate diagram, and then the corresponding optimal video frame is selected based on the angle information, distance information or position relationship information combined with the action evaluation standard.

[0061] The angle information between key points is the counterclockwise rotation angle; combined Figure 2 As shown, the types of counterclockwise rotation angles include the angle between the left forearm and the left upper arm, the angle between the left upper arm and the left shoulder, the angle between the left upper arm and the torso, the angle between the torso and the left thigh, the angle between the left thigh and the left calf, the angle between the right upper arm and the right forearm, the angle between the right shoulder and the right upper arm, the angle between the torso and the right upper arm, the angle between the torso and the right thigh, and the angle between the right thigh and the right calf.

[0062] Calculate the counterclockwise rotation angle by the following steps:

[0063] S301: Obtain key point coordinates A, B, C for calculating the counterclockwise rotation angle;

[0064] S302: Calculate the corresponding key point vector based on the key point coordinates A, B, and C and the key point vector

[0065] S303: The key point vector Rotate counterclockwise until it is aligned with the key point vector coincide; then the key point vector Rotate to keypoint vector The angle is taken as the corresponding counterclockwise rotation angle.

[0066] The distance information between key points is calculated by the following formula:

[0067]

[0068] In the above formula: d(A,B) represents the key point A(x1,x2,…,x n ) and B(y1,y2,…,y n ) distance information.

[0069] The positional relationship information between key points includes slope and difference.

[0070] The skeleton analysis and posture analysis of the video frame results in a two-dimensional posture. The key points in the human body key point coordinate diagram are actually the projection of the real posture on the two-dimensional plane, which makes it difficult for a simple limb angle to accurately represent the limb characteristics of an action. Figure 3 As shown in the figure, when the right arm is bent in front of the chest and bent at the side of the body, the limb angle between the right upper arm and the right forearm is the same. From the data, the limb characteristics of the right arm movement are the same, because the limb angle between the right upper arm and the right forearm is the same, but in fact, there are great differences between the two movements.

[0071] To this end, the present invention adds direction information, i.e., rotation direction, based on the limb angle, so that the generated counterclockwise rotation angle has both angle information and direction information (e.g. Figure 4 As shown in FIG. 1 , the problem of partial posture information loss when the real posture is projected onto a two-dimensional plane can be solved, and the characteristics of the action limbs can be accurately represented, thereby ensuring the accuracy of human motion evaluation. At the same time, the ten counterclockwise rotation angles designed by the present invention can basically cover the important action limb characteristics of the human posture, thereby further ensuring the effect of human motion evaluation.

[0072] During the specific implementation process, when selecting the optimal template video frame, the corresponding key point feature values ​​are calculated based on the human body key point coordinate map of the optimal video frame, and then the video frame corresponding to the preset feature value and the key point feature value is matched from the template video as the optimal template video frame of the optimal video frame.

[0073] Specifically, the key point eigenvalues ​​are calculated based on the counterclockwise rotation angle between the key points;

[0074] The calculation formula of the key point eigenvalue is as follows:

[0075]

[0076] in,

[0077] In the above formula: I represents the key point feature value of the optimal video frame; represents the variance of the nth counterclockwise rotation angle in the optimal video frame; R represents the counterclockwise rotation angle; u r represents the mean of all counterclockwise rotation angles in the optimal video frame; N represents the number of counterclockwise rotation angles in the optimal video frame; λ represents the set adjustment parameter, and its initial value is 0.1.

[0078] After the similarity is calculated, the value of the adjustment parameter λ is adjusted based on the calculation result of the similarity.

[0079] In the present invention, by calculating the key point feature values ​​and matching the preset feature values, the optimal template video frame corresponding to the optimal video frame can be accurately matched, thereby further ensuring the accuracy of human motion evaluation under the premise of reducing the motion evaluation cost and improving the motion evaluation efficiency. At the same time, the present invention can continuously optimize the selection strategy of the optimal template video frame by adjusting the adjustment parameters, thereby better improving the selection accuracy of the optimal template video frame.

[0080] In a specific implementation process, the similarity between the optimal video frame and the optimal template video frame is expressed by calculating the similarity of the counterclockwise rotation angle between the optimal video frame and the optimal template video frame.

[0081] The similarity of the counterclockwise rotation angle between the optimal video frame and the optimal template video frame is calculated by the following steps:

[0082] S501: Determine the weight value of each counterclockwise rotation angle based on the variance of each counterclockwise rotation angle in the optimal video frame; the larger the variance of the counterclockwise rotation angle, the larger the corresponding weight value, and the sum of the weight values ​​of each counterclockwise rotation angle is 1.

[0083] S502: Based on each counterclockwise rotation angle in the optimal video frame and its weight value and the corresponding counterclockwise rotation angle in the optimal template video frame, the corresponding similarity is calculated in combination with the following formula:

[0084]

[0085] in,

[0086] In the above formula: S represents similarity; R k represents the kth counterclockwise rotation angle in the optimal template video frame; R′ K represents the kth counterclockwise rotation angle in the optimal video frame; w k represents the weight value of the kth counterclockwise rotation angle in the optimal video frame; n represents the number of counterclockwise rotation angles in the optimal video frame and the optimal template video frame; represents the variance of the kth counterclockwise rotation angle in the optimal video frame; e represents a natural constant.

[0087] In the present invention, by determining the weight value through the variance and combining the difference between the counterclockwise rotation angles, the similarity of the counterclockwise rotation angles can be well calculated, thereby ensuring the effect of action evaluation. At the same time, since the counterclockwise rotation angle can accurately represent the characteristics of the action limbs, the present invention can effectively represent the similarity between the optimal video frame and the optimal template video frame through the similarity between the counterclockwise rotation angles, thereby better assisting in completing the human body action evaluation.

[0088] Specifically, when generating a human body key point coordinate map, firstly, the corresponding video frame is input into a pre-trained posture estimation model, and the corresponding heat map is output; then, the key point coordinates are calculated by the heat map to obtain the corresponding human body key point coordinate map;

[0089] When training the posture estimation model, a posture data set for training is obtained; then the labels pre-marked on the training graph in the posture data set are converted into corresponding heat map labels to obtain corresponding label heat maps; finally, the posture estimation model is trained based on the label heat maps;

[0090] When generating a label heat map, first set the size of the label heat map (W h ×H h ) to generate a size W h ×H hThe all-zero matrix is ​​then used to calculate the heat distribution of the pre-labeled labels on the label heat map to generate the corresponding label heat map;

[0091]

[0092] When calculating the coordinates of the key points, get the size W h ×H h The heat map is then reduced to 1*W h *H h ; Then calculate the maximum heat value index of the corresponding key point in the heat map by the following formula; Finally, calculate the coordinates of the corresponding key point by combining the index corresponding to the maximum heat value in the heat map with the size of the heat map. Specifically, divide the index by the corresponding W h , the quotient is of size W h ×H h The number of rows of key points is x, and the remainder is of size W h ×H h The number of columns of the key points y, that is, the coordinates of the key points (x, y);

[0093]

[0094] In the above formula: g represents the heat value; x0 and y0 represent the real coordinates of the pre-annotated label; x and y represent the coordinates of the label in the label heat map; σ represents the standard deviation, which is 2 or 3; e represents a natural constant; i and j represent the index of the one-dimensional heat map respectively; x i 、x j represents the heat value corresponding to index i and j; β represents the calibration coefficient.

[0095] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes can be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims. At the same time, common sense such as the well-known specific structures and characteristics in the embodiments are not described in detail here. Finally, the scope of protection claimed by the present invention shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. An action evaluation method based on optimal frame selection, characterized in that: The following steps are involved: S1: Obtain the test video of the tester; S2: Perform skeleton analysis and posture analysis on the video frames of the video to be tested, and generate a corresponding human body key point coordinate map; S3: Select the corresponding optimal video frame based on the human body key point coordinate diagram combined with the corresponding action evaluation criteria; In step S3, when selecting the optimal video frame, the angle information, distance information and position relationship information between the key points are calculated based on the human body key point coordinate diagram, and then the corresponding optimal video frame is selected based on the angle information, distance information or position relationship information combined with the action evaluation standard; S4: selecting a corresponding optimal template video frame from the template video based on the optimal video frame; S5: Calculate the similarity between the optimal video frame and the optimal template video frame, and generate a corresponding action evaluation result based on the calculation result of the similarity; In step S5, the similarity between the optimal video frame and the optimal template video frame is expressed by calculating the similarity of the counterclockwise rotation angle between the optimal video frame and the optimal template video frame; The similarity of the counterclockwise rotation angle between the optimal video frame and the optimal template video frame is calculated by the following steps: S501: Determine the weight value of each counterclockwise rotation angle based on the variance of each counterclockwise rotation angle in the optimal video frame; S502: Based on each counterclockwise rotation angle in the optimal video frame and its weight value and the corresponding counterclockwise rotation angle in the optimal template video frame, the corresponding similarity is calculated in combination with the following formula: in, In the above formula: S represents similarity; R k represents the kth counterclockwise rotation angle in the optimal template video frame; R′ K represents the kth counterclockwise rotation angle in the optimal video frame; w k represents the weight value of the kth counterclockwise rotation angle in the optimal video frame; n represents the number of counterclockwise rotation angles in the optimal video frame and the optimal template video frame; represents the variance of the kth counterclockwise rotation angle in the optimal video frame; e represents a natural constant.

2. The motion evaluation method based on optimal frame selection according to claim 1, characterized in that: In step S2, after the human body key point coordinate map is generated, median filtering is performed on the key points in the human body key point coordinate map.

3. The motion evaluation method based on optimal frame selection according to claim 1, characterized in that: The angle information between key points is the counterclockwise rotation angle; Calculate the counterclockwise rotation angle by the following steps: S301: Obtain key point coordinates A, B, C for calculating the counterclockwise rotation angle; S302: Calculate the corresponding key point vector based on the key point coordinates A, B, and C and the key point vector S303: The key point vector Rotate counterclockwise until it is aligned with the key point vector coincide; then the key point vector Rotate to keypoint vector The angle is taken as the corresponding counterclockwise rotation angle.

4. The motion evaluation method based on optimal frame selection according to claim 3, characterized in that: In step S4, when selecting the optimal template video frame, the corresponding key point feature values ​​are calculated based on the human body key point coordinate map of the optimal video frame, and then the video frame corresponding to the preset feature value and the key point feature value is matched from the template video as the optimal template video frame of the optimal video frame.

5. The motion evaluation method based on optimal frame selection according to claim 4, characterized in that: The key point eigenvalues ​​are calculated based on the counterclockwise rotation angle between the key points; The calculation formula of the key point eigenvalue is as follows: in, In the above formula: I represents the key point feature value of the optimal video frame; represents the variance of the nth counterclockwise rotation angle in the optimal video frame; R represents the counterclockwise rotation angle; u r represents the mean of all counterclockwise rotation angles in the optimal video frame; N represents the number of counterclockwise rotation angles in the optimal video frame; λ represents the set adjustment parameter, and its initial value is 0.

1.

6. The motion evaluation method based on optimal frame selection according to claim 5, characterized in that: In step S5, after the similarity is calculated, the value of the adjustment parameter λ is adjusted based on the calculation result of the similarity.

7. The motion evaluation method based on optimal frame selection according to claim 1, characterized in that: In step S501, the larger the variance of the counterclockwise rotation angle is, the larger the corresponding weight value is, and the sum of the weight values ​​of each counterclockwise rotation angle is 1.

Images