A human motion estimation method for multi-view video images

Abstract

The present invention relates to the technical field of computer vision image processing, and in particular to a method for evaluating human motion in multi-view video images, comprising: obtaining videos to be tested under multiple viewpoints of a tester; selecting key video frames from the videos to be tested under each viewpoint; splicing the key video frames under each viewpoint to obtain a corresponding multi-view spliced ​​video; selecting corresponding template video frames from a multi-view template video based on the multi-view spliced ​​video to construct a multi-view template video; completing human motion evaluation based on the multi-view spliced ​​video and the corresponding multi-view template video to generate corresponding motion evaluation results. The present invention provides a method for evaluating human motion that can be applied to multi-view video images, so as to solve the problem of motion being blocked due to the limitation of the shooting viewpoint, thereby improving the effect of human motion evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image processing, and in particular to a method for evaluating human motion in multi-view video images. Background Art

[0002] With the widespread application of Internet big data information technology, the application scenarios of human behavior detection and recognition technology based on visual images are increasing. By analyzing the movement characteristics of human expressions, postures, etc., it can provide rich identification feature information for the detection and prediction of people's behavior in public places or specific activity spaces, which is an important part of human activity big data information.

[0003] For example, in the fields of sports competition and health screening, it is necessary to identify human movements in order to evaluate the standard conditions of the movements. Movement evaluation needs to be based on movement evaluation standards and movement evaluation decision-making basis. In the early days, the commonly used evaluation method was that the evaluator observed the test subject's movements with his eyes, and manually compared the test subject's movements with the standard movement patterns to give scores; at the same time, the evaluator used a handheld camera to extract the video, and then took a screenshot of the video and saved it for reference. The above existing methods not only waste manpower and material resources, but also the evaluation results are not objective and accurate due to subjective reasons during human judgment.

[0004] With the development of computer technology, methods for evaluating actions based on videos have emerged in the prior art. For example, the Chinese patent with publication number CN108205654B discloses "A video-based action detection method and device", which includes: detecting limb key points in the video frame of the video to be detected; obtaining corresponding behavioral features based on the limb key points; performing action completion detection based on the behavioral features and the global action evaluation model to obtain a global detection result, wherein the global action evaluation model is used to detect the completion of human body actions in the video relative to the standard action.

[0005] The motion detection method in the above-mentioned existing scheme obtains behavioral features based on the limb key points in the video frame, and then performs motion evaluation based on the behavioral features and the global motion evaluation model. Motion evaluation generally compares each video frame in the video to be tested with the corresponding video frame in the template video and calculates the similarity, and then judges the completion of the action based on the similarity. However, for some more complex actions, it is necessary to evaluate the action in multiple directions to avoid the problem of action being blocked due to the limitation of the shooting angle. However, the existing motion evaluation method is mainly for the motion evaluation of a single perspective. When it is applied to the motion evaluation scene of multi-perspective video images, there is a problem that the perspective of the video image is difficult to effectively switch, resulting in poor effect of human motion evaluation. Therefore, how to design a human motion evaluation method that can be applied to multi-perspective video images is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is: how to provide a human motion assessment method that can be applied to multi-view video images to solve the problem of motion being blocked due to limited shooting angle, thereby improving the effect of human motion assessment.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A method for assessing human motion in multi-view video images comprises the following steps:

[0009] S1: Obtain the video to be tested from multiple perspectives of the tester;

[0010] S2: Select key video frames from the video to be tested at each viewing angle;

[0011] S3: stitching the key video frames under each viewing angle to obtain the corresponding multi-view stitching video;

[0012] S4: Select corresponding template video frames from the multi-view template video based on the multi-view stitching video to construct a multi-view template video;

[0013] S5: Complete human motion evaluation based on the multi-view stitched video and the corresponding multi-view template video to generate corresponding motion evaluation results.

[0014] Preferably, in step S2, the key video frame is selected by the following steps:

[0015] S201: performing skeleton analysis and posture analysis on video frames of the video to be tested at each viewing angle, and generating a human body key point coordinate map corresponding to the video frame;

[0016] S202: Calculate the confidence of each key point in the human body key point coordinate map;

[0017] S203: Determine the target key point according to the corresponding action evaluation standard;

[0018] S204: Starting from one perspective, video frames whose confidence of target key points is greater than or equal to a set threshold are selected in chronological order as key video frames; if the confidence of target key points in the current perspective is less than the set threshold, video frames whose confidence of target key points in other perspectives is greater than or equal to the set threshold are selected as key video frames;

[0019] S205: Repeat step S204 until the key video frames under each viewing angle are selected.

[0020] Preferably, in step S201, firstly, the corresponding video frame is input into a pre-trained posture estimation model, and a corresponding heat map is output; then, the key point coordinates are calculated by the heat map to obtain a corresponding human body key point coordinate map;

[0021] Among them, when training the posture estimation model, a posture data set for training is obtained; then the pre-annotated labels on the training image in the posture data set are converted into corresponding heat map labels to obtain the corresponding label heat map; finally, the posture estimation model is trained based on the label heat map; when calculating the coordinates of the key points, the heat map is first obtained and the dimension of the heat map is reduced; then the maximum heat value index of the corresponding key point in the heat map is calculated; finally, the coordinates of the corresponding key point are calculated by the index corresponding to the maximum heat value in the heat map combined with the heat map size.

[0022] Preferably, in step S202, when calculating the confidence of the key point, the key point heat map is firstly reduced in dimension, and then the corresponding confidence is calculated by the following formula;

[0023]

[0024] In the above formula: e represents the natural constant; β represents the calibration coefficient; i and j represent the index of the one-dimensional heat map respectively; x i 、x j Indicates the heat value corresponding to index i and j.

[0025] Preferably, in step S3, after the key video frames under each viewing angle are selected, the key video frames under each viewing angle are spliced ​​in chronological order to obtain a multi-view spliced ​​video.

[0026] Preferably, in step S5, the human body motion assessment is completed by the following steps:

[0027] S501: performing skeleton analysis and posture analysis on key video frames of the multi-view stitched video to generate a human body key point coordinate map corresponding to the key video frames;

[0028] S502: Select multiple key video frames as optimal video frames based on the human body key point coordinate diagram and corresponding action evaluation criteria;

[0029] S503: selecting corresponding multi-frame template video frames from the multi-view template video as optimal template video frames based on the optimal video frames of each frame;

[0030] S504: Calculating motion state information of the multi-view stitching video and motion standard information of the multi-view template video based on the optimal video frames of each frame and the optimal template video frames of each frame;

[0031] S505: Compare the motion state information of the multi-view stitching video with the motion standard information of the multi-view template video to generate a corresponding motion evaluation result.

[0032] Preferably, in step S502, when selecting the optimal video frame of each frame, first, the angle information between each key point is calculated based on the human body key point coordinate map; then, the recommended key points in the human body key point coordinate map are determined based on the angle information between the key points; finally, the corresponding multiple optimal video frames are selected based on the recommended key points in the human body key point coordinate map and the action evaluation criteria.

[0033] Preferably, the angle information between the key points is a counterclockwise rotation angle;

[0034] Calculate the counterclockwise rotation angle by the following steps:

[0035] S511: Obtain key point coordinates A, B, C for calculating the counterclockwise rotation angle;

[0036] S512: Calculate the corresponding key point vector based on the key point coordinates A, B, and C and the key point vector

[0037] S513: The key point vector Rotate counterclockwise until it is aligned with the key point vector coincide; then the key point vector Rotate to keypoint vector The angle is taken as the corresponding counterclockwise rotation angle.

[0038] Preferably, in step S503, when selecting the optimal template video frame, the corresponding key point feature value is calculated based on the human body key point coordinate diagram of the optimal video frame, and then the video frame corresponding to the preset feature value and the key point feature value is matched from the multi-view template video as the optimal template video frame of the optimal video frame;

[0039] The key point eigenvalue is calculated based on the counterclockwise rotation angle between the key points. The calculation formula is as follows:

[0040]

[0041] in,

[0042] In the above formula: I represents the key point feature value of the optimal video frame; represents the variance of the nth counterclockwise rotation angle in the optimal video frame; R represents the counterclockwise rotation angle; u r represents the mean of all counterclockwise rotation angles in the optimal video frame; N represents the number of counterclockwise rotation angles in the optimal video frame; λ represents the set adjustment parameter, and its initial value is 0.1.

[0043] Preferably, in step S504, the motion state information of the video to be tested includes limb motion speed, limb motion stability and limb motion trend;

[0044] The limb movement speed is calculated by the following formula:

[0045]

[0046] The stability of limb movements is determined by the following formula:

[0047]

[0048] The limb movement trend is determined by the following formula:

[0049] θ = R2 - R1;

[0050] In the above formula: V(N1, N2) represents the movement speed of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1; R1 and R2 represent the counterclockwise rotation angle between the key points of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1 respectively; Fps represents the frame rate of the video to be tested; S 2 Indicates the stability characteristic value of the corresponding finger; when judging the stability of limb movements, S 2 Compare with the set stability threshold Y: If S 2 ≥Y, it means the limb movement is stable, otherwise, the limb movement is unstable; R={R1,R2,…R n} represents the counterclockwise rotation angle between the key points of the corresponding limbs in multiple optimal video frames, where n represents the number of optimal video frames; R represents the mean of the counterclockwise rotation angles in R; θ represents the angle difference between the counterclockwise rotation angles R1 and R2 of the corresponding limbs in the optimal video frames N2 and N1. When judging the limb movement trend: if θ>0, it means that the limb rotates counterclockwise; if θ<0, it means that the limb rotates clockwise.

[0051] Compared with the prior art, the human body motion assessment method of the present invention has the following beneficial effects:

[0052] 1. In the present invention, by obtaining key video frames from the videos to be tested at various viewing angles and splicing them to obtain multi-view spliced ​​videos, the videos to be tested at multiple viewing angles can be simultaneously applied to human motion evaluation, thereby solving the problem of motion being blocked due to limited shooting viewing angles, thereby improving the effect of human motion evaluation.

[0053] 2. The present invention constructs a corresponding multi-perspective template video based on the multi-perspective stitching video, so that the multi-perspective template video can form a multi-perspective correspondence with the multi-perspective stitching video, thereby improving the accuracy of human motion assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0055] Figure 1 The logical block diagram of the human motion evaluation method;

[0056] Figure 2 Schematic diagram of ten counterclockwise rotation angles on the human body;

[0057] Figure 3 It is a schematic diagram of the limb angle between the right upper arm and the right forearm;

[0058] Figure 4 Schematic diagram of the counterclockwise rotation angle between the right upper arm and the right forearm. DETAILED DESCRIPTION

[0059] The following is a further detailed description through specific implementation methods:

[0060] Example:

[0061] First, the meaning of action evaluation criteria and action evaluation decision basis are explained.

[0062] Action assessment criteria: refers to the items that need to be assessed during action assessment.

[0063] Taking squat as an example, the movement evaluation criteria include: 1) whether the test rod is directly above the head; 2) whether the torso and calf are parallel or perpendicular to the ground; 3) whether the thigh is below the horizontal line when squatting; 4) whether the knees remain in the same direction as the feet.

[0064] Action evaluation decision basis: refers to the scoring criteria for action evaluation.

[0065] Taking squat as an example, the decision-making basis for action evaluation includes: 1) the test rod is directly above the head, the torso is parallel to the calf or perpendicular to the ground, the thighs are below the horizontal line when squatting, and the knees are kept in the same direction as the feet, and 3 points are scored; 2) Failure to complete the required action or placing a wooden board under the heels to complete the required action will score 2 points; 3) Failure to complete the required action even with a template under the heels will score 1 point; 4) Pain in any part of the body during the test will score 0 points.

[0066] Based on the above description, this embodiment discloses a method for assessing human motion in multi-view video images.

[0067] like Figure 1 As shown, the method for assessing human motion in multi-view video images comprises the following steps:

[0068] S1: Obtain the video to be tested from multiple perspectives of the tester;

[0069] S2: Select key video frames from the video to be tested at each viewing angle;

[0070] S3: stitching the key video frames under each viewing angle to obtain the corresponding multi-view stitching video;

[0071] S4: Select corresponding template video frames from the multi-view template video based on the multi-view stitching video to construct a multi-view template video;

[0072] S5: Complete human motion evaluation based on the multi-view stitched video and the corresponding multi-view template video to generate corresponding motion evaluation results.

[0073] In the present invention, by obtaining key video frames from the videos to be tested at various viewing angles and splicing them to obtain a multi-view spliced ​​video, the videos to be tested at multiple viewing angles can be simultaneously applied to human motion assessment, thereby solving the problem of motion being blocked due to the limitation of the shooting viewing angle, thereby improving the effect of human motion assessment. At the same time, the present invention constructs a corresponding multi-view template video based on the multi-view spliced ​​video, so that the multi-view template video can form a multi-view correspondence with the multi-view spliced ​​video, thereby improving the accuracy of human motion assessment.

[0074] In the specific implementation process, the key video frames are selected through the following steps:

[0075] S201: Perform skeleton analysis and posture analysis on the video frames of the video to be tested at each viewing angle to generate a human body key point coordinate diagram of the corresponding video frame; in this embodiment, the AlphaPose model proposed by Shanghai Jiao Tong University is used to perform skeleton analysis and posture analysis on the video frames of the video to be tested.

[0076] S202: Calculate the confidence of each key point in the human body key point coordinate map;

[0077] S203: Determine the target key point according to the corresponding action evaluation standard;

[0078] S204: Starting from one perspective, video frames whose confidence of target key points is greater than or equal to a set threshold are selected in chronological order as key video frames; if the confidence of target key points in the current perspective is less than the set threshold, video frames whose confidence of target key points in other perspectives is greater than or equal to the set threshold are selected as key video frames;

[0079] S205: Repeat step S204 until the key video frames under each viewing angle are selected. After the key video frames under each viewing angle are selected, the key video frames under each viewing angle are spliced ​​in time sequence to obtain a multi-view spliced ​​video.

[0080] Specifically, in step S201, firstly, the corresponding video frame is input into a pre-trained posture estimation model, and a corresponding heat map is output; then, the key point coordinates are calculated by the heat map to obtain a corresponding human body key point coordinate map;

[0081] When training the posture estimation model, a posture data set for training is obtained; then the labels pre-marked on the training graph in the posture data set are converted into corresponding heat map labels to obtain corresponding label heat maps; finally, the posture estimation model is trained based on the label heat maps;

[0082] When generating a label heat map, first set the size of the label heat map (W h ×H h ) to generate a size W h ×H h The all-zero matrix is ​​then used to calculate the heat distribution of the pre-labeled labels on the label heat map to generate the corresponding label heat map;

[0083]

[0084] When calculating the coordinates of the key points, get the size W h ×H h The heat map is then reduced to 1*W h *H h ; Then calculate the maximum heat value index of the corresponding key point in the heat map by the following formula; Finally, calculate the coordinates of the corresponding key point by combining the index corresponding to the maximum heat value in the heat map with the size of the heat map. Specifically, divide the index by the corresponding W h , the quotient is of size W h ×H h The number of rows of key points is x, and the remainder is of size W h ×H hThe number of columns of the key points y, that is, the coordinates of the key points (x, y);

[0085]

[0086] In the above formula: g represents the heat value; x0 and y0 represent the real coordinates of the pre-annotated label; x and y represent the coordinates of the label in the label heat map; σ represents the standard deviation, which is 2 or 3; e represents a natural constant; i and j represent the index of the one-dimensional heat map respectively; x i 、x j represents the heat value corresponding to index i and j; β represents the calibration coefficient.

[0087] When calculating the confidence of key points, first reduce the dimension of the key point heat map, and then calculate the corresponding confidence by the following formula;

[0088]

[0089] In the above formula: e represents the natural constant; β represents the calibration coefficient; i and j represent the index of the one-dimensional heat map respectively; x i 、x j represents the heat value corresponding to index i and j; β represents the calibration coefficient.

[0090] In the present invention, by calculating the confidence and selecting the key video frames based on the confidence, the confidence of each key video frame in the multi-view stitched video finally obtained can be greater than or equal to the set threshold, that is, the multi-view stitched video can maintain a high reliability, thereby being able to well solve the problem of action being blocked due to the limitation of the shooting angle. At the same time, the reliability of the video frame is well reflected by calculating the confidence in the present invention, thereby being able to improve the effect of human action evaluation.

[0091] In the specific implementation process, human motion evaluation is completed through the following steps:

[0092] S501: performing skeleton analysis and posture analysis on key video frames of the multi-view stitched video to generate a human body key point coordinate map corresponding to the key video frames;

[0093] S502: Select multiple key video frames as optimal video frames based on the human body key point coordinate diagram and corresponding action evaluation criteria;

[0094] S503: selecting corresponding multi-frame template video frames from the multi-view template video as optimal template video frames based on the optimal video frames of each frame;

[0095] S504: Calculating motion state information of the multi-view stitching video and motion standard information of the multi-view template video based on the optimal video frames of each frame and the optimal template video frames of each frame;

[0096] S505: Compare the motion state information of the multi-view stitching video with the motion standard information of the multi-view template video to generate a corresponding motion evaluation result.

[0097] In the present invention, by selecting multiple optimal video frames and calculating the motion state information of the video to be tested, the motion continuity and stability of the tester can be effectively reflected, and then the motion evaluation result is generated by comparing the motion state information of the video to be tested with the motion standard information of the template video, which can ensure the accuracy of human motion evaluation. At the same time, the present invention generates a human key point coordinate diagram by means of skeleton analysis and posture analysis, and then selects multiple optimal video frames in combination with the motion evaluation standard, so that the selected multiple optimal video frames can be associated with the corresponding motion evaluation standard, which can ensure the accuracy of the selection of the optimal video frames, thereby further ensuring the accuracy and effect of the motion evaluation. In addition, the present invention also selects multiple optimal template video frames based on multiple optimal video frames and calculates the corresponding motion standard information, and the motion standard information corresponds to the motion state information, so that the accuracy of the motion state information can be more accurately evaluated through the motion standard information, thereby ensuring the accuracy of the human motion evaluation.

[0098] During the specific implementation process, when selecting the optimal video frame of each frame, the angle information between each key point is first calculated based on the human body key point coordinate map; then the recommended key points in the human body key point coordinate map are determined based on the angle information between the key points; finally, the corresponding multi-frame optimal video frames are selected based on the recommended key points in the human body key point coordinate map and the action evaluation criteria.

[0099] The angle information between key points is the counterclockwise rotation angle; combined Figure 2 As shown, the types of counterclockwise rotation angles include the angle between the left forearm and the left upper arm, the angle between the left upper arm and the left shoulder, the angle between the left upper arm and the torso, the angle between the torso and the left thigh, the angle between the left thigh and the left calf, the angle between the right upper arm and the right forearm, the angle between the right shoulder and the right upper arm, the angle between the torso and the right upper arm, the angle between the torso and the right thigh, and the angle between the right thigh and the right calf.

[0100] Calculate the counterclockwise rotation angle by the following steps:

[0101] S511: Obtain key point coordinates A, B, C for calculating the counterclockwise rotation angle;

[0102] S512: Calculate the corresponding key point vector based on the key point coordinates A, B, and C and the key point vector

[0103] S513: The key point vector Rotate counterclockwise until it is aligned with the key point vector coincide; then the key point vector Rotate to keypoint vector The angle is taken as the corresponding counterclockwise rotation angle.

[0104] In the present invention, by calculating the angle information and selecting recommended key points based on the angle information and finally selecting multiple optimal video frames based on the recommended key points and action evaluation criteria, the accuracy of selecting the optimal video frames can be guaranteed, and the selected multiple optimal video frames can be associated with the corresponding action evaluation criteria, thereby ensuring the accuracy of selecting the optimal video frames.

[0105] The skeleton analysis and posture analysis of the video frame results in a two-dimensional posture. The key points in the human body key point coordinate diagram are actually the projection of the real posture on the two-dimensional plane, which makes it difficult for a simple limb angle to accurately represent the limb characteristics of an action. Figure 3 As shown in the figure, when the right arm is bent in front of the chest and bent at the side of the body, the limb angle between the right upper arm and the right forearm is the same. From the data, the limb characteristics of the right arm movement are the same, because the limb angle between the right upper arm and the right forearm is the same, but in fact, there are great differences between the two movements.

[0106] To this end, the present invention adds direction information, i.e., rotation direction, based on the limb angle, so that the generated counterclockwise rotation angle has both angle information and direction information (e.g. Figure 4 As shown in FIG. 1 , the problem of partial posture information loss when the real posture is projected onto a two-dimensional plane can be solved, and the characteristics of the action limbs can be accurately represented, thereby ensuring the accuracy of human motion evaluation. At the same time, the ten counterclockwise rotation angles designed by the present invention can basically cover the important action limb characteristics of the human posture, thereby further ensuring the effect of human motion evaluation.

[0107] In the specific implementation process, when selecting the optimal template video frame, the corresponding key point feature value is calculated based on the human body key point coordinate map of the optimal video frame, and then the video frame corresponding to the preset feature value and the key point feature value is matched from the template video as the optimal template video frame of the key video frame;

[0108] The key point eigenvalue is calculated based on the counterclockwise rotation angle between the key points. The calculation formula is as follows:

[0109]

[0110] in,

[0111] In the above formula: I represents the key point feature value of the optimal video frame; represents the variance of the nth counterclockwise rotation angle in the optimal video frame; R represents the counterclockwise rotation angle; ur represents the mean of all counterclockwise rotation angles in the optimal video frame; N represents the number of counterclockwise rotation angles in the optimal video frame; λ represents the set adjustment parameter, and its initial value is 0.1.

[0112] In the present invention, by calculating the key point feature values ​​and matching the preset feature values, the optimal template video frame corresponding to the optimal video frame can be accurately matched, thereby further ensuring the accuracy of human motion evaluation under the premise of reducing the motion evaluation cost and improving the motion evaluation efficiency. At the same time, the present invention can continuously optimize the selection strategy of the optimal template video frame by adjusting the adjustment parameters, thereby better improving the selection accuracy of the optimal template video frame.

[0113] During the specific implementation process, the motion state information of the video to be tested includes limb movement speed, limb movement stability and limb movement trend; the motion standard information includes the standard range of motion speed and the standard motion trend; when the limb movement speed is within the standard range of motion speed, the limb movement trend corresponds to the standard motion trend and the movement is stable, the motion evaluation result is passed; otherwise, the motion evaluation fails.

[0114] The limb movement speed is calculated by the following formula:

[0115]

[0116] The stability of limb movements is determined by the following formula:

[0117]

[0118] The limb movement trend is determined by the following formula:

[0119] θ = R2 - R1;

[0120] In the above formula: V(N1, N2) represents the movement speed of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1; R1 and R2 represent the counterclockwise rotation angle between the key points of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1 respectively; Fps represents the frame rate of the video to be tested; S 2 Indicates the stability characteristic value of the corresponding finger; when judging the stability of limb movements, S 2 Compare with the set stability threshold Y: If S 2 ≥Y, it means the limb movement is stable, otherwise, the limb movement is unstable; R={R1,R2,…R n} represents the counterclockwise rotation angle between the key points of the corresponding limbs in multiple optimal video frames, where n represents the number of optimal video frames; Represents the mean of the counterclockwise rotation angles in R; θ represents the angle difference between the counterclockwise rotation angles R1 and R2 of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1. When judging the limb movement trend: if θ>0, it means that the limb rotates counterclockwise, if θ<0, it means that the limb rotates clockwise.

[0121] In the present invention, the motion state information of the tester can be well reflected through the limb movement speed, limb movement stability and limb movement trend, and thus the tester's movement continuity and stability can be effectively reflected. Subsequently, the motion state information of the video to be tested and the motion standard information of the template video are compared to generate the motion evaluation result, thereby ensuring the accuracy of the human body motion evaluation. At the same time, the calculation formula and method of the present invention can well calculate the limb movement speed, limb movement stability and limb movement trend, thereby better reflecting the tester's motion state information.

[0122] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes can be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims. At the same time, common sense such as the well-known specific structures and characteristics in the embodiments are not described in detail here. Finally, the scope of protection claimed by the present invention shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for assessing human motion in multi-view video images, characterized in that: The following steps are involved: S1: Obtain the video to be tested from multiple perspectives of the tester; S2: Select key video frames from the video to be tested at each viewing angle; In step S2, the key video frame is selected by the following steps: S201: performing skeleton analysis and posture analysis on video frames of the video to be tested at each viewing angle to generate a corresponding human body key point coordinate map; S202: Calculate the confidence of each key point in the human body key point coordinate map; S203: Determine the target key point according to the corresponding action evaluation standard; S204: Starting from one perspective, video frames whose confidence of target key points is greater than or equal to a set threshold are selected in chronological order as key video frames; if the confidence of target key points in the current perspective is less than the set threshold, video frames whose confidence of target key points in other perspectives is greater than or equal to the set threshold are selected as key video frames; S205: repeat step S204 until the key video frames under each viewing angle are selected; In step S201, firstly, the corresponding video frame is input into a pre-trained posture estimation model, and the corresponding heat map is output; then, the key point coordinates are calculated by the heat map to obtain the corresponding human body key point coordinate map; When training the posture estimation model, a posture data set for training is obtained; then the labels pre-annotated on the training graph in the posture data set are converted into corresponding heat map labels to obtain corresponding label heat maps; finally, the posture estimation model is trained based on the label heat maps; When calculating the coordinates of key points, first obtain the heat map and reduce the dimension of the heat map; then calculate the maximum heat value index of the corresponding key point in the heat map; finally, calculate the coordinates of the corresponding key point by combining the index corresponding to the maximum heat value in the heat map with the size of the heat map; In step S202, when calculating the confidence of the key point, the key point heat map is first reduced in dimension, and then the corresponding confidence is calculated by the following formula; In the above formula: e represents the natural constant; β represents the calibration coefficient; i and j represent the index of the one-dimensional heat map respectively; x i 、x j Indicates the heat value corresponding to index i and j; S3: stitching the key video frames under each viewing angle to obtain the corresponding multi-view stitching video; S4: Select corresponding template video frames from the multi-view template video based on the multi-view stitching video to construct a multi-view template video; S5: Complete human motion evaluation based on the multi-view stitched video and the corresponding multi-view template video to generate corresponding motion evaluation results.

2. The method for assessing human motion in multi-view video images according to claim 1, wherein: In step S3, after the key video frames under each viewing angle are selected, the key video frames under each viewing angle are spliced ​​in chronological order to obtain a multi-view spliced ​​video.

3. The method for assessing human motion in multi-view video images according to claim 1, wherein: In step S5, human motion assessment is completed through the following steps: S501: performing skeleton analysis and posture analysis on key video frames of the multi-view stitched video to generate a human body key point coordinate map corresponding to the key video frames; S502: Select multiple key video frames as optimal video frames based on the human body key point coordinate diagram and corresponding action evaluation criteria; S503: selecting corresponding multi-frame template video frames from the multi-view template video as the optimal template video frames based on the optimal video frames of each frame; S504: Calculating motion state information of the multi-view stitching video and motion standard information of the multi-view template video based on the optimal video frames of each frame and the optimal template video frames of each frame; S505: Compare the motion state information of the multi-view stitching video with the motion standard information of the multi-view template video to generate a corresponding motion evaluation result.

4. The method for assessing human motion in multi-view video images according to claim 3, wherein: In step S502, when selecting the optimal video frame of each frame, first, the angle information between each key point is calculated based on the human body key point coordinate map; then, the recommended key points in the human body key point coordinate map are determined based on the angle information between the key points; finally, the corresponding multi-frame optimal video frames are selected based on the recommended key points in the human body key point coordinate map and the action evaluation criteria.

5. The method for assessing human motion in multi-view video images according to claim 4, wherein: The angle information between key points is the counterclockwise rotation angle; Calculate the counterclockwise rotation angle by the following steps: S511: Obtain key point coordinates A, B, C for calculating the counterclockwise rotation angle; S512: Calculate the corresponding key point vector based on the key point coordinates A, B, and C and the key point vector S513: The key point vector Rotate counterclockwise until it is aligned with the key point vector coincide; then the key point vector Rotate to keypoint vector The angle is taken as the corresponding counterclockwise rotation angle.

6. The method for assessing human motion in multi-view video images according to claim 5, wherein: In step S503, when selecting the optimal template video frame, the corresponding key point feature value is calculated based on the human body key point coordinate map of the optimal video frame, and then the video frame corresponding to the preset feature value and the key point feature value is matched from the multi-view template video as the optimal template video frame of the optimal video frame; The key point eigenvalue is calculated based on the counterclockwise rotation angle between the key points. The calculation formula is as follows: in, In the above formula: I represents the key point feature value of the optimal video frame; represents the variance of the nth counterclockwise rotation angle in the optimal video frame; R represents the counterclockwise rotation angle; u r represents the mean of all counterclockwise rotation angles in the optimal video frame; N represents the number of counterclockwise rotation angles in the optimal video frame; λ represents the set adjustment parameter, and its initial value is 0.

1.

7. The method for assessing human motion in multi-view video images according to claim 3, wherein: In step S504, the motion state information of the video to be tested includes limb motion speed, limb motion stability and limb motion trend; The limb movement speed is calculated by the following formula: The stability of limb movements is determined by the following formula: The limb movement trend is determined by the following formula: θ = R2 - R1; In the above formula: V(N1, N2) represents the movement speed of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1; R1 and R2 represent the counterclockwise rotation angle between the key points of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1 respectively; Fps represents the frame rate of the video to be tested; S 2 Indicates the stability characteristic value of the corresponding finger; when judging the stability of limb movements, S 2 Compare with the set stability threshold Y: If S 2 ≥Y, it means the limb movement is stable, otherwise, the limb movement is unstable; R={R1,R2,…R n } represents the counterclockwise rotation angle between the key points of the corresponding limbs in multiple optimal video frames, where n represents the number of optimal video frames; Represents the mean of the counterclockwise rotation angles in R; θ represents the angle difference between the counterclockwise rotation angles R1 and R2 of the corresponding limbs in the optimal video frame N2 and the optimal video frame N1. When judging the limb movement trend: if θ>0, it means that the limb rotates counterclockwise, if θ<0, it means that the limb rotates clockwise.

Images