User-defined action matching scoring method based on 3D posture recognition

Abstract

The present invention discloses a user-defined action matching and scoring method based on three-dimensional posture recognition, comprising the following steps: obtaining a standard action image; extracting first rotation Euler angle parameters of the standard action limb from the standard action image; obtaining an action image to be evaluated; extracting second rotation Euler angle parameters of the action limb to be evaluated from the action image; calculating the cosine similarity of the first rotation Euler angle parameters and the second rotation Euler angle parameters; and obtaining an evaluation result based on the calculation result. In the user-defined action matching and scoring method based on three-dimensional posture recognition, the three-dimensional posture estimation allows action matching to be unrestricted by the action image's viewing angle, reducing manual repetitive work in formulating action standards and saving labor costs.

Description

Technical Field

[0001] The present invention belongs to the technical field of posture evaluation, and in particular relates to a user-defined action matching scoring method based on three-dimensional posture recognition. Background Art

[0002] Motion matching is often used in various fields. For example, in the field of sports and fitness, when following exercise videos and performing exercises, motion matching is needed to inform users whether the movements are standard and avoid injuries caused by improper movements. In the field of science and education, schools need to evaluate students' morning exercise performance and need to match the movements with standard movements to score the movements. In the field of driving, the vehicle needs to respond to the driver's condition through motion matching. For example, if any movements outside the standard driving posture are used, prompts and warnings must be given.

[0003] Existing three-dimensional motion matching solutions that are not restricted by viewing angles require the use of inertial sensors for motion capture, which capture human body movements based on the sensors and return them for data processing. However, the sensors need to be worn by the person being evaluated, which is not applicable in many scenarios. Without the use of inertial sensors for motion capture, in the current two-dimensional motion matching solutions based only on images, the action to be evaluated must be photographed from the same perspective as the standard action, which has very large limitations. In addition, if matching scores are to be performed for different actions, it is necessary to repeatedly define the action matching rules for multiple actions. For example, in sports scenes, both Tai Chi and Ba Duan Jin require developers to redefine the rules for body movements, manually select and limit the parameters of the two sets of actions for the system, which is very inconvenient for migration applications. Summary of the Invention

[0004] The present invention provides a user-defined action matching scoring method based on three-dimensional posture recognition to solve the above-mentioned technical problems, specifically adopting the following technical solutions:

[0005] A user-defined action matching scoring method based on three-dimensional posture recognition, comprising:

[0006] Acquire standard action images;

[0007] Extracting first rotation Euler angle parameters of the standard action limb from the standard action image;

[0008] Obtaining an action image to be evaluated;

[0009] Extracting a second rotation Euler angle parameter of the limb to be evaluated from the action image to be evaluated;

[0010] Calculating the cosine similarity between the first rotation Euler angle parameter and the second rotation Euler angle parameter;

[0011] The evaluation results are obtained based on the calculation results.

[0012] Furthermore, the specific method of extracting the first rotation Euler angle parameter of the standard action limb from the standard action image is:

[0013] Input the standard action image into the yolov5 model to perform human target detection on the image;

[0014] Inputting the detected target frame coordinates and the standard action image into the OSX three-dimensional posture detection model to obtain a human posture SMPL model;

[0015] Select the axis angle parameters representing limb rotation in the SMPL model of human posture;

[0016] The rotation parameter is converted from axis-angle representation to Euler angle representation to obtain the first rotation Euler angle parameter.

[0017] Furthermore, the specific method of extracting the second rotation Euler angle parameters of the standard action limb from the action image to be evaluated is:

[0018] Input the action image to be evaluated into the yolov5 model to perform human target detection on the image;

[0019] Inputting the detected target frame coordinates and the action image to be evaluated into the OSX three-dimensional posture detection model to obtain a human posture SMPL model;

[0020] Select the axis angle parameters representing limb rotation in the SMPL model of human posture;

[0021] The rotation parameter is converted from the axis-angle representation to the Euler angle representation to obtain the second rotation Euler angle parameter.

[0022] Furthermore, the specific method for selecting the axis angle parameters representing limb rotation in the human posture SMPL model is:

[0023] Select the Rot6D rotation parameter representing the limb rotation from the posture feature parameter θ in the human posture SMPL model;

[0024] Convert the selected Rot6D rotation parameters into rotation angles to obtain axis-angle parameters.

[0025] Furthermore, the specific method of converting the rotation parameter from the axis-angle representation to the Euler angle representation to obtain the first rotation Euler angle parameter or the second rotation Euler angle parameter is:

[0026] The axis angle parameters are converted into a rotation matrix using the following formula:

[0027]

[0028] Among them, the axis-angle parameter is (k, ω), where k represents the rotation axis, x, y, z represent the three dimensions of the rotation axis, ω represents the rotation angle, and R(k, ω) is the rotation matrix;

[0029] Then convert the rotation matrix into Euler angles. Euler angles have three dimensions: Yaw, pitch, and roll, which represent the yaw angle, pitch angle, and roll angle respectively. Yaw is α, pitch is β, and roll is γ. The formula for obtaining Euler angles from the rotation matrix is:

[0030] α=atan2(R 13 , R 33 )

[0031] β=arcsin(-R 23 )

[0032] y=atan2(R 21 , R 22 ).

[0033] Further, a plurality of standard action images are obtained;

[0034] Extracting a plurality of first rotation Euler angle parameters of a standard action limb from a plurality of the standard action images;

[0035] Acquire multiple action images to be evaluated;

[0036] Extracting a plurality of second rotation Euler angle parameters of the limb to be evaluated from a plurality of the action images to be evaluated;

[0037] Calculating a plurality of cosine similarities between the first rotation Euler angle parameters and the corresponding second rotation Euler angle parameters;

[0038] Calculate the average of multiple cosine similarities;

[0039] The evaluation results are obtained based on the calculation results.

[0040] Furthermore, the specific method of obtaining multiple standard action images is:

[0041] Obtaining a standard action video with background music, and decomposing the standard action video into a plurality of standard action images;

[0042] The specific method of obtaining multiple action images to be evaluated is:

[0043] An action video to be evaluated with the same background music is obtained, and the action video to be evaluated is decomposed into a plurality of action images to be evaluated.

[0044] Furthermore, the specific method of obtaining multiple standard action images is:

[0045] Obtaining a standard action video with background music, and extracting a plurality of standard action images from the standard action video at preset time intervals;

[0046] The specific method of obtaining multiple action images to be evaluated is:

[0047] A to-be-evaluated action video with the same background music is obtained, and a plurality of to-be-evaluated action images are extracted from the to-be-evaluated action video at the same preset time interval.

[0048] Furthermore, for each action image to be evaluated, after extracting a plurality of action images to be evaluated from the action video to be evaluated at the same preset time interval, auxiliary evaluation action images of a preset number of frames before and after the action image to be evaluated are obtained;

[0049] Extracting a plurality of third rotation Euler angle parameters of the limb to be evaluated from the plurality of auxiliary evaluation action images;

[0050] For each of the action images to be evaluated, after calculating the cosine similarity between the corresponding first rotation Euler angle parameter and the corresponding second rotation Euler angle parameter, the cosine similarity between the corresponding first rotation Euler angle parameter and the corresponding multiple third rotation Euler angle parameters is calculated. Based on the calculated cosine similarities between the first rotation Euler angle parameter and the corresponding multiple third rotation Euler angle parameters, it is determined whether the action corresponding to the action image to be evaluated is ahead or behind.

[0051] Furthermore, the preset time interval is determined according to the beat of the background music.

[0052] The present invention is beneficial in that it provides a user-defined action matching scoring method based on 3D posture recognition, innovatively utilizing a 3D human posture estimation model for a user-defined action matching scoring scheme. The image-based posture estimation model eliminates the need for motion estimation using inertial sensors used for motion capture, broadening its application scenarios. 3D posture estimation allows action matching to be performed independently of the perspective of the action image, enabling successful judgment regardless of the angle of the image. By simply inputting an image of a standard action, the parameters of the standard action can be automatically extracted and stored. This ability to customize standard action schemes reduces the workload for developers, eliminates the manual repetitive work involved in developing action standards, and saves labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A schematic diagram of a user-defined action matching and scoring method based on three-dimensional posture recognition provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a three-dimensional gesture recognition process provided by an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of another implementation of a user-defined action matching scoring method based on three-dimensional posture recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0058] like Figure 1 The present application shows a user-defined action matching scoring method based on three-dimensional posture recognition, comprising: S1: obtaining a standard action image. S2: extracting the first rotation Euler angle parameters of the standard action limb from the standard action image. S3: obtaining the action image to be evaluated. S4: extracting the second rotation Euler angle parameters of the action limb to be evaluated from the action image to be evaluated. S5: calculating the cosine similarity of the first rotation Euler angle parameter and the second rotation Euler angle parameter. S6: obtaining an evaluation result based on the calculation result. Through the above steps, the user-defined action matching scoring method based on three-dimensional posture recognition of the present application is not limited by the viewing angle limitation and can directly determine the action matching based on the image. The above steps are described in detail below.

[0059] Regarding step S1: obtain a standard action image.

[0060] First, a customized standard action image is determined and used as a comparison object.

[0061] For step S2: extract the first rotation Euler angle parameters of the standard action limbs from the standard action image.

[0062] The specific method for extracting the first rotation Euler angle parameters of the standard action limbs from the standard action image is:

[0063] The standard action image is fed into the Yolov5 model to detect human subjects, returning the coordinates of a bounding box containing the subject. The detected bounding box coordinates and the standard action image are fed into the OSX 3D pose detection model, which outputs a human pose SMPL model. The axis-angle parameters representing limb rotations in the SMPL model are selected. The rotation parameters are converted from axis-angle representation to Euler angles to obtain the Euler angle parameters for the first rotation.

[0064] Specifically, in 2D posture comparisons, the identity of an action is often determined by the same skeletal orientation. However, identical skeletal orientations can also be ambiguous. Therefore, this patent uses the three-dimensional Euler angles of each joint relative to its parent joint to represent the action. The joints are shown in Table 1. Since relative rotations in three-dimensional space can be uniquely determined, this ambiguity is eliminated.

[0065] Table 1 Correspondence table of human joint parameters

[0066] serial number Joint Name variable name serial number Joint Name Joint Name 1 left hip L_Hip 12 neck Neck 2 right hip R_Hip 13 Left clavicle L_Collar 3 Spinal Point 1 Spine_1 14 Right clavicle R_Collar 4 left knee L_Knee 15 head Head 5 right knee R_Knee 16 left shoulder L_Shoulder 6 Spinal Point 2 Spine_2 17 right shoulder R_Shoulder 7 left ankle L_Ankle 18 left elbow L_Elbow 8 right ankle R_Ankle 19 right elbow R_Elbow 9 Spinal Point 3 Spine_3 20 left wrist L_Wrist 10 left foot L_Foot 21 right wrist R_Wrist 11 right foot R_Foot

[0067] like Figure 2 As shown in the figure, during the 3D pose recognition stage, to ensure effective human pose estimation, the YOLOv5 object detection model is first used to delineate the human body region. This region is then fed into the OSX human pose estimation model. The model first feeds the RGB image containing the human body into the embedding layer, generating a high-dimensional feature vector (body_tokens) representing the body pose. The model then generates human pose feature parameters through a Transformer encoder and a human pose regression function. The parameters of these general image embedding layers, Transformer encoders, and human pose regression functions are all trained using extensive data. This results in the SMPL model of human pose.

[0068] The posture feature parameter θ in the human posture SMPL model is the focus of action matching. In the embodiment of the present application, the specific method of selecting the axis angle parameter representing the limb rotation in the human posture SMPL model is:

[0069] The Rot6D rotation parameters representing limb rotations are selected from the posture feature parameters θ in the SMPL human posture model. The selected Rot6D rotation parameters are converted to rotation angles to obtain axis-angle parameters. In motion matching, Rot6D rotation parameters are not suitable for human body model visualization or comparison. Therefore, for the output posture feature parameters θ, the Rot6D is first converted to axis-angle representation. The axis-angle representation facilitates the use of SMPL human body model parameters to draw 3D human images. The axis-angle parameters use 21×3 dimensions to describe human body movements and postures; the 21 refers to the 21 joints. The specific joints and their numbering relationships are shown in Table 1. The 3 refers to the axis-angle value representing the three-dimensional rotation.

[0070] During the 3D pose recognition phase, we ultimately obtain the rotation angle parameters representing the body's limb rotations in the image. However, the axis-angle representation is still relatively abstract and not very interpretable. In this application, we convert the rotation angle parameters into Euler angles for similarity estimation.

[0071] As a preferred embodiment, the specific method of converting the rotation parameter from the axis-angle representation to the Euler angle representation to obtain the first rotation Euler angle parameter is:

[0072] The axis angle parameters are converted into a rotation matrix using the following formula:

[0073]

[0074] The axis-angle parameters are (k, ω). k represents the rotation axis, x, y, z represent the three dimensions of the rotation axis, ω represents the rotation angle, and R(k, ω) is the rotation matrix.

[0075] Then convert the rotation matrix into Euler angles. Euler angles have three dimensions: Yaw, pitch, and roll, which represent the yaw angle, pitch angle, and roll angle respectively. Yaw is written as α, pitch is written as β, and roll is written as γ. The formula for obtaining Euler angles from the rotation matrix is:

[0076] α=atan2(R 13 , R 33 )

[0077] β=arcsin(-R 23 )

[0078] γ=atan2(R 21 , R 22 ).

[0079] Regarding step S3: obtaining the action image to be evaluated.

[0080] Regarding step S4: extracting the second rotation Euler angle parameters of the limb to be evaluated from the action image to be evaluated.

[0081] The specific method for extracting the second rotation Euler angle parameters of the standard action limb from the action image to be evaluated is:

[0082] Input the action image to be evaluated into the yolov5 model to perform human target detection on the image.

[0083] The detected target box coordinates and the action image to be evaluated are input into the OSX 3D posture detection model to obtain the human posture SMPL model.

[0084] Select the axis angle parameters representing limb rotation in the SMPL model of human posture.

[0085] The rotation parameter is converted from the axis-angle representation to the Euler angle representation to obtain the second rotation Euler angle parameter.

[0086] The specific process is referred to step S2 and will not be described again here.

[0087] For step S5: calculate the cosine similarity between the first rotation Euler angle parameter and the second rotation Euler angle parameter.

[0088] Regarding step S6: obtaining an evaluation result according to the calculation result.

[0089] Specifically, a preset threshold range may be set, and the evaluation result may be obtained by comparing the calculated cosine similarity with the threshold range.

[0090] The method described above can evaluate a single action. It is understandable that the method of the present application can also perform a comprehensive evaluation of multiple actions. Specifically, Figure 3 As shown in FIG, another user-defined action matching scoring method based on three-dimensional posture recognition of the present application includes:

[0091] S10: Acquire multiple standard action images.

[0092] S20: Extracting a plurality of first rotation Euler angle parameters of the standard action limbs from a plurality of standard action images.

[0093] S30: Acquire multiple action images to be evaluated.

[0094] S40: extracting a plurality of second rotation Euler angle parameters of the limb to be evaluated from a plurality of images of the action to be evaluated.

[0095] S50: Calculating a plurality of first rotation Euler angle parameters and a plurality of cosine similarities corresponding to the second rotation Euler angle parameters.

[0096] S60: Calculate the average values ​​of multiple cosine similarities.

[0097] S70: Obtain an evaluation result according to the calculation result.

[0098] Specifically, the specific method for obtaining multiple standard action images is:

[0099] A standard motion video with background music is obtained, and the standard motion video is decomposed into multiple standard motion images.

[0100] The specific method for obtaining multiple action images to be evaluated is:

[0101] Obtain an action video to be evaluated with the same background music, and decompose the action video to be evaluated into multiple action images to be evaluated.

[0102] As you can understand, in practice, it's often necessary to evaluate the standardization of a continuous action. For example, when evaluating a dance, a video of a standard action with background music can be split into multiple standard action images based on the number of frames. Similarly, a video of the action to be evaluated with the same background music can be split into multiple action images based on the number of frames. Then, using the parameter extraction and comparison methods described above, each image is compared frame by frame. The cosine similarities of the multiple images are averaged, and the evaluation result is obtained based on the average value.

[0103] As another preferred embodiment, the specific method for obtaining multiple standard motion images is:

[0104] A standard action video with background music is obtained, and multiple standard action images are extracted from the standard action video according to a preset time interval.

[0105] The specific method for obtaining multiple action images to be evaluated is:

[0106] A video of an action to be evaluated with the same background music is obtained, and multiple images of the action to be evaluated are extracted from the video of the action to be evaluated at the same preset time interval.

[0107] It is understandable that comparing every frame of a video requires a large amount of computation. Therefore, selecting the same number of images from the video using the same preset time interval and then comparing them one by one can reduce the computational effort. In the embodiment of the present application, the preset time interval is determined based on the beat of the background music.

[0108] It is understandable that the aforementioned comparison method captures images according to the rhythm of the music and then compares them one by one. This comparison method may not fully reflect the performer's performance. For example, some performers may execute their movements very standardly, but they may not completely match the beat of the music. Using the aforementioned evaluation method, the evaluation results may be poor. However, for such performers, the greater problem is the matching of the movements with the music. To address this issue, for each action image to be evaluated, multiple action images to be evaluated are extracted from the action video to be evaluated at the same preset time interval. Then, auxiliary evaluation action images are obtained for a preset number of frames before and after the images. Then, multiple third rotation Euler angle parameters of the limb to be evaluated are extracted from the multiple auxiliary evaluation action images. For each action image to be evaluated, the cosine similarity between the corresponding first rotation Euler angle parameter and the corresponding second rotation Euler angle parameter is calculated. Then, the cosine similarity between the corresponding first rotation Euler angle parameter and the corresponding multiple third rotation Euler angle parameters is calculated. Based on the calculation results, it is determined whether the action corresponding to the action image to be evaluated is ahead or behind.

[0109] It is understandable that after extracting multiple action images to be evaluated from the action video to be evaluated at the same preset time interval, for each action image to be evaluated, auxiliary evaluation action images of the preset number of frames before and after it are obtained, such as auxiliary evaluation action images of 10 frames before and after. All 21 frames of images are compared with the corresponding standard images to find the frame with the highest similarity. Since the frame with the highest similarity is the action image to be evaluated extracted from the action video to be evaluated at the preset time interval, it indicates that there is no problem with the rhythm. If the frame with the highest similarity is in the 10 frames before the action image to be evaluated, it indicates that the action was rushed. According to its position in these 10 frames, it can be judged how much the action was rushed. If the frame with the highest similarity is in the 10 frames after the action image to be evaluated, it indicates that the action is delayed. Similarly, according to its position in these 10 frames, it can be judged how much the action is delayed.

[0110] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the scope of protection of the present invention.

Claims

1. A user-defined action matching scoring method based on three-dimensional posture recognition, characterized in that: Include: Acquire standard action images; Extracting first rotation Euler angle parameters of the standard action limb from the standard action image; Obtaining an action image to be evaluated; Extracting a second rotation Euler angle parameter of the limb to be evaluated from the action image to be evaluated; Calculating the cosine similarity between the first rotation Euler angle parameter and the second rotation Euler angle parameter; Obtaining an evaluation result based on the calculation result; The specific method of extracting the first rotation Euler angle parameter of the standard action limb from the standard action image is: Input the standard action image into the yolov5 model to perform human target detection on the image; Inputting the detected target frame coordinates and the standard action image into the OSX three-dimensional posture detection model to obtain a human posture SMPL model; Select the axis angle parameters representing limb rotation in the SMPL model of human posture; Converting the rotation parameter from axis-angle representation to Euler angle representation to obtain the first rotation Euler angle parameter; The specific method of extracting the second rotation Euler angle parameter of the action limb to be evaluated from the action image to be evaluated is: Input the action image to be evaluated into the yolov5 model to perform human target detection on the image; Inputting the detected target frame coordinates and the action image to be evaluated into the OSX three-dimensional posture detection model to obtain a human posture SMPL model; Select the axis angle parameters representing limb rotation in the SMPL model of human posture; Converting the rotation parameter from axis-angle representation to Euler angle representation to obtain the second rotation Euler angle parameter; The specific method for selecting the axis angle parameters representing limb rotation in the human posture SMPL model is: Select the Rot6D rotation parameter representing the limb rotation from the posture feature parameter θ in the human posture SMPL model; Convert the selected Rot6D rotation parameters into rotation angles to obtain axis angle parameters.

2. The user-defined action matching scoring method based on three-dimensional posture recognition according to claim 1, characterized in that: The specific method of converting the rotation parameter from the axis-angle representation to the Euler angle representation to obtain the first rotation Euler angle parameter or the second rotation Euler angle parameter is: The axis angle parameters are converted into a rotation matrix using the following formula: Among them, the axis-angle parameter is (k, ω), where k represents the rotation axis, x, y, z represent the three dimensions of the rotation axis, ω represents the rotation angle, and R(k, ω) is the rotation matrix; Then convert the rotation matrix into Euler angles. Euler angles have three dimensions: Yaw, pitch, and roll, which represent the yaw angle, pitch angle, and roll angle respectively. Yaw is α, pitch is β, and roll is γ. The formula for obtaining Euler angles from the rotation matrix is: α=atan2(R 13 ,R 33 ) β=arcsin(-R 23 ) γ=atan2(R 21 ,R 22 )。 3. The user-defined action matching scoring method based on three-dimensional posture recognition according to claim 1, characterized in that: Acquire multiple standard action images; Extracting a plurality of first rotation Euler angle parameters of a standard action limb from a plurality of the standard action images; Acquire multiple action images to be evaluated; Extracting a plurality of second rotation Euler angle parameters of the limb to be evaluated from a plurality of the action images to be evaluated; Calculating a plurality of cosine similarities between the first rotation Euler angle parameters and the corresponding second rotation Euler angle parameters; Calculate the average of multiple cosine similarities; The evaluation results are obtained based on the calculation results.

4. The user-defined action matching scoring method based on three-dimensional posture recognition according to claim 3, characterized in that: The specific method of obtaining multiple standard action images is: Obtaining a standard action video with background music, and decomposing the standard action video into a plurality of standard action images; The specific method of obtaining multiple action images to be evaluated is: An action video to be evaluated with the same background music is obtained, and the action video to be evaluated is decomposed into a plurality of action images to be evaluated.

5. The user-defined action matching scoring method based on three-dimensional posture recognition according to claim 4, characterized in that: The specific method of obtaining multiple standard action images is: Obtaining a standard action video with background music, and extracting a plurality of standard action images from the standard action video at preset time intervals; The specific method of obtaining multiple action images to be evaluated is: A to-be-evaluated action video with the same background music is obtained, and a plurality of to-be-evaluated action images are extracted from the to-be-evaluated action video at the same preset time interval.

6. The user-defined action matching scoring method based on three-dimensional posture recognition according to claim 5, characterized in that: For each action image to be evaluated, after extracting multiple action images to be evaluated from the action video to be evaluated at the same preset time interval, obtain auxiliary evaluation action images of a preset number of frames before and after the action image to be evaluated; Extracting a plurality of third rotation Euler angle parameters of the limb to be evaluated from the plurality of auxiliary evaluation action images; For each of the action images to be evaluated, after calculating the cosine similarity between the corresponding first rotation Euler angle parameter and the corresponding second rotation Euler angle parameter, the cosine similarity between the corresponding first rotation Euler angle parameter and the corresponding multiple third rotation Euler angle parameters is calculated. Based on the calculated cosine similarities between the first rotation Euler angle parameter and the corresponding multiple third rotation Euler angle parameters, it is determined whether the action corresponding to the action image to be evaluated is ahead or behind.

7. The user-defined action matching scoring method based on three-dimensional posture recognition according to claim 5, characterized in that: The preset time interval is determined according to the beat of the background music.

Images