A key frame fast correction method and system based on action recognition

Abstract

The application relates to a key frame fast correction method and system based on action recognition. The application obtains human joint point coordinates containing confidence based on OpenPose neural network for posture recognition, and performs abnormal screening on the human joint point coordinates through the confidence. A triple fusion correction strategy is adopted to perform correction processing on abnormal human joint point coordinates in real time, so that human joint point parameters are quickly and accurately obtained. The application method is free from a laboratory environment and realizes low-cost human posture recognition.

Description

Technical Field

[0001] This invention relates to the field of action recognition technology, and more specifically, to a method and system for fast keyframe correction based on action recognition. Background Technology

[0002] Digital monitoring of outdoor sports scenarios has become a core infrastructure for smart sports. The demand for sports health management is experiencing explosive growth, extending beyond just sports monitoring. In high-frequency sports such as jogging, joint and muscle injuries caused by improper posture have become the primary health threat.

[0003] Human posture recognition technology currently faces a dual bottleneck. On the one hand, dynamic capture and force table data obtained in laboratory environments can provide relatively accurate human posture data, but due to the strict environmental conditions and high equipment costs, it is difficult to implement in mass sports activities that require most outdoor sports scenarios. On the other hand, the rise of artificial intelligence in recent years has promoted the rapid development of computer vision and human posture recognition, but artificial intelligence recognition technology using monocular cameras as recognition tools has certain errors in obtaining data in complex environments with multiple people.

[0004] Therefore, in order to perform fast and accurate motion recognition in outdoor environments and obtain more accurate kinematic information, there is an urgent need for a fast keyframe correction method. This would enable the motion monitoring process of mass sports to be carried out at low cost and with low environmental requirements, in order to solve the problem of high costs in laboratory scenarios and correct the errors generated by artificial intelligence-based human posture recognition. Summary of the Invention

[0005] The technical problem to be solved by this invention is how to perform fast and accurate human posture recognition at low cost and in low environment.

[0006] This invention provides a keyframe correction method based on action recognition, comprising: Step 1, acquire the exercise video; Step 2: Use the OpenPose neural network to perform pose recognition on the motion video to obtain the coordinate information and confidence information of human joint points in each frame of video image; Step 3: Filter and anomaly labeling of human joint coordinates based on confidence information; Step 4: Apply a triple fusion correction strategy to the abnormal human joint coordinates, and output the corrected human joint coordinates and correction report.

[0007] Compared with the prior art, the method of this application has the following advantages: This application uses OpenPose neural network to perform pose recognition to obtain human joint coordinates including confidence, and performs anomaly screening of human joint coordinates through confidence. It adopts a triple fusion correction strategy to correct abnormal human joint coordinates in real time, so as to quickly and accurately obtain human joint parameters. The method of this application is free from the laboratory environment and realizes low-cost human pose recognition.

[0008] In one possible implementation, the expression for step 3, which filters and anomaly-labels all human joint coordinates based on confidence level, is as follows: .

[0009] In one possible implementation, the triple fusion correction strategy in step 4 specifically employs dynamic weight fusion of bone length constraint correction. Inverse kinematics model correction and symmetric compensation correction The coordinates of human joints are corrected using the following expression: ; In the formula, All of these represent dynamically generated weighting coefficients.

[0010] In one possible implementation, the bone length constraint correction Specifically, it includes: Femur length according to biomechanical standards and tibia length With the hip joint point and ankle joint Calculate candidate points for the knee joint with the center of the circle. The constraint equations are: ; Based on the constraint equations, with the hip joint point With the center as the point, the length of the femur A circle formed by a radius, and a point at the ankle joint. Centered on the circle, the length of the tibia The intersection of two circles, defined by the radius, is the candidate point for the knee joint. .

[0011] In one possible implementation, the inverse kinematics model correction Specifically, it includes: Construct based on joint angle The two-link model is used to solve for the optimal knee joint coordinates by minimizing the objective function, expressed as: ; In the formula, Represents the positive kinematic function, indicating the kinematics based on joint angles. Calculate the mathematical mapping of the ankle position. , Indicates the rotation angle of the hip joint. Indicates the knee flexion angle. This represents the rotation matrix.

[0012] In one possible implementation, the symmetry compensation correction specifically includes: From the human joint coordinates obtained in step 2, screen for symmetrical joints that are symmetrical to the physiological positions of abnormal joints. ; From the normally labeled human joint coordinates obtained in step 3, obtain the coordinates of joints with symmetrical physiological positions. and ; Baseline vector Thus, the normal vector is obtained. Based on normal vectors Locate the vertical plane that is symmetrical about the left and right sides of the human body, and use the vertical plane as the plane of symmetry; Based on the normal vector of the plane of symmetry and the physiologically symmetrical point The expression for correcting abnormal joint coordinates is: ; In the formula, This indicates the corrected coordinates of the abnormal joints. The normal vector of the plane of symmetry. The center point of the pelvis is used as a calculation parameter to locate the position of the plane of symmetry.

[0013] In one possible implementation, the weighting coefficients are dynamically generated. The specific expression is: ; In the formula, Represents the motion velocity mapping function, Using the exponential decay form, the expression is: ; In the formula, The weighting coefficients are constants; dynamically generated weighting coefficients. The specific expression is: ; In the formula, This represents the historical trajectory variance mapping function. Using the arctangent function form, the expression is: ; In the formula, It is a constant; Dynamically generated weight coefficients The specific expression is: , This represents the confidence level of the symmetrical point.

[0014] In one possible implementation, the correction report output in step 4 includes the coordinates of the human joint points before correction, the coordinates of the human joint points after correction, and... Weight value; Variance mapping functions are used to change the variance in fusion decision-making. of The weighting values ​​are intended to adapt the three correction schemes to different motion scenarios. A large variance in historical trajectory points indicates significant motion jitter. Weighting coefficient Increase.

[0015] A keyframe correction system based on action recognition, used to implement the aforementioned keyframe correction method based on action recognition, includes: The motion recognition module is used to perform pose recognition on motion videos using the OpenPose neural network. The abnormal joint point filtering module, connected to the motion recognition module, is used to filter the coordinates of all human joint points and anomaly annotations; The joint point correction module, connected to the abnormal joint point screening module and the motion recognition module, is used to execute a triple fusion correction strategy to correct the coordinates of human joint points with abnormal annotations. The visualization output module, connected to the motion recognition module, abnormal joint point filtering module, and joint point correction module, is used to generate and display the human joint point coordinates, abnormal joint point coordinates, corrected human joint point coordinates, and correction reports obtained from posture recognition.

[0016] Compared with existing technologies, the system of this application has the following advantages: it performs posture recognition on motion videos through the motion recognition module, and displays the coordinates of all human joints in each frame of the motion video in real time; it marks the coordinates of the identified abnormal joints through the abnormal joint filtering module, displays the abnormal joints marked in red through the visualization output module, and corrects the abnormal joints marked in red through the joint correction module, thereby obtaining a correction report and the corrected distribution of human joint coordinates. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a specific embodiment of this application. Detailed Implementation

[0018] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0019] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0020] In the embodiments of this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0021] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0022] Example 1: See Figure 1 As shown in the figure, this application discloses a keyframe correction method based on action recognition, including: Step 1: Obtain the motion video.

[0023] Step 2: Use the OpenPose neural network to perform pose recognition on the motion video to obtain the coordinate information and confidence information of human joint points in each frame of video image; The OpenPose neural network in this application includes the OpenPose multi-person pose recognition library, body_25, or the COCO model; it performs single-person and multi-person pose recognition on input video data to obtain the coordinate information and confidence information of multiple human joints such as the body, hands, and face of a single or multiple people in each frame of video image; Three-dimensional storage, The confidence level for OpenPose neural network to identify human joint coordinates.

[0024] Step 3: Filter and anomaly labeling of human joint coordinates based on confidence information; the expression is: ; In this embodiment, the coordinates of human joints that are normally marked are displayed in green; the coordinates of human joints that are marked with warnings are displayed in yellow; and the coordinates of human joints that are marked with abnormalities are displayed in red. Then, a correction procedure is initiated, and the process proceeds to step 4.

[0025] Step 4: Apply a triple fusion correction strategy to the abnormal human joint coordinates, and output the corrected human joint coordinates and correction report. Specifically, the triple fusion correction strategy employs dynamic weight fusion to correct bone length constraints. Inverse kinematics model correction and symmetric compensation correction The coordinates of human joints are corrected using the following expression: ; In the formula, All represent dynamically generated weight coefficients; where: The bone length constraint correction Specifically, it includes: Femur length according to biomechanical standards and tibia length With the hip joint point and ankle joint Calculate candidate points for the knee joint with the center of the circle. The constraint equations are: ; The constraint equations in this application are derived from the rigid skeletal characteristics of human anatomy, based on the femur length. and tibia length The invariant property during movement, input hip joint point and ankle joint Perform a geometric solution, that is, based on the constraint equations, using the hip joint points With the center as the point, the length of the femur A circle formed by a radius, and a point at the ankle joint. Centered on the circle, the length of the tibia The intersection of two circles, defined by the radius, is the candidate point for the knee joint. .

[0026] The inverse kinematics model correction Specifically, it includes: Construct based on joint angle The two-link model is used to solve for the optimal knee joint coordinates by minimizing the objective function, expressed as: ; In the formula, Represents the positive kinematic function, indicating the kinematics based on joint angles. Calculate the mathematical mapping of the ankle position. , Indicates the rotation angle of the hip joint. Indicates the knee flexion angle. This represents the rotation matrix.

[0027] The symmetry compensation correction specifically includes: From the human joint coordinates obtained in step 2, screen for symmetrical joints that are symmetrical to the physiological positions of abnormal joints. ; From the normally labeled human joint coordinates obtained in step 3, obtain the coordinates of joints with symmetrical physiological positions. and ; Baseline vector Thus, the normal vector is obtained. Based on normal vectors Locate the vertical plane that is symmetrical about the left and right sides of the human body, and use the vertical plane as the plane of symmetry; Based on the normal vector of the plane of symmetry and the physiologically symmetrical point The expression for correcting abnormal joint coordinates is: ; In the formula, This indicates the corrected coordinates of the abnormal joints. The normal vector of the plane of symmetry. The center point of the pelvis is used as a calculation parameter to locate the position of the plane of symmetry.

[0028] Dynamically generated weight coefficients The specific expression is: ; In the formula, Represents the motion velocity mapping function, Using the exponential decay form, the expression is: ; In the formula, It is a constant; in the embodiments of this application, , , ; Dynamically generated weight coefficients The specific expression is: ; In the formula, This represents the historical trajectory variance mapping function. Using the arctangent function form, the expression is: ; In the formula, It is a constant; Dynamically generated weight coefficients The specific expression is: , This represents the confidence level of the symmetrical point.

[0029] After the abnormal human joint coordinates are corrected by the triple fusion in step 4, the corrected human joint coordinates are obtained. This application embodiment also includes biomechanical verification of the corrected human joint coordinates (including bone length and joint angles); the biomechanical verification in this application embodiment includes a conflict resolution mechanism. When the joint angle exceeds the limit threshold, the results are corrected using a limited inverse kinematics model. When the bone length error is greater than 10%, the bone length constraint is used to correct the structure to a limited extent.

[0030] Output the coordinates of human joints that pass verification, report errors for human joints that fail verification, and check the original video data. The corrected human joint coordinates and a correction report are output. In this embodiment, the corrected human joint coordinates are replaced with the human joint coordinates marked in red in the corresponding video image, and the modified human joint coordinates are marked normally. The revised report includes bone length constraint corrections. Inverse kinematics model correction and symmetric compensation correction The proportion; In addition, the kinematics calculation engine maps the coordinates of the two-dimensional human joints in the motion video into three-dimensional kinematic parameters, including stride width, joint angular velocity, and center of mass displacement, forming kinematic data that is saved in CSV format.

[0031] Historical trajectory variance mapping functions are used to change the course of fusion decision-making. The purpose of the β weight value is to adapt the three correction schemes to different motion scenarios. A large variance in historical trajectory points indicates a large amplitude of motion jitter, so the IK weight β is increased, as shown in the correction report. The value indicates which method of correction has a greater weight in the correction process for different actions.

[0032] Example 2: A keyframe correction system based on action recognition, used to implement the keyframe correction method based on action recognition described in Embodiment 1, includes: The motion recognition module is used to perform pose recognition on motion videos using the OpenPose neural network. The abnormal joint point filtering module, connected to the motion recognition module, is used to filter the coordinates of all human joint points and anomaly annotations; The joint point correction module, connected to the abnormal joint point screening module and the motion recognition module, is used to execute a triple fusion correction strategy to correct the coordinates of human joint points with abnormal annotations. The visualization output module, in this embodiment of the application, is a visualization interface, which is connected to the motion recognition module, the abnormal joint point filtering module, and the joint point correction module to generate and display the human joint point coordinates, abnormal joint point coordinates, corrected human joint point coordinates, and correction reports obtained from posture recognition.

[0033] In addition, the joint point correction module in this embodiment also includes a correction function menu, through which abnormal joint points can be manually corrected, and the X-axis coordinate value and Y-axis coordinate value of the abnormal joint point can be manually corrected.

[0034] In the description of the embodiments of this application, it should be noted that the terms "inner" and "outer" and other terms indicating direction or positional relationship are based on the direction or positional relationship shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this application.

[0035] In the description of this application, the references to terms such as "an embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0036] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A keyframe correction method based on action recognition, characterized in that, include: Step 1, acquire the exercise video; Step 2: Use the OpenPose neural network to perform pose recognition on the motion video to obtain the coordinate information and confidence information of human joint points in each frame of video image; Step 3: Filter and anomaly labeling of human joint coordinates based on confidence information; Step 4: Apply a triple fusion correction strategy to the abnormal human joint coordinates, and output the corrected human joint coordinates and a correction report; the triple fusion correction strategy specifically uses dynamic weight fusion to correct bone length constraints. Inverse kinematics model correction and symmetric compensation correction The coordinates of human joints are corrected using the following expression: ； In the formula, Both represent dynamically generated weighting coefficients; dynamically generated weighting coefficients The specific expression is: ； In the formula, Represents the motion velocity mapping function, Using the exponential decay form, the expression is: ； In the formula, It is a constant; Dynamically generated weight coefficients The specific expression is: ； In the formula, This represents the historical trajectory variance mapping function. Using the arctangent function form, the expression is: ； In the formula, It is a constant; Dynamically generated weight coefficients The specific expression is: , Indicates the confidence level of the symmetrical point; Bone length constraint correction Specifically, it includes: Femur length according to biomechanical standards and tibia length With the hip joint point and ankle joint Calculate candidate points for the knee joint with the center of the circle. The constraint equations are: ； Based on the constraint equations, with the hip joint point With the center as the point, the length of the femur A circle formed by a radius, and a point at the ankle joint. Centered on the circle, the length of the tibia The intersection of two circles, defined by the radius, is the candidate point for the knee joint. ; The inverse kinematics model correction Specifically, it includes: Construct based on joint angle The two-link model is used to solve for the optimal knee joint coordinates by minimizing the objective function, expressed as: ； In the formula, Represents the positive kinematic function, indicating the kinematics based on joint angles. Calculate the mathematical mapping of the ankle position. , Indicates the rotation angle of the hip joint. Indicates the knee flexion angle. This represents the rotation matrix.

2. The keyframe correction method based on action recognition according to claim 1, characterized in that, The expression for step 3, which filters and anomaly-labels human joint coordinates based on confidence level, is as follows: 。 3. The keyframe correction method based on action recognition according to claim 1, characterized in that, The symmetry compensation correction specifically includes: From the human joint coordinates obtained in step 2, screen for symmetrical joints that are symmetrical to the physiological positions of abnormal joints. ; From the normally labeled human joint coordinates obtained in step 3, obtain the coordinates of joints with symmetrical physiological positions. and ; Baseline vector Thus, the normal vector is obtained. Based on normal vectors Locate the vertical plane that is symmetrical about the left and right sides of the human body, and use the vertical plane as the plane of symmetry; Based on the normal vector of the plane of symmetry and the physiologically symmetrical joints The expression for correcting abnormal joint coordinates is: ； In the formula, This indicates the corrected coordinates of the abnormal joints. The normal vector of the plane of symmetry. The center point of the pelvis is used as a calculation parameter to locate the position of the plane of symmetry.

4. The keyframe correction method based on action recognition according to claim 1, characterized in that, The correction report output in step 4 includes the coordinates of the human joint points before correction and the coordinates of the human joint points after correction. Weight value.

5. A keyframe correction system based on action recognition, used to implement the keyframe correction method based on action recognition as described in any one of claims 1-4, characterized in that, include: The motion recognition module is used to perform pose recognition on motion videos using the OpenPose neural network. The abnormal joint point filtering module, connected to the motion recognition module, is used to filter all human joint point coordinates and anomaly annotations; The joint point correction module, connected to the abnormal joint point screening module and the motion recognition module, is used to execute a triple fusion correction strategy to correct the coordinates of human joint points with abnormal annotations. The visualization output module, connected to the motion recognition module, abnormal joint point filtering module, and joint point correction module, is used to generate and display the human joint point coordinates, abnormal joint point coordinates, corrected human joint point coordinates, and correction reports obtained from posture recognition.

Images