A markerless motion acquisition and analysis method and system

Abstract

This invention provides a label-free motion capture and analysis method and system, comprising: identifying static and dynamic 2D keypoints based on static and dynamic image data of the test subject, and then reconstructing them into static and dynamic 3D keypoints; calculating and optimizing bone length based on the static 3D keypoints, and then scaling and adjusting a standard bone and muscle model to obtain a static bone and muscle model of the test subject; estimating bone movement posture information based on the original and optimized dynamic 3D keypoints, and matching it to the static bone and muscle model of the test subject for inverse kinematics analysis; estimating the ground support force (GRF) of the left and right feet based on the dynamic balance relationship of human movement using the results of the inverse kinematics analysis; and using the inverse kinematics analysis results and the estimated GRF values ​​of the ground support force of the left and right feet, calculating the required reaction force and torque of each joint of the human body frame by frame using the inverse dynamics method of rigid body systems, and then calculating muscle strength and activation level. This invention enables label-free motion capture and analysis.

Description

Technical Field

[0001] This invention relates to the field of motion capture and analysis technology, and in particular to a markerless motion capture and analysis method and system. Background Technology

[0002] With the development of technology, motion analysis has been widely used in rehabilitation medicine, sports training, and medical diagnostic reference. Traditional human motion analysis often relies on optical motion capture systems (such as Vicon and OptiTrack) and force platforms. Although they have high measurement accuracy, they are expensive, have high requirements for the operating environment, and usually require specific marker placement, professional operators, and dedicated laboratory space. Summary of the Invention

[0003] To address the technical problems existing in the prior art, the present invention provides a markerless motion acquisition and analysis method and system, the technical solution of which is as follows:

[0004] On the one hand, a markerless motion acquisition and analysis method is provided, which includes:

[0005] S1. Use two or more cameras to record the test subject standing still and in different movements to obtain static image data and dynamic image data of the test subject;

[0006] S2. Based on the static image data of the test subject, identify static 2D key points and reduce noise. Based on the dynamic image data of the test subject, identify dynamic 2D key point time series and reduce noise. Reconstruct the denoised static 2D key points and dynamic 2D key point time series into static 3D key points and dynamic 3D key point time series, respectively.

[0007] S3. Based on the static 3D key points, calculate the bone length and optimize it. Scale the standard bone model according to the optimization results and simultaneously adjust the parameters of the standard muscle model accordingly to obtain the static bone and muscle model of the test subject.

[0008] S4. Based on the original and optimized dynamic 3D key point time series, estimate the skeletal motion posture information, and match the estimated skeletal motion posture information to the static skeletal and muscle model of the test subject, perform inverse kinematic analysis, and obtain the original and optimized human body center of mass motion trajectory, joint motion angle and skeletal motion trajectory.

[0009] S5. Based on the dynamic balance relationship of human movement, estimate the ground support force (GRF) of the left and right feet using the optimized human center of mass trajectory and skeletal movement trajectory.

[0010] S6. Based on the optimized human body center of mass trajectory, joint motion angle, skeletal motion trajectory and ground support force GRF estimation values ​​of the left and right feet, the inverse dynamics method of rigid body system is used to calculate the reaction force and torque required by each joint of the human body frame by frame, and then calculate muscle strength and activation level.

[0011] Optionally, in step S3, the bone length is calculated and optimized based on the static 3D key points, specifically including:

[0012] Let X be the 3D reconstructed coordinates of the i-th and j-th keypoints. i Y j The length of the skeleton formed by the i-th and j-th keypoints is:

[0013]

[0014] Compared to the default bone length in the standard skeletal model By comparison, a preliminary scaling factor is calculated to make initial adjustments to the standard skeleton model:

[0015]

[0016] Where ψ is the selected set of key bones used for estimation, and M is the number of bones;

[0017] After initial adjustments, weighted least squares method is used to perform global optimization on each bone to further improve the fitting accuracy. The optimization objective is to minimize the weighted bone residuals, and physiological proportion constraints are introduced to ensure that the scaling ratio conforms to individual differences without deviating from the normal physiological range. The formula is as follows:

[0018]

[0019] Where ω ij The confidence weighting factor is used, and the constraint interval is determined by the human statistical model.

[0020] After global optimization, further local refinement is performed. For key regions, small, controllable displacement adjustments are introduced to improve local fitting and enhance the accuracy and naturalness of pose reconstruction. Let the local key point set be ζ, and the local refinement objective be:

[0021]

[0022] Where δX k This represents the local deformation of the skeleton point, and ε is the maximum allowable offset threshold.

[0023] Optionally, S4 specifically includes:

[0024] Based on the dynamic 3D keypoint time series, skeletal motion posture information is estimated, and the estimated skeletal motion posture information is matched to the static skeletal and muscle model of the test subject. Inverse kinematics analysis is performed to obtain the human center of mass trajectory, joint motion angles, and skeletal motion trajectory. The obtained joint motion angles are then used to optimize and adjust the 3D keypoint time series. The optimized 3D keypoint time series is then used to estimate skeletal motion posture information again, and the re-estimated skeletal motion posture information is matched to the static skeletal and muscle model of the test subject. Inverse kinematics analysis is performed again to obtain the optimized human center of mass trajectory, joint motion angles, and skeletal motion trajectory.

[0025] The process involves using the obtained joint motion angles to optimize and adjust the 3D keypoint time series, then using the optimized 3D keypoint time series to estimate the skeletal motion posture information again. This re-estimated skeletal motion posture information is then matched to the test subject's static skeletal and muscle model, and inverse kinematics analysis is performed again to obtain the optimized human center of mass trajectory, joint motion angles, and skeletal motion trajectory, including:

[0026] For each consecutive frame interval, a sliding window is constructed. A multi-frame joint inverse kinematics optimization strategy is used to improve the fitting accuracy and attitude trajectory continuity. The t-th frame within the window and several frames before and after it are used as optimization units, and a unified objective function is established for solving.

[0027] For each frame k, let the i-th 3D keypoint be... Its joint angle is q k The optimal position of this key point is predicted using forward kinematics. The objective is to minimize the weighted sum of squared residuals across all frames and all keypoints, while also incorporating inter-frame angle smoothing constraints. The joint optimization objective function is:

[0028]

[0029] Where q k ω: Joint angle vector of the k-th frame; i : The weighting coefficient for each key point, reflecting its importance; λ: The regularization coefficient that controls the smoothness of the trajectory; The second term is the time acceleration regularization term, which helps to maintain the smoothness of the angle sequence.

[0030] Optionally, the dynamic balance relationship based on human motion in S5, using the trajectory of the human center of mass and the skeletal motion trajectory, estimates the ground support force (GRF) of the left and right feet, specifically including:

[0031] First, establish the dynamic equilibrium equations at the center of mass:

[0032] m·a CoM(t)=F GRF (t)+m·g

[0033] Where m is mass, a CoM (t) is the acceleration of the center of mass, g is the acceleration due to gravity, and F is the acceleration due to gravity. GRG (t) represents the total ground support force to be estimated. The total ground support force can be directly derived from this formula:

[0034] F GRF (t)=m·(a CoM (t)-g)

[0035] To distribute the total ground support force reasonably to the left and right feet, the contact phase is determined based on the foot-ball contact model. Let z be the height of the ball center below the ankle. ankle (t), the radius of the sphere is r ball ,like

[0036] δ(t)=r ball -z ankte (t)>0

[0037] If the condition is determined to be in contact, the image recognition module determines the current action type, assists in determining the single-leg / double-leg support mode, and distributes the support force according to the contact state and the degree of foot penetration into the ground.

[0038] F GRF =F L (t)+F R (t)

[0039] F L (t)=α(t·F GRF (t),F R (t)=(1-α(t))·F GRF (t)

[0040] Where α(t) is the assigned weight obtained based on contact depth and action type learning;

[0041] Then, by combining the ankle position with a predefined plantar offset vector, the contact point position (COP) of the left and right feet is estimated:

[0042] P COP (t)=r ankle (t)+△r offset

[0043] Where P COP (t) is the three-dimensional spatial position vector of the ground contact point at time t, r ankle (t) is the ankle position vector at time t, derived from the reconstructed 3D keypoints; Δr offsetA fixed spatial offset vector used to translate the ankle reference point to the actual estimated foot contact area;

[0044] Finally, the estimated GRF and COP are input into the dynamic model in reverse for inverse dynamics verification. If the calculation error exceeds the preset threshold, the parameter weights are adjusted and the calculation is repeated until the fitting error meets the dynamic consistency requirements, thus realizing the time-series continuous estimation of ground reaction force.

[0045] Optionally, in step S6, the inverse dynamics method of rigid body systems is used to calculate the reaction forces and torques required for each joint of the human body frame by frame, specifically including:

[0046] For each bone, the following set of dynamic equations is established:

[0047] Translation: ∑F i =m·a cm

[0048] Rotation: ∑τ i =I·α+ω×(I·ω)

[0049] Among them, F i and τ i These represent the external forces and torques acting on each bone, where I is the moment of inertia, and a... cm ω and α are the acceleration of the center of mass, angular velocity and angular acceleration, respectively, derived from the inverse kinematic trajectory and its time derivative;

[0050] The solution process follows a recursive approach, proceeding from the ground upwards:

[0051] First, the reaction force and torque of the ankle joint are calculated using the ground support force and COP position, including:

[0052] The reaction force of the ankle joint is determined by the acceleration of the center of mass at the lower limb's mass end and the ground-light force (GRF): F ankle =m leg ·a leg +m foot ·a foot -F GRF , where m leg : Calf mass; a leg : Calf center of mass acceleration; m foot Foot quality; a foot Foot center of mass acceleration a foot ;F GRF Ground reaction force;

[0053] Ankle torque is the torque generated by the distance between the COP (Coordinate of Optimal Ankle Response) point of application and the ankle joint position: τ ankle =r COP ×FGRF , where r COP :COP vector to ankle joint, r COP =p COP -p ankle p COP : Vector position of COP; p ankle Vector position of the ankle joint;

[0054] Then, the knee joint, hip joint, and torso are recursively calculated frame by frame to obtain the joint torque of each joint at each moment, thus completing the inverse dynamics solution of the entire motion sequence.

[0055] Optionally, step S6 further calculates muscle strength and activation level, specifically including:

[0056] A nonlinear muscle coupling method is introduced for calculation:

[0057] a = f(W·c)

[0058] in A muscle activation vector of length n, representing the activation level of each muscle; Muscle synergy weight matrix, where each column represents a synergy pattern and k represents the number of synergies; Cooperative activation vector; f(·): nonlinear activation function;

[0059] The muscle strength model is as follows:

[0060] F i =a i ·F i，max ·f l (l i )·f v (υ i )

[0061] Where a i =f([W·c]) i ): Activation of the i-th muscle; F i，max f is the maximum isometric tension of the i-th muscle; l (l i ): Length-force relationship; f v (v i ): Velocity-force relationship;

[0062] Moment balance constraint equations:

[0063]

[0064] Where, r i，j (q): The lever arm of the i-th muscle at the j-th joint, determined by the joint angle q; F i (a i , li v i ): The pulling force generated by the i-th muscle depends on the activation value a. i Muscle length l i contraction speed v i ; The net joint torque is obtained from inverse dynamics;

[0065] The objective function to be optimized is:

[0066]

[0067] The first term minimizes the total activation amount; the second term is a regularization term that controls the size of the co-activation value to avoid overfitting; and λ is a regularization parameter used to control the trade-off between activation energy and sparsity.

[0068] On the other hand, a markerless motion acquisition and analysis system is provided, the system comprising:

[0069] The recording module is used to record the test subject standing still and in different movements using two or more cameras, so as to obtain static image data and dynamic image data of the test subject;

[0070] The recognition and reconstruction module is used to identify static 2D key points and reduce noise based on the static image data of the test subject, identify dynamic 2D key point time series based on the dynamic image data of the test subject and reduce noise, and reconstruct the denoised static 2D key points and dynamic 2D key point time series into static 3D key points and dynamic 3D key point time series, respectively.

[0071] The first calculation module is used to calculate the bone length based on the static 3D key points, optimize and adjust it, scale the standard bone model according to the optimization and adjustment results, and simultaneously adjust the various parameters of the standard muscle model accordingly to obtain the static bone and muscle model of the test subject.

[0072] The first estimation module is used to estimate the skeletal motion posture information based on the original and optimized dynamic 3D key point time series, and match the estimated skeletal motion posture information to the static skeletal and muscle model of the test subject to perform inverse kinematic analysis to obtain the original and optimized human body center of mass motion trajectory, joint motion angle and skeletal motion trajectory.

[0073] The second estimation module is used to estimate the ground support force (GRF) of the left and right feet based on the dynamic balance relationship of human movement and using the optimized human center of mass trajectory and skeletal movement trajectory.

[0074] The second calculation module is used to calculate the reaction force and torque required by each joint of the human body frame by frame based on the optimized human body center of mass trajectory, joint motion angle, skeletal motion trajectory and ground support force GRF estimation value of the left and right feet, using the inverse dynamics method of rigid body system, and then calculate muscle strength and activation level.

[0075] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described markerless motion acquisition and analysis method.

[0076] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described markerless motion acquisition and analysis method.

[0077] The beneficial effects of the technical solution provided by this invention include at least the following:

[0078] 1) The present invention provides a markerless motion acquisition and analysis method and system, which realizes an integrated closed-loop system from motion acquisition and attitude estimation to mechanical analysis and result output, thereby improving the overall operability and portability.

[0079] 2) This invention uses two or more cameras to record data and combine them with deep learning algorithms to achieve the analysis of inverse kinematics and inverse dynamics of the human body, the estimation of ground support force, the calculation of reaction forces and torques required by each joint of the human body, and the calculation of muscle strength and activation level without the need for force platforms and marker points. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1 This is a flowchart of a markerless motion acquisition and analysis method provided by an embodiment of the present invention;

[0082] Figure 2 This is a general block diagram of a markerless motion acquisition and analysis method provided in an embodiment of the present invention;

[0083] Figure 3 This is a diagram showing the layout of the data acquisition equipment for a markerless motion acquisition and analysis method provided in an embodiment of the present invention.

[0084] Figure 4This is a flowchart of the data acquisition process of a markerless motion acquisition and analysis method provided in an embodiment of the present invention;

[0085] Figure 5 This is a structured folder diagram of a markerless motion acquisition and analysis method provided in this embodiment of the invention;

[0086] Figure 6 This is a visualization interface diagram of the inverse kinematics analysis results of a markerless motion acquisition and analysis method provided in an embodiment of the present invention;

[0087] Figure 7 This is a graph of ground support force for a markerless motion acquisition and analysis method provided in an embodiment of the present invention;

[0088] Figure 8 This is a block diagram of a markerless motion acquisition and analysis system provided in an embodiment of the present invention;

[0089] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0090] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0091] This invention provides a markerless motion acquisition and analysis method, which can be implemented by an electronic device, such as a terminal or a server. Figure 1 The diagram shown is a flowchart of the method. The processing flow may include the following steps:

[0092] S1. Use two or more cameras to record the test subject standing still and in different movements to obtain static image data and dynamic image data of the test subject;

[0093] like Figure 3 and Figure 4 As shown, in this embodiment of the invention, the cameras used for recording are arranged at an angle of ±35° in front of the recording area (2 cameras), about 2m apart; if there are three cameras, one more is added directly in front of the recording area, and if there are more, they are arranged symmetrically as needed to ensure that the video preview area can see the captured effect, and all cameras must include the tester.

[0094] First, open the software, fill in the test subject's basic information, including name, age, gender, height, weight, etc., and save it accordingly.

[0095] Select the number of cameras to record from and the cameras to record from, and ensure that the tester can be seen in the software preview area.

[0096] Place the chessboard grid in the center of the recording area and begin recording. Remove the chessboard grid after recording. The chessboard grid serves to calibrate the extrinsic parameters of the recording cameras. Both cameras can calculate the rotation matrix, translation vector, and Euler angle parameters by recognizing the chessboard grid and its parameters, and establish a spatial coordinate system using the chessboard grid to complete the establishment of the recognition space.

[0097] The test subject stands in a triangle stance (feet shoulder-width apart, standing upright with arms outstretched, forming a triangle with the ground). Static recording of the test subject is performed, followed by preparation for dynamic recording.

[0098] According to the recording steps, the recording includes dynamic postures of different actions such as walking, sitting up (sitting in a chair and then standing up), and squatting. Walking should include a complete gait, sitting up should be repeated three times (i.e., repeating the process: sitting in a chair - standing up - sitting up - sitting up - sitting up - standing up), and squatting should be repeated three times (i.e., standing - squatting - standing - squatting - standing - squatting).

[0099] Recorded files are saved in a structured directory, with consistent structure across different testers, facilitating subsequent data analysis and result storage. Figure 5 As shown.

[0100] S2. Based on the static image data of the test subject, identify static 2D key points and reduce noise. Based on the dynamic image data of the test subject, identify dynamic 2D key point time series and reduce noise. Reconstruct the denoised static 2D key points and dynamic 2D key point time series into static 3D key points and dynamic 3D key point time series, respectively.

[0101] This invention can utilize the HRNet deep learning algorithm to identify static 2D key points and dynamic 2D key point time series, identifying 18 to 33 key points (including shoulders, knees, hip joints, elbow joints, etc.);

[0102] The 2D keypoints output by deep learning networks are often limited by factors such as the frame rate of the acquisition device, lighting occlusion, and rapid target movement, resulting in noise, jitter, or loss of keypoint trajectories. Therefore, this invention employs a two-stage noise reduction strategy: the first stage uses a confidence screening mechanism to eliminate unreliable keypoint predictions; the second stage introduces a nonlinear temporal filtering module.

[0103] The time series of denoised static 2D key points and dynamic 2D key points are reconstructed into static 3D key points and dynamic 3D key points, respectively. 3D key points can be reconstructed by combining 2D key points from multiple cameras using triangulation, camera calibration, and weighted least squares. The specific reconstruction process is based on existing technology and will not be elaborated here.

[0104] S3. Based on the static 3D key points, calculate the bone length and optimize it. Scale the standard bone model according to the optimization results and simultaneously adjust the parameters of the standard muscle model accordingly to obtain the static bone and muscle model of the test subject.

[0105] Optionally, in step S3, the bone length is calculated and optimized based on the static 3D key points, specifically including:

[0106] Let X be the 3D reconstructed coordinates of the i-th and j-th keypoints. i Y j The length of the skeleton formed by the i-th and j-th keypoints is:

[0107]

[0108] Compared to the default bone length in the standard skeletal model By comparison, a preliminary scaling factor is calculated to make initial adjustments to the standard skeleton model:

[0109]

[0110] Where ψ is the selected set of key bones used for estimation, and M is the number of bones;

[0111] After initial adjustments, weighted least squares method is used to perform global optimization on each bone to further improve the fitting accuracy. The optimization objective is to minimize the weighted bone residuals, and physiological proportion constraints are introduced to ensure that the scaling ratio conforms to individual differences without deviating from the normal physiological range. The formula is as follows:

[0112]

[0113] Where ω ij The confidence weighting factor is used, and the constraint interval is determined by the human statistical model.

[0114] After global optimization, further local refinement is performed. For key areas (such as the pelvis and hands), small, controllable displacement adjustments are introduced to improve local fitting and enhance the accuracy and naturalness of pose reconstruction. Let the local key point set be ζ, and the local refinement objective be:

[0115]

[0116] Where δX k This represents the local deformation of the skeleton point, and ε is the maximum allowable offset threshold.

[0117] S4. Based on the original and optimized dynamic 3D key point time series, estimate the skeletal motion posture information, and match the estimated skeletal motion posture information to the static skeletal and muscle model of the test subject, perform inverse kinematic analysis, and obtain the original and optimized human body center of mass motion trajectory, joint motion angle and skeletal motion trajectory.

[0118] Optionally, S4 specifically includes:

[0119] Based on the dynamic 3D keypoint time series, skeletal motion posture information is estimated. This estimated skeletal motion posture information is then matched to the test subject's static skeletal and muscle model for inverse kinematic analysis, yielding the human center of mass trajectory, joint motion angles, and skeletal motion trajectory. The obtained joint motion angles are then used to optimize and adjust the 3D keypoint time series. The optimized 3D keypoint time series is then used again to estimate skeletal motion posture information. This re-estimated skeletal motion posture information is then matched to the test subject's static skeletal and muscle model for another round of inverse kinematic analysis, resulting in optimized human center of mass trajectory, joint motion angles, and skeletal motion trajectory. Figure 6 The figure shown is a visualization interface diagram of the inverse kinematics results of a markerless motion acquisition and analysis method provided in an embodiment of the present invention.

[0120] The process involves using the obtained joint motion angles to optimize and adjust the 3D keypoint time series, then using the optimized 3D keypoint time series to estimate the skeletal motion posture information again. This re-estimated skeletal motion posture information is then matched to the test subject's static skeletal and muscle model, and inverse kinematics analysis is performed again to obtain the optimized human center of mass trajectory, joint motion angles, and skeletal motion trajectory, including:

[0121] For each consecutive frame interval, a sliding window is constructed. A multi-frame joint inverse kinematics optimization strategy is used to improve the fitting accuracy and attitude trajectory continuity. The t-th frame within the window and several frames before and after it are used as optimization units, and a unified objective function is established for solving.

[0122] For each frame k, let the i-th 3D keypoint be... Its joint angle is q k The optimal position of this key point is predicted using forward kinematics. The objective is to minimize the weighted sum of squared residuals across all frames and all keypoints, while also incorporating inter-frame angle smoothing constraints. The joint optimization objective function is:

[0123]

[0124] Where q k ω: Joint angle vector of the k-th frame; i: The weighting coefficient for each key point, reflecting its importance; λ: The regularization coefficient that controls the smoothness of the trajectory; The second term is the time acceleration regularization term, which helps to maintain the smoothness of the angle sequence.

[0125] The joint optimization problem in this embodiment of the invention is solved window-by-window using the trust region method, and the optimal solution for each frame is output by sliding the center of the window. Compared with solving the problem independently frame by frame, this method can more effectively suppress jitter, correct errors in occluded frames, and significantly improve the continuity and robustness of the inverse kinematics solution.

[0126] S5. Based on the dynamic balance relationship of human movement, estimate the ground support force (GRF) of the left and right feet using the optimized human center of mass trajectory and skeletal movement trajectory.

[0127] Optionally, the dynamic balance relationship based on human motion in S5, using the trajectory of the human center of mass and the skeletal motion trajectory, estimates the ground support force (GRF) of the left and right feet, specifically including:

[0128] First, establish the dynamic equilibrium equations at the center of mass:

[0129] m·a CoM (t)=F GRF (t)+m·g

[0130] Where m is mass, a CoM (t) is the acceleration of the center of mass, g is the acceleration due to gravity, and F is the acceleration due to gravity. GRF (t) represents the total ground support force to be estimated. The total ground support force can be directly derived from this formula:

[0131] F GRF (t)=m·(a CoM (t)-g)

[0132] To rationally distribute the total ground support force to the left and right feet, the contact phase is determined based on the foot contact ball model (a simplified biomechanical modeling method; contact ball: simplifying key areas such as the foot into a sphere, making it easier to determine whether there is contact with the ground). The height of the contact ball center below the ankle is set as z. ankle (t), the radius of the sphere is r ball ,like

[0133] δ(t)=r ball -z ankle (t)>0

[0134] If a contact state is identified, the image recognition module determines the current action type (e.g., walking, running, jumping) to assist in determining the single-leg / double-leg support mode. The support force is then distributed according to the contact state and the degree of foot penetration into the ground.

[0135] FGRF =F L (t)+F R (t)

[0136] F L (t)=α(t·F GRF (t),F R (t)=(1-α(t))·F GRF (t)

[0137] Where α(t) is the assigned weight obtained based on contact depth and action type learning;

[0138] Then, by combining the ankle position with a predefined plantar offset vector, the contact point position (COP) of the left and right feet is estimated:

[0139] P COP (t)=r ankle ((t)+△r offset

[0140] Where P COP (t) is the three-dimensional spatial position vector of the ground contact point at time t, r ankle (t) is the ankle position vector at time t, derived from the reconstructed 3D keypoints; Δr offset A fixed spatial offset vector used to translate the ankle reference point (such as the lateral malleolus) to the actual estimated plantar contact area;

[0141] Finally, the estimated GRF and COP are input into the dynamic model in reverse for inverse dynamics verification. If the calculation error exceeds the preset threshold, the parameter weights (such as the size of the contact ball, the confidence level of motion recognition, the strength of smoothing filter, etc.) are adjusted and re-estimated until the fitting error meets the dynamic consistency requirements, thus realizing the time-series continuous estimation of ground reaction force.

[0142] S6. Based on the optimized human body center of mass trajectory, joint motion angle, skeletal motion trajectory and ground support force GRF estimation values ​​of the left and right feet, the inverse dynamics method of rigid body system is used to calculate the reaction force and torque required by each joint of the human body frame by frame, and then calculate muscle strength and activation level.

[0143] Optionally, in step S6, the inverse dynamics method of rigid body systems is used to calculate the reaction forces and torques required for each joint of the human body frame by frame, specifically including:

[0144] For each bone, the following set of dynamic equations is established:

[0145] Translation: ∑F i =m·a cm

[0146] Rotation: ∑τi =I·α+ω×(I·ω)

[0147] Among them, F i and τ i These represent the external forces and torques acting on each bone, where I is the moment of inertia, and a... cm ω and α are the acceleration of the center of mass, angular velocity and angular acceleration, respectively, derived from the inverse kinematic trajectory and its time derivative;

[0148] The solution process follows a recursive approach, proceeding from the ground upwards:

[0149] First, the reaction force and torque of the ankle joint are calculated using the ground support force and COP position, including:

[0150] The reaction force of the ankle joint is determined by the acceleration of the center of mass at the lower limb's mass end and the ground-light force (GRF): F ankle =m leg ·a leg +m foot ·a foot -F GRF , where m leg : Calf mass; a leg : Calf center of mass acceleration; m foot Foot quality; a foot Foot center of mass acceleration a foot ;F GRF Ground reaction force;

[0151] Ankle torque is the torque generated by the distance between the COP (Coordinate of Optimal Ankle Response) point of application and the ankle joint position: τ ankle =r COP ×F GRF , where r COP :COP vector to ankle joint, r COP =p COP -p ankle p COP : Vector position of COP; p ankle Vector position of the ankle joint;

[0152] Then, the knee joint, hip joint, and torso are recursively calculated frame by frame to obtain the joint torque of each joint at each moment, thus completing the inverse dynamics solution of the entire motion sequence.

[0153] Optionally, step S6 further calculates muscle strength and activation level, specifically including:

[0154] A nonlinear muscle coupling method is introduced for calculation:

[0155] a = f(W·c)

[0156] in A muscle activation vector of length n, representing the activation level of each muscle; Muscle synergy weight matrix, where each column represents a synergy pattern and k represents the number of synergies; Cooperative activation vector; f(·): nonlinear activation function (mainly including ReLU, Sigmoid, Tanh, etc.);

[0157] The muscle strength model is as follows:

[0158] F i =a i ·F i，max ·f l (l i )·f v (υ i )

[0159] Where a i =f([W·c]) i ): Activation of the i-th muscle; F i，max f is the maximum isometric tension of the i-th muscle; l (l i ): Length-force relationship; f v (v i ): Velocity-force relationship;

[0160] Moment balance constraint equations:

[0161]

[0162] Where, r i，j (q): The lever arm of the i-th muscle at the j-th joint, determined by the joint angle q; F i (a i , l i v i ): The pulling force generated by the i-th muscle depends on the activation value a. i Muscle length l i contraction speed v i ; The net joint torque is obtained from inverse dynamics;

[0163] The objective function to be optimized is:

[0164]

[0165] The first term minimizes the total activation (muscle metabolic load); the second term is a regularization term that controls the magnitude of co-activation values ​​to avoid overfitting; and λ is a regularization parameter used to control the trade-off between activation energy and sparsity.

[0166] This invention can plot different joint movements, muscle strength, activation levels, and ground support forces (e.g., based on the results calculated in the preceding steps). Figure 7 (See diagrams for knee joint contact torque, etc.)

[0167] like Figure 8 As shown, this embodiment of the invention also provides a markerless motion acquisition and analysis system, the system comprising:

[0168] The recording module 810 is used to record the test subject standing still and in different movements using two or more cameras to obtain static image data and dynamic image data of the test subject.

[0169] The recognition and reconstruction module 820 is used to identify static 2D key points and reduce noise based on the static image data of the test subject, identify dynamic 2D key point time series based on the dynamic image data of the test subject and reduce noise, and reconstruct the denoised static 2D key points and dynamic 2D key point time series into static 3D key points and dynamic 3D key point time series, respectively.

[0170] The first calculation module 830 is used to calculate the bone length based on the static 3D key points, optimize and adjust it, scale the standard bone model according to the optimization and adjustment results, and simultaneously adjust the various parameters of the standard muscle model accordingly to obtain the static bone and muscle model of the test subject.

[0171] The first estimation module 840 is used to estimate the skeletal motion posture information based on the original and optimized dynamic 3D key point time series, and match the estimated skeletal motion posture information to the static skeletal and muscle model of the test subject to perform inverse kinematic analysis to obtain the original and optimized human body center of mass motion trajectory, joint motion angle and skeletal motion trajectory.

[0172] The second estimation module 850 is used to estimate the ground support force (GRF) of the left and right feet based on the dynamic balance relationship of human movement and using the optimized human center of mass trajectory and skeletal movement trajectory.

[0173] The second calculation module 860 is used to calculate the reaction force and torque required by each joint of the human body frame by frame based on the optimized human body center of mass motion trajectory, joint motion angle, skeletal motion trajectory and ground support force GRF estimation value of the left and right feet, and then calculate muscle strength and activation level.

[0174] The labelless motion acquisition and analysis system provided in this embodiment of the invention has a functional structure that corresponds to the labelless motion acquisition and analysis method provided in this embodiment of the invention, and will not be described again here.

[0175] Figure 9This is a schematic diagram of the structure of an electronic device 900 provided in an embodiment of the present invention. The electronic device 900 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 901 and one or more memories 902. The memory 902 stores at least one instruction, which is loaded and executed by the processor 901 to implement the steps of the above-described markerless motion acquisition and analysis method.

[0176] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the above-described tagless motion acquisition and analysis method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0177] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A markerless motion capture analysis method, characterized by, The method comprises: S1, using two or more cameras to record the tester standing still and different actions respectively, obtaining tester static image data and dynamic image data; S2, according to the tester static image data, identifying static 2D key points and denoising, according to the tester dynamic image data, identifying dynamic 2D key point time series and denoising, and reconstructing the denoised static 2D key points and dynamic 2D key point time series into static 3D key points and dynamic 3D key point time series respectively; S3, based on the static 3D key points, calculating the bone length and optimizing the adjustment, scaling the standard bone model according to the optimization adjustment result, and synchronously adjusting the corresponding parameters of the standard muscle model to obtain the tester static bone and muscle model; S4, according to the original and optimized dynamic 3D key point time series, estimating the bone motion posture information, and matching the estimated bone motion posture information to the tester static bone and muscle model, and performing inverse kinematics analysis to obtain the original and optimized human body center of mass motion trajectory, joint motion angle and bone motion trajectory; S5, based on the dynamic balance relationship of human motion, using the optimized human body center of mass motion trajectory and bone motion trajectory to estimate the ground reaction force GRF of the left and right feet; S6, based on the optimized human body center of mass motion trajectory, joint motion angle, bone motion trajectory and left and right foot ground reaction force GRF estimate value, using the inverse dynamics method of rigid body system to calculate the required reaction force and moment of each joint of the human body frame by frame, and then calculating the muscle strength and activation degree; In S3, based on the static 3D key points, the bone length is calculated and the optimization adjustment is performed, which specifically includes: Let the three-dimensional reconstruction coordinates of the ith,jthkey point be X i ,Y j The length of the skeleton formed by the ith,jthkey point is: with the default bone lengths in the standard bone model In comparison, the preliminary scaling factor is calculated to preliminarily adjust the standard bone model: Wherein ψ is the selected key bone set for estimation, and M is the number of bones; After preliminary adjustment, the weighted least squares method is used to globally optimize each bone to further improve the fitting accuracy, and the optimization target is to minimize the weighted bone residual, and the physiological proportion constraint is introduced to ensure that the scaling ratio conforms to individual differences and does not deviate from the physiological conventional range, and the formula is as follows: where ω ij is a confidence weighting factor, and the constraint interval is determined by a human body statistical model. After global optimization, further local refinement is performed, and for the key area, a small controllable displacement adjustment is introduced to improve the local fitting degree and improve the accuracy and naturalness of posture reconstruction, and the local key point set is ζ, and the local refinement target is: where δX k represents the local deformation of the skeleton point, and ε is the maximum allowed offset threshold.

2. The method of claim 1, wherein, S4 specifically includes: According to the dynamic 3D key point time series, the bone motion posture information is estimated, and the estimated bone motion posture information is matched to the tester static bone and muscle model, and inverse kinematics analysis is performed to obtain the human body center of mass motion trajectory, joint motion angle and bone motion trajectory, and then the obtained joint motion angle is used to optimize and adjust the 3D key point time series, and then the 3D key point time series after optimization and adjustment is used to estimate the bone motion posture information again, and the estimated bone motion posture information is matched to the tester static bone and muscle model, and inverse kinematics analysis is performed again to obtain the optimized human body center of mass motion trajectory, joint motion angle and bone motion trajectory; The re-use obtained joint movement angle optimizes adjustment of the 3D key point time sequence, the re-use optimizes the adjusted 3D key point time sequence, re-estimates the skeletal motion posture information, matches the re-estimated skeletal motion posture information to the tester static skeleton and muscle model, re-performs inverse kinematics analysis, and obtains the optimized human body center of mass motion trajectory, joint movement angle and skeletal motion trajectory, including: A sliding window is constructed for each time continuous frame, a multi-frame joint inverse kinematics optimization strategy is used to improve fitting accuracy and posture trajectory continuity, the tth frame and a plurality of frames before and after the tth frame are jointly used as an optimization unit, a unified objective function is established for solving: For each frame k, let the ith 3D keypoint be with joint angle q k The optimized position of this keypoint is predicted by forward kinematics as The goal is to minimize the weighted sum of squared residuals for all frames and all keypoints, while introducing inter-frame angle smoothness constraints. The joint optimization objective function is: where q k : joint angle vector of the k-th frame; ω i : weighting coefficient of each key point, reflecting its importance; λ: positive regular term coefficient to control the smoothness of the trajectory; the second term is the time acceleration regular term, which helps to keep the angle sequence smooth.

3. The method of claim 1, wherein, The S5 uses the human body center of mass trajectory and the skeletal motion trajectory to estimate the ground reaction force GRF of the left and right feet, and specifically includes: First, the dynamic balance equation at the center of mass is established: m·a CoM (t) = F GRF (t) + m·g where m is the mass, a CoM (t) is the center of mass acceleration, g is the gravitational acceleration, F GRF (t) is the total ground support force to be estimated, which is directly back calculated by this equation: F GRF (t) = m · (a CoM (t)-g) To distribute the total ground support force to the left and right feet reasonably, the contact phase is determined based on a foot contact ball model, and the height of the contact ball center below the ankle is defined as z ankle (t), and the ball radius is r ball , if δ(t) = r ball - z ankle (t) > 0 Then, the current action type is determined according to the image recognition module, and the single-leg / double-leg support mode is determined, and the support force is distributed according to the contact state and the degree of foot invasion into the ground: F GRF = F L (t) + F R (t) F L (t) = a(t) · F GRF (t), F R (t) = (1 - a(t)) · F GRF (t) Wherein α(t) is a distribution weight learned based on the contact depth and the action type; Subsequently, the contact point position COP of the left and right feet is estimated by combining the ankle position and the predefined foot bottom offset vector: P COP (t) = r ankle (t) + Δr offset where P COP (t) is the three-dimensional spatial position vector of the ground contact point at time t, r ankle (t) is the ankle position vector at time t, derived from the reconstructed 3D key points; Δr offset : a fixed spatial offset vector to translate the ankle reference point to the actual estimated foot-ground contact area; Finally, the estimated GRF and COP are input into the dynamic model in reverse to perform inverse dynamics verification, and if the calculation error exceeds the preset threshold, the parameter weight is adjusted and re-estimated until the fitting error meets the dynamic consistency requirement, thereby realizing the time sequence continuous estimation of the ground reaction force.

4. The method of claim 3, wherein, The S6 uses the inverse dynamics method of the rigid body system to calculate the reaction force and torque required by each joint of the human body frame by frame, and specifically includes: For each skeleton, the following dynamic equation set is established: Translation: Sum of forces = m * a i = m·a cm Rotation:∑τ i = I · α + ω × (I · ω) where F i and τ i are the external forces and moments on each bone, I is the moment of inertia, a cm , w, and a are the linear and angular accelerations, velocities, and angles, respectively, from inverse kinematics and their time derivatives; The solving sequence adopts a step-by-step recursive manner from the ground to the top: First, the reaction force and torque of the ankle joint are calculated by using the ground support force and the COP position, including: The ground reaction force of the ankle joint is determined by the mass center acceleration of the lower limb mass end and the GRF: F ankle = m leg · a leg + m foot · a foot - F GRF , where m leg : calf mass; a leg : calf mass center acceleration; m foot : foot mass; a foot : foot mass center acceleration a foot ; F GRF : ground reaction force; Ankle joint torque is the torque due to the distance between the COP of the GRF action point and the ankle joint position: τ ankle = r COP × F GRF , where r COP : vector from COP to ankle joint, r COP = p COP -p ankle , P COP : vector position of COP; p ankle : vector position of ankle joint; Subsequently, the knee joint, the hip joint and the trunk part are recursively calculated in sequence, and the joint torque of each joint at each time is calculated to complete the inverse dynamics solution of the entire motion sequence.

5. The method of claim 4, wherein, The S6 further calculates the muscle strength and activation degree, and specifically includes: The nonlinear muscle coupling method is introduced for calculation: a = f(W·c) wherein a muscle activation vector of length n, representing the activation level of each muscle; a muscle synergy weight matrix, each column being a synergy mode, k representing the number of synergies; a synergy activation vector; f(·): a non-linear activation function; The muscle force model is: F i = a i ·F i,max ·f l (l i )·f v (v i ) where a i = f([W · c i ): activation of the ith muscle; F i,max is the maximum isometric force of the ith muscle; f l (l i ): length-force relationship; f v (v i ): velocity-force relationship; The torque balance constraint equation is: where r i，j (q) : the moment arm of the ith muscle at the jth joint, determined by the joint angle q; F i (a i , l i , v i ) : the force generated by the ith muscle, dependent on the activation value a i , muscle length l i , and contraction velocity v i ; the net joint torque from inverse dynamics; The optimization objective function is: Wherein the first term is the minimization of the total activation amount; the second term is a regularization term, which controls the size of the collaborative activation value to avoid overfitting, and λ is a regularization parameter for controlling the trade-off between activation energy and sparsity.

6. A markerless motion capture analysis system characterized by, The system includes: A recording module for recording the tester standing statically and different actions using two or more cameras to obtain tester static image data and dynamic image data; a recognition reconstruction module, configured to recognize and denoise static 2D key points according to the tester static image data, recognize and denoise dynamic 2D key point time series according to the tester dynamic image data, and reconstruct the denoised static 2D key points and dynamic 2D key point time series into static 3D key points and dynamic 3D key point time series respectively; a first calculation module, configured to calculate bone lengths based on the static 3D key points, and perform optimization adjustment, scale a standard bone model according to the optimization adjustment result, and correspondingly adjust parameters of a standard muscle model to obtain a tester static bone and muscle model; a first estimation module, configured to estimate bone motion posture information according to the original and optimized dynamic 3D key point time series, match the estimated bone motion posture information to the tester static bone and muscle model, perform inverse kinematics analysis, and obtain original and optimized human body center of mass motion trajectories, joint motion angles, and bone motion trajectories; a second estimation module, configured to estimate left and right foot ground reaction forces (GRFs) based on a dynamic balance relationship of human body motion, and using the optimized human body center of mass trajectory and bone motion trajectory; a second calculation module, configured to calculate, frame by frame, reaction forces and moments required by each joint of the human body by using an inverse dynamics method of a rigid body system, based on the optimized human body center of mass motion trajectory, joint motion angle, bone motion trajectory, and estimated left and right foot ground reaction forces (GRFs), and further calculate muscle strength and activation level; the first calculation module is specifically configured to: Let the three-dimensional reconstruction coordinates of the ith,jth key point be X i ,Y j The length of the skeleton formed by the ith,jth key point is: with the default bone lengths in the standard bone model In comparison, a preliminary scaling factor is computed to preliminarily adjust the standard bone model: wherein ψ is a selected key bone set for estimation, and M is the number of bones; after the preliminary adjustment, a weighted least squares method is used to globally optimize each bone to further improve fitting accuracy, and an optimization target is to minimize weighted bone residuals, and a physiological proportion constraint is introduced to ensure that the scaling proportion complies with individual differences and does not deviate from a physiological conventional range, and a formula is as follows: where ω ij is a confidence weighting factor, and the constraint interval is determined by a human body statistical model. after the global optimization is completed, further local refinement is performed, a micro-controllable displacement adjustment is introduced for a key area to improve local fitting degree and improve the accuracy and naturalness of posture reconstruction, a local key point set is ζ, and a local refinement target is as follows: where δX k represents the local deformation of the skeleton point, and ε is the maximum allowed offset threshold. 7.An electronic device, comprising a processor and a memory having stored therein at least one instruction, wherein, the at least one instruction is loaded and executed by the processor to implement the markerless motion acquisition and analysis method according to any one of claims 1-5.

8. A computer-readable storage medium having stored therein at least one instruction, wherein the at least one instruction is loaded and executed by the processor to implement the markerless motion acquisition and analysis method according to any one of claims 1-5.

Images