Mark-free action acquisition and analysis method and system

Abstract

The invention provides an unmarked action acquisition and analysis method and system, and the method comprises the steps: recognizing static and dynamic 2D key points according to the static and dynamic image data of a tester, and reconstructing the static and dynamic 2D key points into static and dynamic 3D key points; based on the static 3D key points, calculating and optimizing the skeleton length, and performing scaling adjustment on the standard skeleton and muscle model to obtain a static skeleton and muscle model of the testee; estimating skeleton motion attitude information according to the original and optimized dynamic 3D key points, and matching the skeleton motion attitude information to a static skeleton and muscle model of a tester to perform inverse kinematics analysis; estimating ground support force GRF of left and right feet by using an inverse kinematics analysis result based on a dynamic balance relationship of human motion; on the basis of an inverse kinematics analysis result and ground support force GRF estimated values of the left and right feet, counter-acting force and torque required by each joint of the human body are calculated frame by frame by adopting an inverse kinematics method of a rigid body system, and then muscle strength and activation degree are calculated. According to the invention, unmarked action acquisition and analysis can be carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of motion acquisition and analysis, and in particular to a markerless motion acquisition and analysis method and system. Background Art

[0002] With the advancement of technology, motion analysis is widely used in rehabilitation medicine, sports training, medical diagnostic reference, and other fields. Traditional human motion analysis often relies on optical motion capture systems (such as Vicon and OptiTrack) and force platforms. Although these systems offer high measurement accuracy, they are expensive, have strict requirements for the operating environment, and typically require a specific marker arrangement, professional operators, and dedicated laboratory space. Summary of the Invention

[0003] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides a markerless motion collection and analysis method and system, and the technical solution is as follows:

[0004] In one aspect, a markerless motion collection and analysis method is provided, the method comprising:

[0005] S1. Use two or more cameras to record the tester's static standing and different movements, respectively, to obtain static image data and dynamic image data of the tester;

[0006] S2. Based on the static image data of the test subject, identify static 2D key points and reduce noise; based on the dynamic image data of the test subject, identify a time series of dynamic 2D key points and reduce noise; and reconstruct the static 2D key points and the time series of dynamic 2D key points after noise reduction into static 3D key points and a time series of dynamic 3D key points, respectively;

[0007] S3. Calculate the bone length based on the static 3D key points and perform optimization adjustments. Scaling the standard bone model according to the optimization adjustment results and adjusting the parameters of the standard muscle model accordingly to obtain the static bone and muscle model of the test subject.

[0008] S4. Estimating skeletal motion posture information based on the original and optimized dynamic 3D key point time series, and matching the estimated skeletal motion posture information to the test subject's static skeletal and muscle model to perform inverse kinematic analysis to obtain the original and optimized human center of mass motion trajectories, joint motion angles, and skeletal motion trajectories;

[0009] S5. Based on the dynamic balance relationship of human motion, using the optimized human center of mass trajectory and skeletal motion trajectory, estimate the ground support force GRF of the left and right feet;

[0010] S6. Based on the optimized human center of mass motion trajectory, joint motion angles, skeletal motion trajectory, and estimated ground support force GRF values ​​of the left and right feet, the inverse dynamics method of the rigid body system is used to calculate the reaction force and torque required for each joint of the human body frame by frame, and then calculate the muscle strength and activation degree.

[0011] Optionally, the step S3 calculates the bone length based on the static 3D key points and performs optimization adjustment, specifically including:

[0012] Let the 3D reconstruction coordinates of the i-th and j-th key points be X i , Y j , then the length of the skeleton composed of the i-th and j-th key points is:

[0013]

[0014] Compared to the default bone length in the standard bone model For comparison, calculate the initial scaling factor to make preliminary adjustments to the standard skeleton model:

[0015]

[0016] Where ψ is the set of key bones selected for estimation, and M is the number of bones;

[0017] After the initial adjustment, the weighted least squares method is used to perform global optimization on each skeleton to further improve the fitting accuracy. The optimization goal is to minimize the weighted skeleton residuals, and the physiological proportion constraint is introduced to ensure that the scaling ratio is consistent with individual differences and does not deviate from the physiological normal range. The formula is as follows:

[0018]

[0019] where ω ij is the confidence weighting factor, and the constraint interval is determined by the human statistical model;

[0020] After the global optimization is completed, local refinement is further performed. For key areas, small controllable displacement adjustments are introduced to improve local fit and the accuracy and naturalness of pose reconstruction. Let the local key point set be ζ, and the local refinement goal is:

[0021]

[0022] where δX k represents the local deformation of the skeleton point, and ε is the maximum allowable offset threshold.

[0023] Optionally, the S4 specifically includes:

[0024] According to the dynamic 3D key point time series, the skeletal motion posture information is estimated, and the estimated skeletal motion posture information is matched to the tester's static skeletal and muscle model, and inverse kinematics analysis is performed to obtain the human body center of mass motion trajectory, joint motion angles, and skeletal motion trajectory, and then the obtained joint motion angles are used to optimize and adjust the 3D key point time series, and then the skeletal motion posture information is estimated again using the optimized and adjusted 3D key point time series, and the re-estimated skeletal motion posture information is matched to the tester's static skeletal and muscle model, and inverse kinematics analysis is performed again to obtain the optimized human body center of mass motion trajectory, joint motion angles, and skeletal motion trajectory;

[0025] The obtained joint motion angle is then used to optimize and adjust the 3D key point time series, and the optimized and adjusted 3D key point time series is then used to estimate the skeletal motion posture information again, and the estimated skeletal motion posture information is matched to the tester's static skeletal and muscle model, and inverse kinematics analysis is performed again to obtain the optimized human center of mass motion trajectory, joint motion angle, and skeletal motion trajectory, including:

[0026] A sliding window is constructed for each continuous frame in a period of time. The multi-frame joint inverse kinematics optimization strategy is used to improve the fitting accuracy and posture trajectory continuity. The t-th frame in the window and several frames before and after it are taken as the optimization unit, and a unified objective function is established for solution:

[0027] For each frame k, let the i-th 3D key point be Its joint angle is q k , the optimal position of this key point is predicted by forward kinematics as The goal is to minimize the weighted residual sum of squares of all frames and all key points, while introducing inter-frame angle smoothness constraints. The joint optimization objective function is:

[0028]

[0029] where q k : joint angle vector of the kth frame; ω i : The weighted coefficient of each key point, reflecting its importance; λ: The regularization coefficient that controls the smoothness of the trajectory; The second term is the time acceleration regularization term, which helps to keep the angle sequence smooth.

[0030] Optionally, the step S5, based on the dynamic balance relationship of human motion, uses the trajectory of the human center of mass and the trajectory of skeletal motion to estimate the ground support forces GRF of the left and right feet, specifically including:

[0031] First, establish the dynamic equilibrium equation at the center of mass:

[0032] m·a CoM(t) = F GRF (t)+m·g

[0033] Where m is the mass, a CoM (t) is the acceleration of the center of mass, g is the acceleration due to gravity, F GRG (t) is the total ground support force to be estimated, and the total ground support force can be directly inferred by this formula:

[0034] F GRF (t) = m·(a CoM (t)-g)

[0035] In order to reasonably distribute the total ground support force to the left and right feet, the contact phase is determined based on the foot contact ball model, and the height of the contact ball center below the ankle is set as z ankle (t), the radius of the sphere is r ball ,like

[0036] δ(t)=r ball -z ankte (t)>0

[0037] The user is then judged to be in contact state. Based on this, the image recognition module is used to determine the current action type, assist in determining the single-leg / double-leg support mode, and distribute the support force according to the contact state and the degree of foot penetration into the ground:

[0038] F GRF =F L (t)+F R (t)

[0039] F L (t) = α(t)·F GRF (t),F R (t)=(1-α(t))·F GRF (t)

[0040] Where α(t) is the distribution weight learned based on contact depth and action type;

[0041] Then, the contact point positions COP of the left and right feet are estimated by combining the ankle position with the predefined plantar offset vector:

[0042] P COP (t) = r ankle (t)+△r offset

[0043] Among them, P COP (t) is the three-dimensional spatial position vector of the ground contact point at time t, r ankle (t) is the ankle position vector at time t, which comes from the reconstructed 3D key points; Δr offseta fixed spatial offset vector to translate the ankle reference point to the actual estimated foot-ground contact area;

[0044] Finally, the estimated GRF and COP are fed back into the dynamic model for inverse dynamics verification. If the calculation error exceeds the preset threshold, the parameter weight is adjusted and re-estimated until the fitting error meets the dynamic consistency requirement, achieving time-sequential estimation of the ground reaction force.

[0045] Optionally, the inverse dynamics method of the rigid body system is adopted in S6 to calculate the required reaction force and torque of each joint of the human body frame by frame, specifically including:

[0046] For each bone, the following dynamic equation set is established:

[0047] Translation: ∑F i = m·a cm

[0048] Rotation: ∑τ i = I·α + ω × (I·ω)

[0049] Wherein, F i and τ i are the external force and torque of each bone, I is the moment of inertia, a cm , ω, α are the center of mass acceleration, angular velocity and angular acceleration, respectively, from inverse kinematics trajectory and its time derivative;

[0050] The solving sequence adopts a step-by-step recursive manner from the ground upwards:

[0051] First, the ground support force and COP position are used to calculate the reaction force and torque of the ankle joint, including:

[0052] The reaction force of the ankle joint is determined by the center of mass acceleration of the lower limb mass end and the GRF: F ankle = m leg ·a leg + m foot ·a foot - F GRF , wherein m leg : calf mass; a leg : calf center of mass acceleration; m foot : foot mass; a foot : foot center of mass acceleration a foot ; F GRF : ground reaction force;

[0053] The ankle torque is the torque generated by the distance between the GRF action point COP and the ankle joint position: τ ankle = r COP × FGRF , where r COP : vector from COP to ankle joint, r COP =p COP -p ankle , p COP : vector position of COP; p ankle The vector position of the ankle joint;

[0054] Then, the knee joint, hip joint, and torso parts are recursively calculated in sequence, and the joint torque of each joint at each moment is calculated frame by frame to complete the inverse dynamics solution of the entire motion sequence.

[0055] Optionally, the step S6 of further calculating muscle strength and activation degree specifically includes:

[0056] Introduce nonlinear muscle coupling method for calculation:

[0057] a=f(W·c)

[0058] in A muscle activation vector of length n, representing the activation level of each muscle; Muscle synergy weight matrix, each column is a synergy pattern, and k represents the number of synergies; Co-activation vector; f(·): nonlinear activation function;

[0059] The muscle force model is:

[0060] F i =a i ·F i，max ·f l (l i )·f v (υ i )

[0061] where a i =f([W·c] i ): activation of the i-th muscle; F i，max is the maximum isometric tension of the i-th muscle; f l (l i ): length-force relationship; f v (v i ): velocity-force relationship;

[0062] Torque equilibrium constraint equation:

[0063]

[0064] Among them, r i，j (q): The moment arm of the i-th muscle at the j-th joint, determined by the joint angle q; F i (a i , li , v i ): The pulling force generated by the i-th muscle depends on the activation value a i , muscle length l i , contraction speed v i ; The net moments of the joints solved from inverse dynamics;

[0065] The optimization objective function is:

[0066]

[0067] The first term minimizes the total activation amount; the second term is the regularization term, which controls the size of the co-activation value to avoid overfitting, and λ is the regularization parameter, which is used to control the trade-off between activation energy and sparsity.

[0068] In another aspect, a markerless motion acquisition and analysis system is provided, the system comprising:

[0069] A recording module is used to use two or more cameras to record the tester's static standing and different movements, respectively, to obtain static image data and dynamic image data of the tester;

[0070] an identification and reconstruction module for identifying and denoising static 2D key points based on the static image data of the test subject, identifying and denoising a time series of dynamic 2D key points based on the dynamic image data of the test subject, and reconstructing the denoised static 2D key points and dynamic 2D key point time series into static 3D key points and dynamic 3D key point time series, respectively;

[0071] A first calculation module is configured to calculate the bone length based on the static 3D key points, perform optimization adjustments, scale the standard bone model according to the optimization adjustment results, and simultaneously adjust various parameters of the standard muscle model accordingly to obtain the static bone and muscle model of the test subject;

[0072] The first estimation module is used to estimate the skeletal motion posture information based on the original and optimized dynamic 3D key point time series, and match the estimated skeletal motion posture information to the static skeletal and muscle model of the tester to perform inverse kinematic analysis to obtain the original and optimized human center of mass motion trajectory, joint motion angle and skeletal motion trajectory;

[0073] A second estimation module is configured to estimate the ground support forces GRF of the left and right feet based on the dynamic balance relationship of human motion and using the optimized human center of mass trajectory and skeletal motion trajectory;

[0074] The second calculation module is used to calculate the reaction force and torque required for each joint of the human body frame by frame based on the optimized human center of mass motion trajectory, joint motion angle, bone motion trajectory and ground support force GRF estimation values ​​of the left and right feet, using the inverse dynamics method of the rigid body system, and then calculate the muscle strength and activation degree.

[0075] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned markerless motion collection and analysis method.

[0076] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned markerless motion acquisition and analysis method.

[0077] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0078] 1) The present invention provides a markerless motion acquisition and analysis method and system, which realizes an integrated closed-loop system from motion acquisition, posture estimation to mechanical analysis and result output, thereby improving the overall operational portability.

[0079] 2) The present invention uses two or more camera recordings combined with a deep learning algorithm to achieve analysis of the inverse kinematics and inverse dynamics of the human body, estimation of ground support force, calculation of the reaction force and torque required for each joint of the human body, and calculation of muscle strength and activation level without the need for a force platform and markers. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0081] Figure 1 This is a flow chart of a markerless motion collection and analysis method provided by an embodiment of the present invention;

[0082] Figure 2 This is an overall block diagram of a markerless motion collection and analysis method provided by an embodiment of the present invention;

[0083] Figure 3 This is a diagram of the arrangement of collection equipment for a markerless motion collection and analysis method provided by an embodiment of the present invention;

[0084] Figure 4This is a flow chart of the collection process of a markerless action collection and analysis method provided by an embodiment of the present invention;

[0085] Figure 5 This is a folder structure diagram of structured storage of a markerless action collection and analysis method provided by an embodiment of the present invention;

[0086] Figure 6 This is a visualization interface diagram of the inverse kinematics analysis results of a markerless motion acquisition and analysis method provided by an embodiment of the present invention;

[0087] Figure 7 is a graph of ground support force of a markerless motion acquisition and analysis method provided by an embodiment of the present invention;

[0088] Figure 8 This is a block diagram of a markerless motion collection and analysis system provided by an embodiment of the present invention;

[0089] Figure 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0091] An embodiment of the present invention provides a markerless motion collection and analysis method, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of the method is shown, and the processing flow may include the following steps:

[0092] S1. Use two or more cameras to record the tester's static standing and different movements, respectively, to obtain static image data and dynamic image data of the tester;

[0093] like Figure 3 and Figure 4 As shown, in the embodiment of the present invention, the cameras used for recording are arranged at an angle of ±35° (2 cameras) in front of the recording area, at a distance of about 2m; if there are three cameras, one more camera is added directly in front of the recording area, and more cameras are arranged symmetrically according to the situation to ensure that the captured effect can be seen in the video preview area, and all cameras must include the tester.

[0094] First, open the software and fill in the basic information of the tester, including name, age, gender, height, weight and other basic information, enter it into the software, and save it accordingly.

[0095] Select the number of cameras to be recorded and the cameras to be recorded, and make sure that the tester can be seen in the software preview area.

[0096] Place the checkerboard in the center of the recording area and begin recording. Remove the checkerboard when finished. The checkerboard serves to calibrate the extrinsic parameters of the recording camera. The two cameras can calculate the rotation matrix, translation vector, and Euler angle parameters by identifying the checkerboard and its parameters. The checkerboard is then used to establish a spatial coordinate system, completing the establishment of the recognition space.

[0097] The tester stands in a triangle position (legs shoulder-width apart, standing upright, hands open, hands forming a triangle with the ground), and the static recording of the tester is performed. After completion, prepare for dynamic recording.

[0098] According to the recording steps, record dynamic postures including walking, sitting up (sitting on a chair, then standing up), squatting, etc. Walking should include a complete gait, sitting up should be repeated three times (i.e. repeat the following process: chair sitting-up-sitting-up-sitting-up), and squatting should be repeated three times (i.e. standing-squatting-standing-squatting-standing-squatting).

[0099] The recorded files are saved in a structured directory. The structure saved by different testers is consistent, which is convenient for subsequent data analysis and result storage. Figure 5 shown.

[0100] S2. Based on the static image data of the test subject, identify static 2D key points and reduce noise; based on the dynamic image data of the test subject, identify a time series of dynamic 2D key points and reduce noise; and reconstruct the static 2D key points and the time series of dynamic 2D key points after noise reduction into static 3D key points and a time series of dynamic 3D key points, respectively;

[0101] The embodiment of the present invention can use the HRNet deep learning algorithm to identify static 2D key points and dynamic 2D key point time series, and identify 18 to 33 key points (including shoulders, knees, hip joints, elbow joints, etc.);

[0102] The 2D keypoints output by deep learning networks are often limited by interference factors such as the frame rate of the acquisition device, lighting obstructions, and rapid object motion, resulting in noisy, jittery, or missing keypoint trajectories. Therefore, embodiments of the present invention employ a two-stage noise reduction strategy: the first stage uses a confidence screening mechanism to eliminate unreliable keypoint predictions; the second stage introduces a nonlinear temporal filtering module.

[0103] The denoised static 2D key points and dynamic 2D key point time series are reconstructed into static 3D key points and dynamic 3D key point time series respectively. The 2D key points of multiple cameras can be combined to use triangulation, camera calibration, and weighted least squares method to reconstruct 3D key points. The specific reconstruction process is existing technology and will not be repeated here.

[0104] S3, based on the static 3D key points, calculate the bone length, and make optimization adjustment, according to the optimization adjustment result, scale the standard bone model, and make corresponding adjustment to each parameter of the standard muscle model, to obtain the tester static bone and muscle model;

[0105] Optionally, in the S3, based on the static 3D key points, the bone length is calculated, and the optimization adjustment is made, specifically including:

[0106] Let the three-dimensional reconstruction coordinates of the i, j key points be X i , Y j , and the bone length formed by the i, j key points is:

[0107]

[0108] Compare with the default bone length in the standard bone model , calculate the preliminary scaling factor, and make preliminary adjustment to the standard bone model:

[0109]

[0110] Where ψ is the selected key bone set for estimation, and M is the number of bones;

[0111] After preliminary adjustment, the weighted least squares method is used to globally optimize each bone to further improve the fitting accuracy, and the optimization target is to minimize the weighted bone residual, and the physiological proportion constraint is introduced to ensure that the scaling proportion conforms to the individual difference and does not deviate from the physiological conventional range, and the formula is as follows:

[0112]

[0113] Where ω ij is a confidence weighting factor, and the constraint interval is determined by the human body statistical model;

[0114] After global optimization, further local refinement is performed, and for key areas (such as pelvis, hands, etc.), a small controllable displacement adjustment is introduced to improve the local fitting degree and improve the accuracy and naturalness of the posture reconstruction, and the local key point set is ζ, and the local refinement target is:

[0115]

[0116] Where δX k represents the local deformation of the skeleton point, and ε is the maximum allowed offset threshold.

[0117] S4. Estimating skeletal motion posture information based on the original and optimized dynamic 3D key point time series, and matching the estimated skeletal motion posture information to the test subject's static skeletal and muscle model to perform inverse kinematic analysis to obtain the original and optimized human center of mass motion trajectories, joint motion angles, and skeletal motion trajectories;

[0118] Optionally, the S4 specifically includes:

[0119] According to the dynamic 3D key point time series, the skeletal motion posture information is estimated, and the estimated skeletal motion posture information is matched to the tester's static skeletal and muscle model, and inverse kinematics analysis is performed to obtain the human body center of mass motion trajectory, joint motion angle and skeletal motion trajectory, and then the obtained joint motion angle is used to optimize and adjust the 3D key point time series, and then the optimized and adjusted 3D key point time series is used to estimate the skeletal motion posture information again, and the estimated skeletal motion posture information is matched to the tester's static skeletal and muscle model, and inverse kinematics analysis is performed again to obtain the optimized human body center of mass motion trajectory, joint motion angle and skeletal motion trajectory, as shown in FIG. Figure 6 , which is a visualization interface diagram of the inverse kinematics results of a markerless motion acquisition and analysis method provided by an embodiment of the present invention;

[0120] The obtained joint motion angle is then used to optimize and adjust the 3D key point time series, and the optimized and adjusted 3D key point time series is then used to estimate the skeletal motion posture information again, and the estimated skeletal motion posture information is matched to the tester's static skeletal and muscle model, and inverse kinematics analysis is performed again to obtain the optimized human center of mass motion trajectory, joint motion angle, and skeletal motion trajectory, including:

[0121] A sliding window is constructed for each continuous frame in a period of time. The multi-frame joint inverse kinematics optimization strategy is used to improve the fitting accuracy and posture trajectory continuity. The t-th frame in the window and several frames before and after it are taken as the optimization unit, and a unified objective function is established for solution:

[0122] For each frame k, let the i-th 3D key point be Its joint angle is q k , the optimal position of this key point is predicted by forward kinematics as The goal is to minimize the weighted residual sum of squares of all frames and all key points, while introducing inter-frame angle smoothness constraints. The joint optimization objective function is:

[0123]

[0124] where q k : joint angle vector of the kth frame; ω i: The weighted coefficient of each key point, reflecting its importance; λ: The regularization coefficient that controls the smoothness of the trajectory; The second term is the time acceleration regularization term, which helps to keep the angle sequence smooth.

[0125] In this embodiment of the present invention, the joint optimization problem is solved window by window using the trust region method, and the optimal solution for each frame is output based on the sliding window center. Compared with solving each frame independently, this method can more effectively suppress jitter and repair occlusion frame errors, significantly improving the continuity and robustness of the inverse kinematics solution.

[0126] S5. Based on the dynamic balance relationship of human motion, using the optimized human center of mass trajectory and skeletal motion trajectory, estimate the ground support force GRF of the left and right feet;

[0127] Optionally, the step S5, based on the dynamic balance relationship of human motion, uses the trajectory of the human center of mass and the trajectory of skeletal motion to estimate the ground support forces GRF of the left and right feet, specifically including:

[0128] First, establish the dynamic equilibrium equation at the center of mass:

[0129] m·a CoM (t) = F GRF (t)+m·g

[0130] Where m is the mass, a CoM (t) is the acceleration of the center of mass, g is the acceleration due to gravity, F GRF (t) is the total ground support force to be estimated, and the total ground support force can be directly inferred by this formula:

[0131] F GRF (t) = m·(a CoM (t)-g)

[0132] In order to distribute the total ground support force to the left and right feet reasonably, the contact phase is determined based on the foot contact ball model (the foot contact ball model is a simplified biomechanical modeling method; the contact ball simplifies the key areas such as the foot into a sphere, making it easier to determine whether they are in contact with the ground). The height of the contact ball center below the ankle is set as z ankle (t), the radius of the sphere is r ball ,like

[0133] δ(t)=r ball -z ankle (t)>0

[0134] The system then determines the state as contact. Based on this, it combines the image recognition module to determine the current action type (such as walking, running, jumping), assists in determining the single-leg / double-leg support mode, and distributes the support force according to the contact state and the degree of foot penetration into the ground:

[0135] FGRF =F L (t)+F R (t)

[0136] F L (t) = α(t)·F GRF (t),F R (t)=(1-α(t))·F GRF (t)

[0137] Where α(t) is the distribution weight learned based on contact depth and action type;

[0138] Then, the contact point positions COP of the left and right feet are estimated by combining the ankle position with the predefined plantar offset vector:

[0139] P COP (t) = r ankle ((t)+△r offset

[0140] Among them, P COP (t) is the three-dimensional spatial position vector of the ground contact point at time t, r ankle (t) is the ankle position vector at time t, which comes from the reconstructed 3D key points; Δr offset : A fixed spatial offset vector used to translate the ankle reference point (e.g., lateral malleolus) to the actual estimated foot contact area;

[0141] Finally, the estimated GRF and COP are fed back into the dynamics model for inverse dynamics testing. If the calculation error exceeds the preset threshold, the parameter weights (such as contact ball size, action recognition confidence, smoothing filter strength, etc.) are adjusted and re-estimated until the fitting error meets the dynamic consistency requirements, thus achieving a time-series continuous estimation of the ground reaction force.

[0142] S6. Based on the optimized human center of mass motion trajectory, joint motion angles, skeletal motion trajectory, and estimated ground support force GRF values ​​of the left and right feet, the inverse dynamics method of the rigid body system is used to calculate the reaction force and torque required for each joint of the human body frame by frame, and then calculate the muscle strength and activation degree.

[0143] Optionally, the inverse dynamics method of the rigid body system is used in S6 to calculate the reaction force and torque required for each joint of the human body frame by frame, specifically including:

[0144] For each bone, the following dynamic equations are established:

[0145] Translation: ∑F i =m·a cm

[0146] Rotation: ∑τi =I·α+ω×(I·ω)

[0147] Among them, F i and τ i are the external forces and moments acting on each bone, I is the moment of inertia, a cm , ω, and α are the center-of-mass acceleration, angular velocity, and angular acceleration, respectively, which come from the inverse kinematics trajectory and its time derivative;

[0148] The solution sequence is a recursive method from the ground up:

[0149] First, the reaction force and torque of the ankle joint are calculated using the ground support force and COP position, including:

[0150] The reaction force of the ankle joint is determined by the center of mass acceleration of the lower limb mass end and GRF: F ankle =m leg ·a leg +m foot ·a foot -F GRF , where m leg : calf mass; a leg : acceleration of the center of mass of the calf; m foot : foot mass; a foot : acceleration of the foot center of mass a foot ; F GRF : ground reaction force;

[0151] The ankle joint torque is the torque generated by the distance between the GRF action point COP and the ankle joint position: τ ankle =r COP ×F GRF , where r COP : vector from COP to ankle joint, r COP =p COP -p ankle , p COP : vector position of COP; p ankle The vector position of the ankle joint;

[0152] Then, the knee joint, hip joint, and torso parts are recursively calculated in sequence, and the joint torque of each joint at each moment is calculated frame by frame to complete the inverse dynamics solution of the entire motion sequence.

[0153] Optionally, the step S6 of further calculating muscle strength and activation degree specifically includes:

[0154] Introduce nonlinear muscle coupling method for calculation:

[0155] a=f(W·c)

[0156] in Muscle activation vector of length n, representing the activation level of each muscle; Muscle synergy weight matrix, each column represents a synergy mode, k represents the number of synergies; Synergy activation vector; f(·): nonlinear activation function (mainly including ReLU, Sigmoid, Tanh, etc.);

[0157] Muscle force model is:

[0158] F i = a i · F i，max · f l (l i )· f v (v i )

[0159] Where a i = f([W· c] i ): the activation of the ith muscle; F i，max is the maximum isometric tension of the ith block of muscle; f l (l i ): length-force relationship; f v (v i ): velocity-force relationship;

[0160] Moment balance constraint equation:

[0161]

[0162] Where, r i，j (q): the force arm of the ith block of muscle at the jth joint, determined by the joint angle q; F i (a i , l i , v i ): the tension generated by the ith block of muscle, depending on the activation value a i , muscle length l i , contraction velocity v i ; Joint net moment from inverse dynamics solution;

[0163] The optimization objective function is:

[0164]

[0165] Where the first term is the minimization of total activation (muscle metabolic load); the second term is the regularization term, which controls the size of synergy activation value to avoid overfitting, and λ is the regularization parameter, which is used to control the trade-off between activation energy and sparsity.

[0166] The embodiment of the present invention can draw different joint movements, muscle forces, activations, ground support forces (such as Figure 7 As shown in the figure), knee joint contact torque, etc.

[0167] like Figure 8 As shown, an embodiment of the present invention further provides a markerless motion collection and analysis system, the system comprising:

[0168] The recording module 810 is used to use two or more cameras to record the tester's static standing and different movements, respectively, to obtain static image data and dynamic image data of the tester;

[0169] an identification and reconstruction module 820 for identifying and denoising static 2D key points based on the static image data of the subject, identifying and denoising a time series of dynamic 2D key points based on the dynamic image data of the subject, and reconstructing the denoised static 2D key points and dynamic 2D key point time series into static 3D key points and dynamic 3D key point time series, respectively;

[0170] A first calculation module 830 is configured to calculate the bone length based on the static 3D key points, perform optimization adjustments, scale the standard bone model according to the optimization adjustment results, and simultaneously adjust various parameters of the standard muscle model accordingly to obtain the static bone and muscle model of the test subject;

[0171] A first estimation module 840 is configured to estimate skeletal motion posture information based on the original and optimized dynamic 3D key point time series, match the estimated skeletal motion posture information to the test subject's static skeletal and muscle model, perform inverse kinematics analysis, and obtain the original and optimized human center of mass motion trajectories, joint motion angles, and skeletal motion trajectories;

[0172] A second estimation module 850 is configured to estimate the ground support forces GRF of the left and right feet based on the dynamic balance relationship of human motion and using the optimized human center of mass trajectory and skeletal motion trajectory;

[0173] The second calculation module 860 is used to calculate the reaction force and torque required for each joint of the human body frame by frame based on the optimized human center of mass motion trajectory, joint motion angle, bone motion trajectory and ground support force GRF estimation values ​​of the left and right feet, using the inverse dynamics method of the rigid body system, and then calculate the muscle strength and activation degree.

[0174] The functional structure of a markerless motion acquisition and analysis system provided by an embodiment of the present invention corresponds to that of a markerless motion acquisition and analysis method provided by an embodiment of the present invention, and will not be described in detail here.

[0175] Figure 9It is a structural diagram of an electronic device 900 provided in an embodiment of the present invention. The electronic device 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 901 and one or more memories 902, wherein at least one instruction is stored in the memory 902, and the at least one instruction is loaded and executed by the processor 901 to implement the steps of the above-mentioned markerless motion acquisition and analysis method.

[0176] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions. The instructions are executable by a processor in a terminal to implement the above-described markerless motion collection and analysis method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0177] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A markerless motion collection and analysis method, characterized in that: The method comprises: S1. Use two or more cameras to record the tester's static standing and different movements, respectively, to obtain static image data and dynamic image data of the tester; S2. Based on the static image data of the test subject, identify static 2D key points and reduce noise; based on the dynamic image data of the test subject, identify a time series of dynamic 2D key points and reduce noise; and reconstruct the static 2D key points and the time series of dynamic 2D key points after noise reduction into static 3D key points and a time series of dynamic 3D key points, respectively; S3. Calculate the bone length based on the static 3D key points and perform optimization adjustments. Scaling the standard bone model according to the optimization adjustment results and adjusting the parameters of the standard muscle model accordingly to obtain the static bone and muscle model of the test subject. S4. Estimating skeletal motion posture information based on the original and optimized dynamic 3D key point time series, and matching the estimated skeletal motion posture information to the test subject's static skeletal and muscle model to perform inverse kinematic analysis to obtain the original and optimized human center of mass motion trajectories, joint motion angles, and skeletal motion trajectories; S5. Based on the dynamic balance relationship of human motion, using the optimized human center of mass trajectory and skeletal motion trajectory, estimate the ground support force GRF of the left and right feet; S6. Based on the optimized human center of mass motion trajectory, joint motion angles, skeletal motion trajectory, and estimated ground support force GRF values ​​of the left and right feet, the inverse dynamics method of the rigid body system is used to calculate the reaction force and torque required for each joint of the human body frame by frame, and then calculate the muscle strength and activation degree.

2. The method according to claim 1, characterized in that In S3, the bone length is calculated based on the static 3D key points and optimized and adjusted, specifically including: Let the 3D reconstruction coordinates of the i, j key points be X i , Y j , then the length of the skeleton composed of the i-th and j-th key points is: Compared to the default bone length in the standard bone model For comparison, calculate the initial scaling factor to make preliminary adjustments to the standard skeleton model: Where ψ is the set of key bones selected for estimation, and M is the number of bones; After the initial adjustment, the weighted least squares method is used to perform global optimization on each skeleton to further improve the fitting accuracy. The optimization goal is to minimize the weighted skeleton residuals, and the physiological proportion constraint is introduced to ensure that the scaling ratio is consistent with individual differences and does not deviate from the physiological normal range. The formula is as follows: where ω ij is the confidence weighting factor, and the constraint interval is determined by the human statistical model; After the global optimization is completed, local refinement is further performed. For key areas, small controllable displacement adjustments are introduced to improve local fit and the accuracy and naturalness of pose reconstruction. Let the local key point set be ζ, and the local refinement goal is: where δX k represents the local deformation of the skeleton point, and ε is the maximum allowable offset threshold.

3. The method according to claim 1, characterized in that Said S4 specifically includes: According to the dynamic 3D key point time series, the skeletal motion posture information is estimated, and the estimated skeletal motion posture information is matched to the tester's static skeletal and muscle model, and inverse kinematics analysis is performed to obtain the human body center of mass motion trajectory, joint motion angles, and skeletal motion trajectory, and then the obtained joint motion angles are used to optimize and adjust the 3D key point time series, and then the skeletal motion posture information is estimated again using the optimized and adjusted 3D key point time series, and the re-estimated skeletal motion posture information is matched to the tester's static skeletal and muscle model, and inverse kinematics analysis is performed again to obtain the optimized human body center of mass motion trajectory, joint motion angles, and skeletal motion trajectory; The obtained joint motion angle is then used to optimize and adjust the 3D key point time series, and the optimized and adjusted 3D key point time series is then used to estimate the skeletal motion posture information again, and the estimated skeletal motion posture information is matched to the tester's static skeletal and muscle model, and inverse kinematics analysis is performed again to obtain the optimized human center of mass motion trajectory, joint motion angle, and skeletal motion trajectory, including: A sliding window is constructed for each continuous frame in a period of time. The multi-frame joint inverse kinematics optimization strategy is used to improve the fitting accuracy and posture trajectory continuity. The t-th frame in the window and several frames before and after it are taken as the optimization unit, and a unified objective function is established for solution: For each frame k, let the i-th 3D key point be Its joint angle is q k , the optimal position of this key point is predicted by forward kinematics as The goal is to minimize the weighted residual sum of squares of all frames and all key points, while introducing inter-frame angle smoothness constraints. The joint optimization objective function is: where q k : joint angle vector of the kth frame; ω i : The weighted coefficient of each key point, reflecting its importance; λ: The regularization coefficient that controls the smoothness of the trajectory; The second term is the time acceleration regularization term, which helps to keep the angle sequence smooth.

4. The method according to claim 1, wherein The S5 is based on the dynamic balance relationship of human motion, and uses the trajectory of the human body's center of mass and the trajectory of skeletal motion to estimate the ground support force GRF of the left and right feet, specifically including: First, establish the dynamic equilibrium equation at the center of mass: m·a CoM (t)=F GRF (t)+m·g Where m is the mass, a CoM (t) is the acceleration of the center of mass, g is the acceleration due to gravity, F GRF (t) is the total ground support force to be estimated, and the total ground support force can be directly inferred by this formula: F GRF (t)=m·(a CoM (t)-g) In order to reasonably distribute the total ground support force to the left and right feet, the contact phase is determined based on the foot contact ball model, and the height of the contact ball center below the ankle is set as z ankle (t), the radius of the sphere is r ball ,like δ(t)=r ball -z ankle (t)>0 The user is then judged to be in contact state. Based on this, the image recognition module is used to determine the current action type, assist in determining the single-leg / double-leg support mode, and distribute the support force according to the contact state and the degree of foot penetration into the ground: F GRF =F L (t)+F R (t) F L (t)=α(t)·F GRF (t),F R (t)=(1-α(t))·F GRF (t) Where α(t) is the distribution weight learned based on contact depth and action type; Then, the contact point positions COP of the left and right feet are estimated by combining the ankle position with the predefined plantar offset vector: P COP (t)=r ankle (t)+△r offaet Among them, P COP (t) is the three-dimensional spatial position vector of the ground contact point at time t, r ankle (t) is the ankle position vector at time t, which comes from the reconstructed 3D key points; Δr offset : A fixed spatial offset vector used to translate the ankle reference point to the actual estimated foot contact area; Finally, the estimated GRF and COP are input back into the dynamic model for inverse dynamics testing. If the calculation error exceeds the preset threshold, the parameter weights are adjusted and re-estimated until the fitting error meets the dynamic consistency requirements, thus achieving a time-series continuous estimation of the ground reaction force.

5. The method according to claim 4, characterized in that The S6 uses the inverse dynamics method of the rigid body system to calculate the reaction force and torque required by each joint of the human body frame by frame, specifically including: For each bone, the following dynamic equations are established: Translation: ∑F i =m·a cm Movement: ∑τ i = I·α+ω×(I·ω) Among them, F i and τ i are the external forces and moments acting on each bone, I is the moment of inertia, a cm , ω, and α are the center-of-mass acceleration, angular velocity, and angular acceleration, respectively, which come from the inverse kinematics trajectory and its time derivative; The solution sequence is a recursive method from the ground up: First, the reaction force and torque of the ankle joint are calculated using the ground support force and COP position, including: The reaction force of the ankle joint is determined by the center of mass acceleration of the lower limb mass end and GRF: F ankle =m leg ·a leg +m foot ·a foot -F GRF , where m leg : calf mass; a leg : acceleration of the center of mass of the calf; m foot : foot mass; a foot : acceleration of the foot center of mass a foot ; F GRF : ground reaction force; The ankle joint torque is the torque generated by the distance between the GRF action point COP and the ankle joint position: τ ankle =r COP ×F GRF , where r COP : vector from COP to ankle joint, r COP =p COP -p ankle , p COP : vector position of COP; p ankle The vector position of the ankle joint; Then, the knee joint, hip joint, and torso parts are recursively calculated in sequence, and the joint torque of each joint at each moment is calculated frame by frame to complete the inverse dynamics solution of the entire motion sequence.

6. The method according to claim 5, characterized in that The calculation of muscle strength and activation level in S6 specifically includes: Introduce nonlinear muscle coupling method for calculation: in A muscle activation vector of length n, representing the activation level of each muscle; Muscle synergy weight matrix, each column is a synergy pattern, k represents the number of synergies; Co-activation vector; f(·): nonlinear activation function; The muscle force model is: F i =a i ·F i，max ·f l (l i )·f v (v i ) where a i =f([W·c] i ): activation of the i-th muscle; F i，max is the maximum isometric tension of the i-th muscle; f l (l i ): length-force relationship; f v (v i ): velocity-force relationship; Torque equilibrium constraint equation: Among them, r i，j (q): The moment arm of the i-th muscle at the j-th joint, determined by the joint angle q; F i (a i , l i , v i ): The pulling force generated by the i-th muscle depends on the activation value a i , muscle length l i , contraction speed v i ;; The net moments of the joints solved from inverse dynamics; The optimization objective function is: The first term minimizes the total activation amount; the second term is the regularization term, which controls the size of the co-activation value to avoid overfitting, and λ is the regularization parameter, which is used to control the trade-off between activation energy and sparsity.

7. A markerless motion acquisition and analysis system, characterized in that: The system comprises: A recording module is used to use two or more cameras to record the tester's static standing and different movements, respectively, to obtain static image data and dynamic image data of the tester; an identification and reconstruction module for identifying and denoising static 2D key points based on the static image data of the test subject, identifying and denoising a time series of dynamic 2D key points based on the dynamic image data of the test subject, and reconstructing the denoised static 2D key points and dynamic 2D key point time series into static 3D key points and dynamic 3D key point time series, respectively; A first calculation module is configured to calculate the bone length based on the static 3D key points, perform optimization adjustments, scale the standard bone model according to the optimization adjustment results, and simultaneously adjust various parameters of the standard muscle model accordingly to obtain the static bone and muscle model of the test subject; The first estimation module is used to estimate the skeletal motion posture information based on the original and optimized dynamic 3D key point time series, and match the estimated skeletal motion posture information to the static skeletal and muscle model of the tester to perform inverse kinematic analysis to obtain the original and optimized human center of mass motion trajectory, joint motion angle and skeletal motion trajectory; A second estimation module is configured to estimate the ground support forces GRF of the left and right feet based on the dynamic balance relationship of human motion and using the optimized human center of mass trajectory and skeletal motion trajectory; The second calculation module is used to calculate the reaction force and torque required for each joint of the human body frame by frame based on the optimized human center of mass motion trajectory, joint motion angle, bone motion trajectory and ground support force GRF estimation values ​​of the left and right feet, using the inverse dynamics method of the rigid body system, and then calculate the muscle strength and activation degree.

8. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, characterized in that: The at least one instruction is loaded and executed by the processor to implement the markerless motion collection and analysis method according to any one of claims 1 to 6.

9. A computer-readable storage medium, wherein at least one instruction is stored in the storage medium, characterized in that: The at least one instruction is loaded and executed by the processor to implement the markerless motion collection and analysis method according to any one of claims 1 to 6.

Images