A pose estimation method based on adaptive smoothing and physical closed loop

Abstract

The application discloses a pose estimation method based on adaptive smoothing and physical closed loop, comprising: acquiring a video sequence, sequentially traversing each time of the video sequence in time sequence, for each time, estimating the physical state of the human body in each time video sequence, the pose estimation method based on adaptive smoothing and physical closed loop does not need to depend on multi-camera or large-scale motion capture annotation, through the joint design of monocular video sequence, motion estimation network, intermediate fusion pose, pose speed filtering, expected control target modulation, closed loop physical control and dynamics integration, high-precision, low-delay and physically consistent pose tracking is realized; the method can significantly improve the performance of the pose tracking result in terms of time sequence continuity, local detail stability, contact rationality and overall physical consistency, thereby constructing a closed loop pose tracking system strictly complying with the physical self-recurrence law.

Description

Technical Field

[0001] This invention belongs to the field of attitude estimation technology, specifically relating to an attitude estimation method based on adaptive smoothing and physical loop closure. Background Technology

[0002] As core foundational technologies of computer vision and artificial intelligence, 3D human pose estimation and motion capture directly determine the effectiveness of downstream applications through the quality of the motion data they generate. In cutting-edge fields such as virtual reality (VR), augmented reality (AR), embodied intelligent robots, film and animation production, and metaverse digital human-driven systems, recovering smooth, natural, and physically consistent 3D motion sequences from monocular videos or real-time visual streams has become a key requirement for enhancing system immersion and interactive realism.

[0003] In recent years, deep learning-based pure vision kinematic pose estimation methods (such as various monocular 3D pose estimation algorithms) have been widely used. These methods, relying on powerful spatial feature extraction capabilities, can estimate skeletal keypoints and joint rotation parameters of the human body in complex image sequences. However, in practical dynamic scenes and physical interaction applications, existing pure vision estimation techniques still face many significant limitations:

[0004] First, due to limitations such as monocular depth blur, self-occlusion, and lack of real physical constraints, the original visual kinematic estimation results (i.e., the original pose) often contain a large number of high-frequency spurious variations and temporal discontinuities. These non-physical jitters and abrupt changes manifest as obvious artifacts visually.

[0005] Secondly, when driving physical entities such as digital humans or robots, directly using the raw visual estimation results containing high-frequency noise as the control target of the physics engine or actuator will cause the underlying physics controller to produce a violent and unexpected response. This direct input of high-frequency errors will trigger severe control oscillations, and further lead to severe foot sliding and contact instability when the model interacts with the ground or environment, resulting in dynamic irrationality such as the final generated motion completely violating the physical laws of the real world.

[0006] Finally, to mitigate the aforementioned jitter problem, existing technologies typically employ empirical, traditional low-pass filters for post-processing kinematic data. However, in actual high-dynamic human motion, traditional filters inherently involve a static trade-off between noise suppression and high-frequency response—strong filtering leads to severe response delays during rapid movements, while weak filtering fails to effectively suppress control jitter at low speeds. Furthermore, existing filtering schemes are mostly open-loop structures, lacking real-world feedback from the physics engine, and thus cannot fundamentally guarantee the physical consistency of posture.

[0007] In summary, existing technologies struggle to simultaneously achieve high-frequency noise suppression, zero-latency agile tracking, and rigorous physical plausibility when processing visual pose estimation results. Particularly in highly dynamic and complex contact scenarios, a system solution is still lacking that can intelligently fuse intentional pose and observation data, adaptively smooth motion trajectories, and incorporate physical simulation for closed-loop feedback. Summary of the Invention

[0008] The purpose of this invention is to address the problems raised in the background art by proposing an attitude estimation method based on adaptive smoothing and physical loop closure.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] This invention proposes a pose estimation method based on adaptive smoothing and physical loop closure, comprising:

[0011] Obtain the video sequence, iterate through each moment of the video sequence in chronological order, and for each moment, estimate the physical state of the human body in the video sequence. The estimation process for the physical state of the human body at each moment is as follows:

[0012] Step 1: Obtain the original attitude at the current moment and obtain the historical physical state at the previous moment. Calculate the target intention attitude using the historical physical state, where each physical state includes attitude and attitude velocity.

[0013] Step 2: Use the original pose and historical physical state at the current moment as input to the trained motion estimation network to obtain the confidence gating matrix, Kalman gain matrix, proportional gain matrix and differential gain matrix of the controller at the current moment, as well as the contact probability, contact Jacobian matrix and contact constraint force.

[0014] Step 3: Use the confidence gating matrix to perform residual modulation fusion on the original pose and the target intention pose at the current time to obtain the intermediate fused pose at the current time.

[0015] Step 4: Calculate the filtered attitude velocity at the current moment using the Kalman gain matrix and the attitude velocities in the historical physical states;

[0016] Step 5: Calculate the effective smoothing step size at the current moment based on the filtered attitude velocity at the current moment, the integral duration between the current moment and the previous moment, and the dynamic cutoff frequency;

[0017] Step 6: Based on the effective smoothing step size, perform cross-frame temporal filtering on the attitudes in the intermediate fused attitudes and historical physical states to obtain the desired control target at the current moment.

[0018] Step 7: Input the desired control target at the current moment into the physics simulation engine, and use the PD controller to calculate the motion torque of each joint at the current moment;

[0019] Step 8: Combining the motion torques of all joints, use the Euler-Lagrange dynamics equations with contact constraints to solve for the instantaneous acceleration at the current moment;

[0020] Step 9: Based on the instantaneous acceleration at the current moment, perform forward integration using a semi-implicit second-order Euler integral scheme to update the physical state at the current moment.

[0021] Preferably, each posture includes the root node displacement of the human body and the rotational parameters of all joints of the human body;

[0022] All joints in the human body are divided into multiple branches and connected in a preset order. The rotation parameters of each joint in the human body are relative to the rotation parameters of the previous joint in the connection order, and the rotation parameters of the root node are relative to the world coordinate system. The root node is the pelvic root node.

[0023] Postural velocity includes the root node displacement velocity of the human body and the rotational velocity of all joints in the human body.

[0024] Preferably, the step of obtaining the original pose at the current moment, obtaining the historical physical state at the previous moment, and calculating the target intention pose using the historical physical state includes:

[0025] The changed attitude is calculated based on the integral duration between the current moment and the previous moment and the attitude velocity in the historical physical state. The changed attitude is then added to the attitude in the historical physical state to obtain the target intention attitude at the current moment.

[0026] Preferably, the motion estimation network includes a shared hidden feature extraction branch, as well as parallel optimal state prediction branches, proportional gain prediction branches, differential gain prediction branches, contact state prediction branches, external force prediction branches, and contact Jacobian prediction branches.

[0027] In the shared hidden feature extraction branch, the original pose, historical physical state, and the expanded features of the joint space inertia matrix of the historical physical state at the current moment are concatenated and then passed through three linear units in sequence to obtain the shared hidden features. Each linear unit includes a linear layer, a layer normalization layer, and an activation layer connected in sequence. The joint space inertia matrix of the historical physical state is calculated using a rigid body dynamics library.

[0028] In the optimal state prediction branch, the shared hidden features pass through GRU, layer normalization, activation layer and linear layer in sequence to obtain temporal features. At the same time, the shared hidden features pass through the linear layer to obtain state features. The temporal features and state features are concatenated and then passed through the Dropout layer to obtain the first feature. The first feature passes through the linear layer and rearrangement operation in sequence to obtain the Kalman gain matrix. At the same time, the first feature passes through the linear layer and nonlinear activation function in sequence to obtain the confidence gating matrix.

[0029] In the proportional gain prediction branch and the differential gain prediction branch, the shared hidden features pass through a linear layer, an activation layer, a linear layer and a nonlinear activation function in sequence to obtain the diagonal proportional gain matrix and the diagonal differential gain matrix, respectively.

[0030] In the contact state prediction branch, the shared hidden features pass through a linear layer to obtain contact hidden features. The contact hidden features are then concatenated with a preset contact input to obtain contact fusion features. The contact fusion features then pass through a linear layer, an activation layer, a linear layer, layer normalization, an activation layer, a linear layer, and a nonlinear activation function in sequence to obtain the contact probabilities of the left and right feet of the human body.

[0031] In the external force prediction branch, the shared hidden features pass through the linear layer to obtain the external force hidden features. The external force hidden features are then spliced ​​with the preset contact input to obtain the external force fusion features. The external force fusion features then pass through the linear layer, activation layer, linear layer, layer normalization, activation layer, and linear layer in sequence to obtain the contact constraint forces of the left and right feet of the human body.

[0032] In the contact Jacobi prediction branch, the shared hidden features pass through two linear layers in sequence to obtain the Jacobi parameters. The Jacobi parameters are then concatenated with the identity matrix corresponding to the root node displacement to obtain the contact Jacobi matrix.

[0033] Preferably, the preset contact input includes the three-dimensional position of the local joint at the previous moment, the motion amplitude of the local joint at the previous moment relative to the moment before that, the rotation parameters of the local node at the previous moment, the contact probability at the previous moment, and the position of the human body's center of mass at the previous moment, wherein the human body's execution position is calculated using a rigid body dynamics library.

[0034] Preferably, the step of using a confidence gating matrix to perform residual modulation fusion on the original pose and the target intention pose at the current moment to obtain the intermediate fused pose at the current moment includes:

[0035] The difference between the target intention pose at the current moment and the original pose at the current moment is used as a residual term. The residual term is weighted by the confidence gating matrix at the current moment and then superimposed with the original pose at the current moment to obtain the intermediate fused pose at the current moment.

[0036] Preferably, the step of calculating the filtered attitude velocity at the current moment using the Kalman gain matrix and the attitude velocities in the historical physical states includes:

[0037] The attitude velocity in the historical physical state of the previous moment is taken as the observation term, and the filtered attitude velocity of the previous moment is taken as the prior term. The difference between the observation term and the prior term is calculated. The difference is weighted by the Kalman gain matrix of the current moment and then superimposed with the prior term to obtain the filtered attitude velocity of the current moment.

[0038] Preferably, the calculation formula for the effective smoothing step size at the current time is as follows:

[0039] ;

[0040] in,

[0041] ;

[0042] in, The effective smoothing step size at the current moment, The integral duration between the current time and the previous time. For the current moment The dynamic cutoff frequency, The cutoff frequency, This is the speed adjustment coefficient. The current filtered attitude velocity;

[0043] The difference between the intermediate fused attitude and the attitude in the historical physical state is weighted by the effective smoothing step size at the current moment, and then superimposed with the attitude in the historical physical state to obtain the desired control target at the current moment.

[0044] Preferably, the calculation of the driving torque of each joint at the current moment using the PD controller is based on the following formula:

[0045] in, A matrix consisting of the current motion torques of all joints. For the current moment The proportional gain matrix, For the current moment The differential gain matrix, The desired control objective at the current moment, The attitude in the previous moment's historical physical state. The attitude velocity in the previous physical state;

[0046] The instantaneous acceleration at the current moment is solved by combining the motion torques of all joints and using the Euler-Lagrange dynamics equations with contact constraints.

[0047] Preferably, the step of performing forward integration based on the instantaneous acceleration at the current moment using a semi-implicit second-order Euler integral scheme to update the physical state at the current moment includes:

[0048] Phase 1: Based on the instantaneous acceleration at the current moment Update the attitude and velocity at the current moment: ;

[0049] Update the current attitude using the current attitude velocity: ;

[0050] Substitute the current attitude velocity and current attitude into step 7 to recalculate the current motion torque of each joint, and then execute step 8 to recalculate the current instantaneous acceleration. ;

[0051] Second stage: Utilizing the recalculated instantaneous acceleration at the current moment And the attitude at the current moment obtained from the first stage calculation. and attitude velocity The current attitude and attitude velocity are updated and referred to as the final attitude, respectively. and final attitude velocity final posture and final attitude velocity This constitutes the physical state at the current moment.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] This attitude estimation method based on adaptive smoothing and physical closed loop does not rely on multiple cameras or large-scale motion capture annotation. It achieves high-precision, low-latency and physically consistent attitude tracking by combining monocular video sequences with motion estimation networks, intermediate attitude fusion, attitude velocity filtering, desired control target modulation, closed-loop physical control and dynamic integration.

[0054] This method can significantly improve the performance of attitude tracking results in terms of temporal continuity, local detail stability, contact rationality, and overall physical consistency, thereby constructing a closed-loop attitude tracking system that strictly conforms to the autoregressive law of physics. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the estimation process of the human body's physical state at each moment in the attitude estimation method based on adaptive smoothing and physical closure of the present invention.

[0056] Figure 2 This is a schematic diagram of the video sequence obtained by the present invention;

[0057] Figure 3 This is a schematic diagram of the human physical state obtained by estimating the video sequence according to the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0060] In one embodiment, such as Figures 1-3 As shown, a pose estimation method based on adaptive smoothing and physical loop closure is provided, including:

[0061] Obtain a (monocular) video sequence (including video frames from the current moment and consecutive historical moments). Iterate through each moment of the video sequence in chronological order. For each moment, estimate the physical state of the human body in the video sequence (accurately). The estimation process for the physical state of the human body at each moment is as follows:

[0062] Step 1: Obtain the original attitude at the current moment and obtain the historical physical state at the previous moment. Calculate the target intention attitude using the historical physical state, where each physical state includes attitude and attitude velocity.

[0063] Step 1.1: The original pose (inaccurate) at each time moment is obtained by the TRACE model predicting the video frames at the corresponding time moment. The original pose is preferably obtained by converting the human SMPL parameters.

[0064] Each pose (including the original pose and the pose in the historical physical state) includes the root node displacement of the human body and the rotation parameters of all joints of the human body (the rotation parameters of all joints of the human body include the rotation parameters of the root node, such as ZXY Euler angles, or axis angles or equivalent three-dimensional rotation vectors); in this embodiment, the human body has 21 joints: pelvic root node, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, lower spine, upper spine, left clavicle, left shoulder, left elbow, left wrist, right clavicle, right shoulder, right elbow, right wrist, neck and head;

[0065] All joints in the human body are divided into multiple branches, which are connected in a predetermined order. The branches are as follows: Left lower limb branch: connecting left hip, left knee, left ankle, and left foot in sequence; Right lower limb branch: connecting right hip, right knee, right ankle, and right foot in sequence; Trunk branch: connecting pelvic root node, lower spine, and upper spine in sequence; Left upper limb branch: connecting left clavicle, left shoulder, left elbow, and left wrist in sequence; Right upper limb branch: connecting right clavicle, right shoulder, right elbow, and right wrist in sequence; Head link branch: connecting neck and head in sequence; and the joints above the left hip, right hip, and lower spine are all pelvic root nodes, while the joints above the left clavicle, right clavicle, and neck are all upper spine.

[0066] The original attitude is preferably organized as a 67-dimensional generalized attitude vector in the state dimension, which includes 3-dimensional root node translation, 4-dimensional root node quaternion, and 60-dimensional three-dimensional rotation parameters of 20 controlled joints; the corresponding generalized velocity is preferably organized as a 66-dimensional velocity vector, which includes 3-dimensional root node linear velocity, 3-dimensional root node angular velocity, and 60-dimensional angular velocity of 20 controlled joints.

[0067] The rotation parameters of each joint in the human body are relative to the rotation parameters of the previous joint in the connection sequence. The rotation parameters of the root node are relative to the world coordinate system. The displacement of the root node is the displacement relative to the origin of the world coordinate system. The root node is the pelvic root node (i.e., the pelvic node, which belongs to the 21 joints).

[0068] Postural velocity includes the root node displacement velocity of the human body and the rotational velocity of all joints of the human body (the rotational velocity of all joints of the human body includes the rotational velocity of the root node).

[0069] Step 1.2: Obtain the original pose at the current moment and the historical physical state at the previous moment. Calculate the target's intended pose using the historical physical state, including:

[0070] The changed attitude is calculated based on the integral time between the current moment and the previous moment and the attitude velocity in the historical physical state. Then, the changed attitude is added to the attitude in the historical physical state to obtain the target intention attitude at the current moment. The corresponding calculation formula is as follows:

[0071] ;

[0072] in, For the previous moment Attitude in historical physical states For the current moment Compared to the previous moment The integration time between them For the previous moment Attitude velocity in historical physical states. This calculation method ensures that the target's intended attitude at the current moment inherits the motion trend of the historical physical state at the previous moment, thus maintaining the continuity of the attitude sequence in the time dimension.

[0073] Step 2: Use the original pose and historical physical state at the current moment as input to the trained motion estimation network to obtain the confidence gating matrix, Kalman gain matrix, proportional gain matrix and differential gain matrix of the controller at the current moment, as well as the contact probability, contact Jacobian matrix and contact constraint force.

[0074] Step 2.1: The motion estimation network includes a shared hidden feature extraction branch, as well as parallel optimal state prediction branches, proportional gain prediction branches, differential gain prediction branches, contact state prediction branches, external force prediction branches, and contact Jacobian prediction branches.

[0075] Step 2.2: In the shared hidden feature extraction branch, the original pose, historical physical state, and the expanded features of the joint space inertia matrix of the historical physical state at the current moment are concatenated and passed through three linear units in sequence to obtain the shared hidden features. Each linear unit includes a linear layer connected in sequence (the input and output dimensions of the linear layer in the first linear unit are 4705 and 512, respectively; the input and output dimensions of the linear layer in the second linear unit are 512 and 512, respectively; and the input and output dimensions of the linear layer in the third linear unit are 512 and 512, respectively), layer normalization, and activation layer (LeakyReLU activation function). The joint space inertia matrix of the historical physical state is calculated using the Rigid Body Dynamics Library (RBDL).

[0076] Step 2.3: In the optimal state prediction branch, the shared hidden features are sequentially processed through a GRU (one layer with a hidden dimension of 128), layer normalization, an activation layer (LeakyReLU activation function), and a linear layer (the input and output dimensions of this linear layer are 128 and 128, respectively) to obtain the temporal features. At the same time, the shared hidden features are processed through a linear layer (the input and output dimensions of this linear layer are 512 and 128, respectively) to obtain the state features. The temporal features and state features are concatenated (forming a 256-dimensional layer) and processed through a Dropout layer to obtain the first feature. The first feature is sequentially processed through a linear layer (the input and output dimensions of this linear layer are 256 and 4489, respectively) and a rearrangement operation (rearranged to a 67×67 dimension) to obtain the Kalman gain matrix. At the same time, the first feature is sequentially processed through a linear layer (the input and output dimensions of this linear layer are 256 and 67, respectively) and a non-linear activation function (Sigmoid function) to obtain the confidence gating matrix.

[0077] Step 2.4: In both the proportional gain prediction branch and the differential gain prediction branch, the shared hidden features sequentially pass through a linear layer (where the input and output dimensions of the linear layer in the proportional gain prediction branch are 512 and 128, respectively; and the input and output dimensions of the linear layer in the differential gain prediction branch are 512 and 128, respectively), an activation layer (LeakyReLU activation function), a linear layer (where the input and output dimensions of the linear layer in the proportional gain prediction branch are 128 and 66, respectively; and the input and output dimensions of the linear layer in the differential gain prediction branch are 128 and 66, respectively), and a nonlinear activation function (Sigmoid function), respectively, to obtain the diagonal proportional gain matrix (output of the proportional gain prediction branch) and the diagonal differential gain matrix (output of the differential gain prediction branch), respectively.

[0078] Step 2.5: In the contact state prediction branch, the shared hidden features are passed through a linear layer (the input and output dimensions of this linear layer are 512 and 128, respectively) to obtain contact hidden features. The contact hidden features are concatenated with the preset contact input to obtain contact fusion features. The contact fusion features are then passed through a linear layer (the input and output dimensions of this linear layer are 196 and 128, respectively), an activation layer (Tanh activation function), a linear layer (the input and output dimensions of this linear layer are 128 and 128, respectively), layer normalization, an activation layer (Tanh activation function), a linear layer (the input and output dimensions of this linear layer are 128 and 2, respectively), and a non-linear activation function (Sigmoid function) to obtain the contact probabilities of the left and right feet of the human body.

[0079] Step 2.6: In the external force prediction branch, the shared hidden features are passed through a linear layer (the input and output dimensions of this linear layer are 512 and 128, respectively) to obtain the external force hidden features. The external force hidden features are concatenated with the preset contact input to obtain the external force fusion features. The external force fusion features are then passed through a linear layer (the input and output dimensions of this linear layer are 196 and 128, respectively), an activation layer (LeakyReLU activation function), a linear layer (the input and output dimensions of this linear layer are 128 and 128, respectively), layer normalization, an activation layer (LeakyReLU activation function), and a linear layer (the input and output dimensions of this linear layer are 128 and 6, respectively) to obtain 6-dimensional external force parameters. The 6-dimensional external force parameters are rearranged into the contact constraint forces of the two contact points in three-dimensional space, that is, the contact constraint forces of the left and right feet of the human body are obtained.

[0080] The preset contact input (68-dimensional) includes the three-dimensional position of the local joint at the previous moment (27-dimensional after flattening), the motion amplitude of the local joint at the previous moment relative to the moment before that (9-dimensional), the rotation parameters of the local node at the previous moment (27-dimensional), the contact probability at the previous moment (2-dimensional), and the position of the human body's center of mass at the previous moment (3-dimensional). The position of the human body's center of mass is calculated using a rigid body dynamics library. The local nodes include 9 nodes: pelvic root node, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, and right foot.

[0081] Step 2.7: In the contact Jacobi prediction branch, the shared hidden features pass through two linear layers in sequence (the input and output dimensions of the first linear layer are 512 and 128, respectively, and the input and output dimensions of the second linear layer are 128 and 378, respectively), to obtain the Jacobi parameters (where the Jacobi parameters correspond to the contact points of the two feet, the rotational velocities of all nodes, a total of 63 degrees of freedom, and the six-dimensional contact direction (the six-dimensional contact direction refers to the movement direction of the contact point of each foot is three-dimensional, and there are six dimensions in total for the two feet)). The Jacobi parameters are concatenated with the identity matrix corresponding to the root node displacement to obtain the contact Jacobi matrix (size is 2×66×3).

[0082] Step 3: Using a confidence-gated matrix, perform residual modulation fusion on the original pose and the target intention pose at the current time to obtain the intermediate fused pose at the current time, including:

[0083] The difference between the target intention pose at the current moment and the original pose at the current moment is used as a residual term. After weighting the residual term with the confidence gating matrix at the current moment, it is superimposed with the original pose at the current moment to obtain the intermediate fused pose at the current moment. The corresponding calculation formula is as follows:

[0084] ;

[0085] in, For the current moment The original posture, For the current moment The confidence gating matrix, This refers to the residual term. This step uses a gating mechanism to achieve adaptive weight allocation: a larger weight is assigned when the observation is reliable. To mitigate prediction drift, weights are suppressed when observations are contaminated with noise to maintain temporal continuity. .

[0086] Step 4: Calculate the filtered attitude velocity at the current moment using the Kalman gain matrix and the attitude velocities from historical physical states, including:

[0087] The attitude velocity from the previous time step's historical physical state is used as the observation term, and the filtered attitude velocity from the previous time step is used as the prior term. The difference between the observation term and the prior term is calculated. The difference is then weighted using the Kalman gain matrix at the current time step and superimposed with the prior term to obtain the filtered attitude velocity at the current time step. The corresponding calculation formula is as follows:

[0088] ;

[0089] in, For the current moment Filtered attitude velocity, For the previous moment Filtered attitude velocity, For the current moment The Kalman gain matrix, For the previous moment Attitude velocity in the historical physical state.

[0090] Step 5: Calculate the effective smoothing step size at the current moment based on the filtered attitude velocity, the integral time between the current moment and the previous moment, and the dynamic cutoff frequency. The corresponding calculation formula is as follows:

[0091] ;

[0092] in,

[0093] ;

[0094] in, For the current moment Effective smoothing step size, The integral duration between the current time and the previous time. For the current moment The dynamic cutoff frequency, The cutoff frequency (e.g., 3). This is the speed adjustment coefficient (e.g., 10). For the current moment Filtered attitude velocity;

[0095] This step is used during low-speed movement. make converges to To effectively suppress high-frequency noise that causes control jitter; to enable high-speed maneuvering. and It grows rapidly, thus becoming more dependent on the spatial fusion state. To ensure low-latency tracking.

[0096] Step 6: Based on the effective smoothing step size, perform cross-frame temporal filtering on the attitudes in the intermediate fused attitudes and historical physical states to obtain the desired control target at the current moment, including:

[0097] The difference between the attitude in the intermediate fused attitude and the attitude in the historical physical state is weighted by the effective smoothing step size at the current moment, and then superimposed with the attitude in the historical physical state to obtain the expected control target at the current moment. The corresponding calculation formula is as follows:

[0098] ;

[0099] in, For the current moment The expected control target.

[0100] When the target is in a low-speed motion, a stationary transition, or a fine attitude adjustment phase, the filtered velocity amplitude is small, resulting in a lower dynamic cutoff frequency. Converging to a lower level and approaching This correspondingly places the effective smoothing step size within a stronger smoothing range, thereby enhancing the suppression of high-frequency observation noise, local jitter, and control spikes. When the target is in a rapid turning, arm swinging, leaping, or other high-speed maneuvering phase, the filtered velocity amplitude increases, driving the dynamic cutoff frequency. and effective smoothing step size The synchronous increase reduces the lag caused by excessive filtering, making the system more reliant on the intermediate fusion state. To achieve low-latency tracking, the aforementioned adaptive bandwidth modulation and cross-frame temporal filtering joint mechanism utilizes intermediate fusion states to provide a spatial attitude reference closer to the actual observation and intention prediction in the current frame. Simultaneously, it preserves the temporal continuity of the previous moment after dynamic constraint correction using historical physical states. The final output achieves a desired control target that combines low latency, low jitter, and smooth physical plausibility. This provides a stable input for subsequent PD control and dynamic integration.

[0101] Step 7: Input the desired control target at the current moment into the physics simulation engine, and use the PD controller to calculate the motion torque driving each joint at the current moment. The corresponding calculation formula is as follows:

[0102] in, A matrix consisting of the current motion torques of all joints. For the current moment The proportional gain matrix, For the current moment The differential gain matrix, The desired control objective at the current moment, The attitude in the previous moment's historical physical state. The attitude velocity in the previous physical state;

[0103] The dynamic parameter vector output from step 2, consisting of the proportional gain matrix and the differential gain matrix, is adaptively provided, enabling the control stiffness and damping characteristics to change dynamically under different joints, different motion stages, and different contact conditions. Preferably, for the support phase, the contact stabilization stage, or the load-bearing joint, the proportional gain matrix and the differential gain matrix are increased to enhance disturbance rejection stability; for the airborne stage, the rapid swing stage, or the end joint with high flexibility requirements, the control stiffness is reduced to decrease the proportional gain and differential gain to retain the natural motion amplitude, thereby improving the adaptability of the control process to complex motion modes.

[0104] Step 8: Combining the motion torques of all joints, use the Euler-Lagrange dynamics equations with contact constraints to solve for the instantaneous acceleration at the current moment. The corresponding calculation formula is as follows:

[0105] ;

[0106] in, For the current moment Instantaneous acceleration, The joint space inertia matrix of the historical physical state (where the previous time step is the previous time step). of), For the previous moment The Coriolis force, centrifugal force, and generalized gravity term were calculated using a rigid body dynamics library. For the current moment The contact constraint force between the two feet, To access the Jacobian matrix.

[0107] By explicitly introducing contact constraints in the dynamic solution, abnormal phenomena that do not conform to the laws of physics, such as foot slippage, ground penetration, and suspended support, can be effectively suppressed, thereby improving the environmental consistency and executability of attitude tracking results.

[0108] Step 9: Based on the instantaneous acceleration at the current moment, perform forward integration using a semi-implicit second-order Euler integral scheme to update the physical state at the current moment, including:

[0109] Phase 1: Update the attitude velocity at the current moment based on the instantaneous acceleration at the current moment: ;

[0110] Update the current attitude using the current attitude velocity: ;

[0111] Substitute the current attitude velocity and current attitude into step 7 to recalculate the current motion torque of each joint, and then execute step 8 to recalculate the current instantaneous acceleration. ;

[0112] Second stage: Utilizing the recalculated instantaneous acceleration at the current moment And the attitude at the current moment obtained from the first stage calculation. and attitude velocity The current attitude and attitude velocity are updated and referred to as the final attitude, respectively. and final attitude velocity The corresponding calculation formula is as follows:

[0113] ;

[0114] ;

[0115] Among them, the final posture and final attitude velocity The system constructs the physical state at the current moment, thereby estimating the physical state of the human body in the video sequence at each moment. In the process of estimating the physical state at the next moment, the physical state estimated at the current moment is used as the historical physical state to achieve a physical closed-loop attitude estimation. This ensures that the attitude inference at subsequent moments is continuously based on the reliable historical state after physical correction, thereby gradually reducing the problems of single-frame observation error accumulation, prediction drift, and dynamic inconsistency. Through the above closed-loop iterative process, the system can simultaneously utilize visual measurement information, motion estimation network prediction priors, and physical simulation constraints under monocular video input conditions to achieve joint optimization of the target attitude sequence. This improves the performance of attitude tracking results in terms of temporal continuity, local detail stability, contact rationality, and overall physical consistency, and constructs a closed-loop attitude tracking system that strictly conforms to the laws of physical autoregression.

[0116] Under monocular video input conditions, this method simultaneously utilizes visual measurement information, network prediction priors, and physical simulation constraints to achieve joint optimization of the target pose sequence. Compared to schemes that rely solely on visual observation or solely on motion regression, this method can significantly improve the performance of pose tracking results in terms of temporal continuity, local detail stability, contact rationality, and overall physical consistency, thereby constructing a closed-loop pose tracking system that strictly conforms to the autoregressive laws of physics.

[0117] In another embodiment, during the training of the motion estimation network, the Human3.6M dataset is used as the training data source. The original data size is approximately 3.6 million 3D human poses and corresponding images. During training, the random seed is 0, the number of training epochs is 30, the number of warm-up epochs is 5, the batch size is 32, the initial learning rate is 5×10−4, the optimizer is Adamax, and all trainable parameters of the motion estimation network and the selection fusion network are optimized simultaneously. The learning rate scheduler is MultiStepLR, the milestone lr_steps is [10,13], the decay coefficient gamma is 0.1, the gradient clipping threshold grad_clip is 0.8, its minimum cutoff frequency euro_mincutoff is 3, the velocity adjustment coefficient euro_beta is 10, the derivative cutoff frequency euro_dcutoff is 2, and the sampling frequency euro_freq is 30Hz.

[0118] The total loss of the motion estimation network is constructed as a joint supervised objective weighted by six types of sub-losses, and the total loss... The calculation formula is as follows:

[0119] ;

[0120] in,

[0121] ;

[0122] ;

[0123] ;

[0124] ;

[0125] ;

[0126] ;

[0127] in, , , , , and These are the first sub-loss, second sub-loss, third sub-loss, fourth sub-loss, fifth sub-loss, and sixth sub-loss, respectively, and their weighting coefficients (e.g., 0.5, 5, 0.5, 5, 0.5, and 1.0, respectively). For batch size, For the first One batch, For the current moment No. The final pose of each joint corresponds to its joint position (where the rotation parameters in each pose are the rotation parameters of the previous joint in the connection sequence, and thus the position of each joint can be calculated), and 20 represents the 20 joints other than the root node (in this embodiment, there are a total of 21 human joints). For the current moment The joint position corresponding to the final pose of the root node. For the current moment No. The actual joint positions corresponding to each joint. For the current moment The actual joint position corresponding to the root node. For the current moment No. The joint position corresponding to the target intention pose of each joint. Current moment The joint position corresponding to the target intention pose of the root node. For the current moment No. Predicted contact probability of each foot For the current moment No. The actual contact probability of each foot Previous moment No. The final pose of each joint corresponds to its joint position. Previous moment No. The actual joint positions corresponding to each joint. It is an L1 norm. 2-norm Let be the absolute value, where The value is 0 to 20, representing 21 joints including the root node. The range 1 to 20 represents the 20 joints excluding the root node.

[0128] The parameters of the motion estimation network are updated by backpropagation using the total loss function until the motion estimation network converges, resulting in a well-trained motion estimation network.

[0129] In another embodiment, the present invention also provides an attitude estimation system based on adaptive smoothing and physical loop closure, comprising:

[0130] The acquisition module is used to acquire video sequences, traverse each moment of the video sequence in chronological order, and estimate the physical state of the human body in the video sequence at each moment.

[0131] The first module is used to obtain the original attitude at the current moment and the historical physical state at the previous moment. The target intention attitude is calculated using the historical physical state, where each physical state includes attitude and attitude velocity.

[0132] The second module is used to take the original posture and historical physical state at the current moment as input to the trained motion estimation network to obtain the confidence gating matrix, Kalman gain matrix, proportional gain matrix and differential gain matrix of the controller at the current moment, as well as the contact probability, contact Jacobian matrix and contact constraint force.

[0133] The third module uses a confidence gating matrix to perform residual modulation fusion on the original pose and the target intention pose at the current moment to obtain the intermediate fused pose at the current moment.

[0134] The fourth module calculates the filtered attitude velocity at the current moment using the Kalman gain matrix and the attitude velocities in the historical physical states.

[0135] The fifth module is used to calculate the effective smoothing step size at the current moment based on the filtered attitude velocity at the current moment, the integral duration between the current moment and the previous moment, and the dynamic cutoff frequency.

[0136] The sixth module is used to perform cross-frame temporal filtering between the intermediate fused attitude and the attitude in the historical physical state based on the effective smoothing step size, so as to obtain the desired control target at the current moment.

[0137] The seventh module is used to input the desired control target at the current moment into the physics simulation engine and use the PD controller to calculate the motion torque of each joint at the current moment.

[0138] The eighth module is used to combine the motion torques of all joints and use the Euler-Lagrange dynamics equations with contact constraints to solve for the instantaneous acceleration at the current moment;

[0139] The ninth module is used to perform forward integration based on the instantaneous acceleration at the current moment using a semi-implicit second-order Euler integral scheme to update the physical state at the current moment.

[0140] For specific constraints on attitude estimation systems based on adaptive smoothing and physical loop closure, please refer to the constraints on attitude estimation methods based on adaptive smoothing and physical loop closure mentioned above, which will not be repeated here.

[0141] This attitude estimation method based on adaptive smoothing and physical closed loop does not rely on multiple cameras or large-scale motion capture annotation. It achieves high-precision, low-latency and physically consistent attitude tracking by combining monocular video sequences with motion estimation networks, intermediate attitude fusion, attitude velocity filtering, desired control target modulation, closed-loop physical control and dynamic integration.

[0142] This method can significantly improve the performance of attitude tracking results in terms of temporal continuity, local detail stability, contact rationality, and overall physical consistency, thereby constructing a closed-loop attitude tracking system that strictly conforms to the autoregressive law of physics.

[0143] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A pose estimation method based on adaptive smoothing and physical loop closure, characterized in that: Obtain the video sequence, iterate through each moment of the video sequence in chronological order, and for each moment, estimate the physical state of the human body in the video sequence. The estimation process for the physical state of the human body at each moment is as follows: Step 1: Obtain the original attitude at the current moment and obtain the historical physical state at the previous moment. Calculate the target intention attitude using the historical physical state, where each physical state includes attitude and attitude velocity. Step 2: Use the original pose and historical physical state at the current moment as input to the trained motion estimation network to obtain the confidence gating matrix, Kalman gain matrix, proportional gain matrix and differential gain matrix of the controller at the current moment, as well as the contact probability, contact Jacobian matrix and contact constraint force. Step 3: Use the confidence gating matrix to perform residual modulation fusion on the original pose and the target intention pose at the current time to obtain the intermediate fused pose at the current time. Step 4: Calculate the filtered attitude velocity at the current moment using the Kalman gain matrix and the attitude velocities in the historical physical states; Step 5: Calculate the effective smoothing step size at the current moment based on the filtered attitude velocity at the current moment, the integral duration between the current moment and the previous moment, and the dynamic cutoff frequency; Step 6: Based on the effective smoothing step size, perform cross-frame temporal filtering on the attitudes in the intermediate fused attitudes and historical physical states to obtain the desired control target at the current moment. Step 7: Input the desired control target at the current moment into the physics simulation engine, and use the PD controller to calculate the motion torque of each joint at the current moment; Step 8: Combining the motion torques of all joints, use the Euler-Lagrange dynamics equations with contact constraints to solve for the instantaneous acceleration at the current moment; Step 9: Based on the instantaneous acceleration at the current moment, perform forward integration using a semi-implicit second-order Euler integral scheme to update the physical state at the current moment.

2. The attitude estimation method based on adaptive smoothing and physical loop closure as described in claim 1, characterized in that: Each posture includes the root node displacement of the human body and the rotation parameters of all joints of the human body; All joints in the human body are divided into multiple branches and connected in a preset order. The rotation parameters of each joint in the human body are relative to the rotation parameters of the previous joint in the connection order, and the rotation parameters of the root node are relative to the world coordinate system. The root node is the pelvic root node. Postural velocity includes the root node displacement velocity of the human body and the rotational velocity of all joints in the human body.

3. The attitude estimation method based on adaptive smoothing and physical loop closure as described in claim 1, characterized in that: The process of obtaining the current original pose and the previous historical physical state, and calculating the target intention pose using the historical physical state, includes: The changed attitude is calculated based on the integral duration between the current moment and the previous moment and the attitude velocity in the historical physical state. The changed attitude is then added to the attitude in the historical physical state to obtain the target intention attitude at the current moment.

4. The attitude estimation method based on adaptive smoothing and physical loop closure as described in claim 2, characterized in that: The motion estimation network includes a shared hidden feature extraction branch, as well as parallel optimal state prediction branches, proportional gain prediction branches, differential gain prediction branches, contact state prediction branches, external force prediction branches, and contact Jacobian prediction branches. In the shared hidden feature extraction branch, the original pose, historical physical state and the expanded features of the joint space inertial matrix of the historical physical state at the current moment are concatenated and then passed through three linear units in sequence to obtain the shared hidden features. Each linear unit includes a linear layer, a layer normalization layer and an activation layer connected in sequence. The joint space inertia matrix of the historical physical state is calculated using the rigid body dynamics library; In the optimal state prediction branch, the shared hidden features pass through GRU, layer normalization, activation layer and linear layer in sequence to obtain temporal features. At the same time, the shared hidden features pass through the linear layer to obtain state features. The temporal features and state features are concatenated and then passed through the Dropout layer to obtain the first feature. The first feature passes through the linear layer and rearrangement operation in sequence to obtain the Kalman gain matrix. At the same time, the first feature passes through the linear layer and nonlinear activation function in sequence to obtain the confidence gating matrix. In the proportional gain prediction branch and the differential gain prediction branch, the shared hidden features pass through a linear layer, an activation layer, a linear layer and a nonlinear activation function in sequence to obtain the diagonal proportional gain matrix and the diagonal differential gain matrix, respectively. In the contact state prediction branch, the shared hidden features pass through a linear layer to obtain contact hidden features. The contact hidden features are then concatenated with a preset contact input to obtain contact fusion features. The contact fusion features then pass through a linear layer, an activation layer, a linear layer, layer normalization, an activation layer, a linear layer, and a nonlinear activation function in sequence to obtain the contact probabilities of the left and right feet of the human body. In the external force prediction branch, the shared hidden features pass through the linear layer to obtain the external force hidden features. The external force hidden features are then spliced ​​with the preset contact input to obtain the external force fusion features. The external force fusion features then pass through the linear layer, activation layer, linear layer, layer normalization, activation layer, and linear layer in sequence to obtain the contact constraint forces of the left and right feet of the human body. In the contact Jacobi prediction branch, the shared hidden features pass through two linear layers in sequence to obtain the Jacobi parameters. The Jacobi parameters are then concatenated with the identity matrix corresponding to the root node displacement to obtain the contact Jacobi matrix.

5. The attitude estimation method based on adaptive smoothing and physical loop closure as described in claim 4, characterized in that: The preset contact inputs include the three-dimensional position of the local joint at the previous moment, the motion amplitude of the local joint at the previous moment relative to the moment before that, the rotation parameters of the local node at the previous moment, the contact probability at the previous moment, and the position of the human body's center of mass at the previous moment. The human body's execution position is calculated using a rigid body dynamics library.

6. The attitude estimation method based on adaptive smoothing and physical loop closure as described in claim 1, characterized in that: The process of using a confidence-gated matrix to perform residual modulation fusion on the original pose and the target intention pose at the current moment to obtain the intermediate fused pose at the current moment includes: The difference between the target intention pose at the current moment and the original pose at the current moment is used as a residual term. The residual term is weighted by the confidence gating matrix at the current moment and then superimposed with the original pose at the current moment to obtain the intermediate fused pose at the current moment.

7. The attitude estimation method based on adaptive smoothing and physical loop closure as described in claim 1, characterized in that: The calculation of the filtered attitude velocity at the current moment using the Kalman gain matrix and attitude velocities from historical physical states includes: The attitude velocity in the historical physical state of the previous moment is taken as the observation term, and the filtered attitude velocity of the previous moment is taken as the prior term. The difference between the observation term and the prior term is calculated. The difference is weighted by the Kalman gain matrix of the current moment and then superimposed with the prior term to obtain the filtered attitude velocity of the current moment.

8. The attitude estimation method based on adaptive smoothing and physical loop closure as described in claim 1, characterized in that: The formula for calculating the effective smoothing step size at the current time is as follows: ； in, ； in, The effective smoothing step size at the current moment, The integral duration between the current time and the previous time. For the current moment The dynamic cutoff frequency, The cutoff frequency, This is the speed adjustment coefficient. The current filtered attitude velocity; The difference between the intermediate fused attitude and the attitude in the historical physical state is weighted by the effective smoothing step size at the current moment, and then superimposed with the attitude in the historical physical state to obtain the desired control target at the current moment.

9. The attitude estimation method based on adaptive smoothing and physical loop closure as described in claim 1, characterized in that: The calculation formula for using the PD controller to drive the current motion torque of each joint is as follows: in, A matrix consisting of the current motion torques of all joints. For the current moment The proportional gain matrix, For the current moment The differential gain matrix, The desired control objective at the current moment, The attitude in the previous moment's historical physical state. The attitude velocity in the previous physical state; The instantaneous acceleration at the current moment is solved by combining the motion torques of all joints and using the Euler-Lagrange dynamics equations with contact constraints.

10. The attitude estimation method based on adaptive smoothing and physical loop closure as described in claim 9, characterized in that: The process of updating the physical state at the current moment by performing forward integration using a semi-implicit second-order Euler integral scheme based on the instantaneous acceleration at the current moment includes: Phase 1: Based on the instantaneous acceleration at the current moment Update the attitude and velocity at the current moment: ; Update the current attitude using the current attitude velocity: ; Substitute the current attitude velocity and current attitude into step 7 to recalculate the current motion torque of each joint, and then execute step 8 to recalculate the current instantaneous acceleration. ; Second stage: Utilizing the recalculated instantaneous acceleration at the current moment And the attitude at the current moment obtained from the first stage calculation. and attitude velocity The current attitude and attitude velocity are updated and referred to as the final attitude, respectively. and final attitude velocity final posture and final attitude velocity This constitutes the physical state at the current moment.

Images