Human body posture reconstruction method based on deep learning network and hierarchical optimization
By using deep learning networks and hierarchical optimization methods, combined with kinematic and dynamic constraints, the problems of unreasonable posture and jitter in human posture reconstruction by inertial measurement units were solved, achieving high-precision and natural posture reconstruction results.
Patent Information
- Application Number
- CN202511724294.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-23
- Publication Date
- 2026-02-06
AI Technical Summary
Existing human posture reconstruction technology based on inertial measurement units suffers from problems such as low joint angle positioning accuracy, unreasonable generated posture, and posture jitter during rapid movement, making it difficult to meet physical constraints and kinematic requirements.
A deep learning network combined with a hierarchical optimization method is used. Data is collected through an inertial measurement unit, preprocessed and initially estimated, and then optimized at the joint and physical layers using kinematic and dynamic constraints to eliminate unreasonable postures, ensure that the posture conforms to the laws of human kinematics and physics, and perform smoothing.
It achieves high-precision human posture estimation, eliminates unreasonable postures, improves the naturalness and physical rationality of postures, and reduces jitter and abrupt changes.
Smart Images
Figure CN121482867A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of motion capture, and particularly relates to a human pose reconstruction method based on a deep learning network and hierarchical optimization. BACKGROUND
[0002] Human pose reconstruction refers to the process of converting the motion and posture of a human body in the real world into a digital model through sensor data or other technical means. This technology plays an important role in computer vision, robotics, virtual reality, health monitoring, and other fields, and can provide accurate three-dimensional human pose information for related applications. Human pose reconstruction mainly obtains three-dimensional spatial pose information of the human body from single-view or multi-view images, videos, or sensor data. These data may come from cameras, depth cameras, inertial measurement units (IMUs), or millimeter wave radars. By analyzing these data, a human skeleton or three-dimensional model can be generated, and dynamic changes can be captured. Human pose reconstruction based on inertial measurement units (IMUs) measures acceleration, angular velocity, and other data to capture human motion in real time, has low cost, strong portability, and low requirements for environmental conditions such as lighting, but data noise, sensor drift, pose jitter, and complex timing processing are also important problems that need to be solved.
[0003] Current human pose reconstruction technology based on inertial measurement units has some significant defects. First, using purely data-driven methods relies too much on neural networks and lacks constraints on physical laws, resulting in low positioning accuracy of joint angles and large average skeletal joint position errors. Second, the poses generated by these methods are often physically unreasonable, with joints exhibiting super-limit motion (such as knee hyperextension) or limbs experiencing unnatural stretching, and even characters penetrating the floor, which violates common sense. Finally, when dealing with fast motion, existing technology has difficulty maintaining smooth transitions of poses, and is prone to jitter and mutations, resulting in dynamic discontinuity and affecting the natural fluidity of motion. SUMMARY
[0004] The present application proposes a human pose reconstruction method based on a deep learning network and hierarchical optimization to address the shortcomings of existing technology.
[0005] The above-mentioned object of the present application is achieved by the following technical solutions: A human pose reconstruction method based on a deep learning network and hierarchical optimization, comprising the following steps: Step 1: Collect data from 8 joints of the human body using an inertial measurement unit (IMU) and preprocess the data; Step 2: Establish a deep learning network to preliminarily estimate the human pose using the preprocessed data from Step 1 as input; Step 3: Perform hierarchical optimization on the output information of Step 2: This includes joint layer optimization and physical layer optimization. First, joint layer optimization is performed by using kinematic constraints to correct the initial posture of the neural network output, eliminating physically impossible postures, satisfying the posture sequence of kinematic constraints, and ensuring that the posture conforms to human kinematic constraints. Then, physical layer optimization is performed by using dynamic constraints to ensure that the posture conforms to physical laws and obtains a physically reasonable final posture. Step 4: Smooth the optimized pose to make the pose changes between adjacent frames smooth and output the pose information.
[0006] Furthermore, in step 1, the eight joints are the lower side of the left knee, the lower side of the right knee, the left ankle, the right ankle, the head, the waist, the lower side of the left elbow, and the lower side of the right elbow. An inertial measurement unit (IMU) is worn at each joint. The collected data includes acceleration, angular velocity, and magnetic field data at the corresponding joints. The data preprocessing process is as follows: first, calibration is performed to remove data deviations, then coordinate system transformation is performed to convert the sensor coordinate system to the global coordinate system, and the data is denoised.
[0007] Furthermore, in step 2, the established deep learning network consists of an input layer, a regularization and feature transformation module, a sequence encoding module, an attention mechanism module, and an output layer. The input layer is used to input preprocessed inertial measurement unit (IMU) data. The regularization and feature transformation module uses Dropout for input regularization to prevent overfitting and enhance the model's generalization performance. It also uses Linear for initial feature extraction and ReLU to introduce nonlinear activation, enhancing the model's expressive power and overcoming the limitations of purely linear transformations. The sequence encoding module uses a two-layer BiLSTM neural network to deeply encode sequence information. The attention mechanism module focuses on and extracts key sequence information. The output layer outputs the rotation information of each joint and the human posture sequence information.
[0008] Furthermore, in step 2, the deep learning network uses a hybrid loss function of quaternion angular distance loss and position error: in, This is a quaternion prediction value. For the true value, These are predicted values for joint position. This represents the actual joint position.
[0009] Furthermore, in step 2, the estimation of human posture using a deep learning network employs inverse kinematics, which solves for joint rotation angles by calculating joint positions. This process is divided into three stages: the first stage uses IMU measurements to regress leaf joint positions through a deep learning network; the second stage uses leaf joint positions and IMU measurements to regress whole-body joint positions; and the third stage uses whole-body joint positions and IMU measurements to regress joint rotation angles, which are then used to map human posture movements.
[0010] Furthermore, the specific implementation method of joint layer optimization in step 3 is as follows: The initial pose estimated by the neural network is input, the target loss function is constructed and quaternion normalization constraints are set, and then the minimize function is called, passing in the target loss function, initial pose, optimization method and quaternion normalization constraints; the optimization method uses the Sequential Quadratic Programming (SLSQP) algorithm to perform constrained iterative optimization and solve the problem, and outputs a pose sequence that satisfies the kinematic constraints; The objective loss function includes smoothing constraints and joint angle constraints. The target loss function is: in: For the loss function of the smoothing constraint, The loss function for the joint angle constraint; The loss function for smoothing constraints is: in: Let t be the rotation angle of joint j at time t, and T be the total number of frames in the time series; The loss function formula for joint angle constraint is: in, Let be the current rotation angle of joint j. Let be the current rotation angle of joint j in frame t. Let J be the minimum and maximum allowable rotation angles of joint j; The quaternion normalization constraint formula is: g(q) = ; in =cos( Let the unit vector of the rotation axis be (). ,but = sin( ), = sin( ), = sin( ), Let be the rotation angle of joint j.
[0011] Furthermore, the specific implementation method of physical layer optimization in step 3 is as follows: Input the kinematically optimized attitude and the preprocessed data from the inertial measurement unit (IMU), detect the ground contact state and construct physical constraints, then use an optimizer to minimize the physical loss function, perform iterative optimization, calculate the gradient of the loss function and update the attitude parameters, and output the physically reasonable final attitude. The formula for calculating the probability of contact with the ground is: Where σ(x) represents the Sigmoid function, which maps any real number to the probability interval (0,1); Represents the acceleration vectors of the left and right ankle IMUs; Ground contact determination The physical loss function is: Among them, the contact force loss item ; Indicates the acceleration of the center of mass. Represents ground reaction force; loss of centroid consistency term = ; The position of the centroid obtained from the attitude estimation. The centroid location estimated from IMU data.
[0012] The advantages and positive effects of this invention are as follows: 1. This invention achieves high-precision attitude estimation by integrating deep learning and constraint optimization.
[0013] 2. This invention introduces an attention mechanism into the recurrent neural network, which makes the model focus on the joints with a large amount of information about human joint movement in the sequence when reconstructing human posture and movement, thereby improving the reconstruction accuracy.
[0014] 3. The present invention optimizes the segmental layer by using angle limiting constraints and bone length constraints to eliminate abnormal postures and make the human posture conform to kinematic constraints.
[0015] 4. In the physical layer optimization step, this invention eliminates problems such as floor penetration that do not conform to the laws of physics by using contact force constraints and momentum conservation constraints, so that the final posture conforms to the laws of physics. Attached Figure Description
[0016] Figure 1 This is a flowchart of the human pose reconstruction method based on deep learning networks and hierarchical optimization of the present invention; Figure 2 This is a diagram of the human pose estimation network structure of the present invention; Figure 3 This is a flowchart illustrating the principle of the joint optimization layer of this invention. Figure 4 This is a flowchart illustrating the principle of the physical optimization layer of this invention; Figure 5 Comparison of the evaluation results of the DIP-IMU dataset in this invention; Figure 6 Comparison of the evaluation results of the TotalCapture dataset in this invention. Detailed Implementation
[0017] The structure of the present invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that these embodiments are descriptive and not limiting.
[0018] For a human pose reconstruction method based on deep learning networks and hierarchical optimization, please refer to [link / reference]. Figures 1-6 The key points of this method are: preliminary pose estimation based on deep learning and hierarchical optimization of the preliminary pose estimation results, which are used to further refine the pose reconstruction. Specifically, it includes the following steps: Step 1: Use an inertial measurement unit (IMU) to collect data from eight joints of the human body and preprocess the data. The eight joints are the lower left knee, the lower right knee, the left ankle, the right ankle, the head, the waist, the lower left elbow, and the lower right elbow. An IMU is attached to each joint. The collected data includes acceleration, angular velocity, and magnetic field data at the corresponding joint. The data preprocessing process is as follows: First, calibration is performed to remove data deviations. Then, coordinate system transformation is performed to convert the sensor coordinate system to the global coordinate system, and noise reduction is applied to the data.
[0019] Step 2: Establish a deep learning network to initially estimate human pose using the preprocessed data from Step 1 as input. The established deep learning network consists of an input layer, a regularization and feature transformation module, a sequence encoding module, an attention mechanism module, and an output layer. The input layer is used to input the preprocessed inertial measurement unit (IMU) data. The regularization and feature transformation module uses Dropout for input regularization to prevent overfitting and enhance the model's generalization performance. It also uses Linear for initial feature extraction and ReLU to introduce nonlinear activation, enhancing the model's expressive power and overcoming the limitations of purely linear transformations. The sequence encoding module uses a two-layer BiLSTM neural network to deeply encode sequence information. The attention mechanism module focuses on and extracts key sequence information. The output layer outputs the rotation information of each joint and the human pose sequence information.
[0020] Deep learning networks use a hybrid loss function that combines quaternion angular distance loss and position error: in This is a quaternion prediction value. For the true value, These are predicted values for joint position. This represents the actual joint position.
[0021] Human posture estimation using deep learning networks employs inverse kinematics, which calculates joint rotation angles from joint positions. The process is divided into three stages: the first stage regresses leaf joint positions from IMU measurements using the deep learning network; the second stage regresses whole-body joint positions from leaf joint positions and IMU measurements; and the third stage regresses joint rotation angles from whole-body joint positions and IMU measurements, mapping human posture and movement based on these joint rotation angles.
[0022] Step 3: Perform layered optimization on the output information from Step 2. This includes joint-level optimization and physical-level optimization, using physical and kinematic constraints to refine the pose output by the network. First, joint-level optimization is performed, using kinematic constraints to correct the initial pose output by the neural network, eliminating physically impossible poses and ensuring a pose sequence that satisfies kinematic constraints, thus ensuring the pose conforms to human kinematics limitations. Impossible poses include excessively bent knees, etc. Then, physical-level optimization is performed, using dynamic constraints to ensure the pose conforms to physical laws, obtaining a physically reasonable final pose and resolving physical inconsistencies such as "floor penetration."
[0023] The specific implementation method of joint layer optimization is as follows: The initial pose estimated by the neural network is input, the target loss function is constructed and quaternion normalization constraints are set, and then the minimize function is called, passing in the target loss function, initial pose, optimization method and quaternion normalization constraints; the optimization method uses the Sequential Quadratic Programming (SLSQP) algorithm to perform constrained iterative optimization and solve the problem, and outputs a pose sequence that satisfies the kinematic constraints; The objective loss function includes smoothing constraints and joint angle constraints. The target loss function is: in: For the loss function of the smoothing constraint, The loss function for the joint angle constraint; The loss function for smoothing constraints is: in: Let t be the rotation angle of joint j at time t, and T be the total number of frames in the time series; The loss function formula for joint angle constraint is: in, Let be the current rotation angle of joint j. Let be the current rotation angle of joint j in frame t. Let J be the minimum and maximum allowable rotation angles of joint j; The quaternion normalization constraint formula is: g(q) = ; in =cos( Let the unit vector of the rotation axis be (). ,but = sin( ), = sin( ), = sin( ), Let be the rotation angle of joint j.
[0024] Specific implementation methods for physical layer optimization: Input the kinematically optimized attitude and the preprocessed data from the inertial measurement unit (IMU), detect the ground contact state and construct physical constraints, then use an optimizer to minimize the physical loss function, perform iterative optimization, calculate the gradient of the loss function and update the attitude parameters, and output the physically reasonable final attitude. The formula for calculating the probability of contact with the ground is: Where σ(x) represents the Sigmoid function, which maps any real number to the probability interval (0,1); Represents the acceleration vectors of the left and right ankle IMUs; Ground contact determination The physical loss function is: Among them, the contact force loss item ; Indicates the acceleration of the center of mass. Represents ground reaction force; loss of centroid consistency term = ; The position of the centroid obtained from the attitude estimation. The centroid location estimated from IMU data.
[0025] Step 4, Pose Output: Smooth the optimized pose to make the pose changes between adjacent frames smooth and output the pose information.
[0026] To verify the superiority of the human pose reconstruction method based on deep learning networks and hierarchical optimization, this invention uses the publicly available network datasets DIP-IMU and TotalCapture to evaluate the performance of the method. The performance reference metrics include SIP Error (pose error), Angular Error (angle error), Positional Error (position error), Mesh Error (mesh error), and Jitter Error (jitter error). The performance metrics are compared with those of the classic human pose reconstruction methods DIP and TransPose, highlighting the superiority of this method.
[0027] Pose error measures the deviation of the overall pose (including rotation and translation) from the true pose. Reducing this error makes the reconstructed human pose closer to the true pose, improving the global accuracy of the method.
[0028] The reduction in joint angle error makes the estimation of joint rotation more accurate, thereby increasing the naturalness and realism of human posture.
[0029] Position error measures the deviation of a joint or body part from its position in 3D space. Reducing position error makes the reconstructed joint positions more accurate and reduces misalignment of body parts.
[0030] Mesh error refers to the deviation between the vertices of the human body mesh and the actual surface. Reducing this error makes the entire body mesh model more closely resemble the real shape.
[0031] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.
Claims
1. A human pose reconstruction method based on deep learning networks and hierarchical optimization, characterized in that: Includes the following steps: Step 1: Use an inertial measurement unit (IMU) to collect data from eight joints in the human body and preprocess the data; Step 2: Establish a deep learning network and use the preprocessed data from Step 1 as input to make a preliminary estimate of human pose; Step 3: Perform hierarchical optimization on the output information of Step 2: This includes joint layer optimization and physical layer optimization. First, joint layer optimization is performed by using kinematic constraints to correct the initial posture of the neural network output, eliminating physically impossible postures, satisfying the posture sequence of kinematic constraints, and ensuring that the posture conforms to human kinematic constraints. Then, physical layer optimization is performed by using dynamic constraints to ensure that the posture conforms to physical laws and obtains a physically reasonable final posture. Step 4: Smooth the optimized pose to make the pose changes between adjacent frames smooth and output the pose information.
2. The human pose reconstruction method based on deep learning networks and hierarchical optimization according to claim 1, characterized in that: In step 1, the eight joints are the lower side of the left knee, the lower side of the right knee, the left ankle, the right ankle, the head, the waist, the lower side of the left elbow, and the lower side of the right elbow. An inertial measurement unit (IMU) is worn at each joint. The collected data includes acceleration, angular velocity, and magnetic field data at the corresponding joints. The data preprocessing process is as follows: first, calibration is performed to remove data deviations, then coordinate system transformation is performed to convert the sensor coordinate system to the global coordinate system, and the data is denoised.
3. The human pose reconstruction method based on deep learning networks and hierarchical optimization according to claim 1, characterized in that: In step 2, the established deep learning network consists of an input layer, a regularization and feature transformation module, a sequence encoding module, an attention mechanism module, and an output layer. The input layer is used to input preprocessed inertial measurement unit (IMU) data. The regularization and feature transformation module is used to perform input regularization through Dropout, initial feature extraction through Linear, and nonlinear activation through ReLU. The sequence encoding module uses a two-layer BiLSTM neural network to deeply encode sequence information. The attention mechanism module is used to focus on and extract key sequence information. The output layer outputs the rotation information of each joint and the human posture sequence information.
4. The human pose reconstruction method based on deep learning networks and hierarchical optimization according to claim 1, characterized in that: In step 2, the deep learning network uses a hybrid loss function of quaternion angular distance loss and position error: ； in, This is a quaternion prediction value. For the true value, These are predicted values for joint position. This represents the actual joint position.
5. The human pose reconstruction method based on deep learning networks and hierarchical optimization according to claim 1, characterized in that: In step 2, the estimation of human posture is performed using a deep learning network and the inverse kinematics method is applied. The joint rotation angle is calculated by the joint position and is divided into three stages. In the first stage, the leaf joint position is regressed from the inertial measurement unit (IMU) measurement value through the deep learning network. In the second stage, the whole body joint position is regressed from the leaf joint position and the IMU measurement value. In the third stage, the joint rotation angle is regressed from the whole body joint position and the IMU measurement value. The human posture movement is mapped according to the joint rotation angle.
6. The human pose reconstruction method based on deep learning networks and hierarchical optimization according to claim 1, characterized in that: In step 3, the specific implementation method of joint layer optimization is as follows: Input the initial pose estimated by the neural network, construct the target loss function and set the quaternion normalization constraint, and then call the minimize function, passing in the target loss function, initial pose, optimization method and quaternion normalization constraint. The optimization method uses the Sequential Quadratic Programming (SLSQP) algorithm to perform constrained iterative optimization and outputs a sequence of postures that satisfy kinematic constraints. The objective loss function includes smoothing constraints and joint angle constraints. The target loss function is: ； in: For the loss function of the smoothing constraint, The loss function for the joint angle constraint; The loss function for smoothing constraints is: ； in: Let t be the rotation angle of joint j at time t, and T be the total number of frames in the time series; The loss function formula for joint angle constraint is: ； ； in, Let be the current rotation angle of joint j. Let be the current rotation angle of joint j in frame t. Let J be the minimum and maximum allowable rotation angles of joint j; The quaternion normalization constraint formula is: g(q) = ; in =cos( ), set the unit vector of the joint rotation axis as ( ,but = sin( ), = sin( ), = sin( ), Let be the rotation angle of joint j.
7. The human pose reconstruction method based on deep learning networks and hierarchical optimization according to claim 1, characterized in that: Step 3, the specific implementation method of physical layer optimization: Input the kinematically optimized attitude and the preprocessed data from the inertial measurement unit (IMU), detect the ground contact state and construct physical constraints, then use an optimizer to minimize the physical loss function, perform iterative optimization, calculate the gradient of the loss function and update the attitude parameters, and output the physically reasonable final attitude. The formula for calculating the probability of contact with the ground is: ； Where σ(x) represents the Sigmoid function, which maps any real number to the probability interval (0,1); Represents the acceleration vectors of the left and right ankle IMUs; The ground contact determination is as follows: ； The physical loss function is: ； Among them, the contact force loss item ; Indicates the acceleration of the center of mass. Represents ground reaction force; loss of centroid consistency term = ; The position of the centroid obtained from the attitude estimation. The centroid location estimated from IMU data.