Human body posture estimation method and device using diffusion model reconstruction

Abstract

The invention belongs to the technical field of human body posture recognition, and relates to a human body posture estimation method using diffusion model reconstruction, which comprises the following steps: acquiring data sets, integrating the data sets, uniformly converting the data sets to an SMPL coordinate system to form a fusion input vector, and constructing local input vectors of a left ear and a right ear; embedding the local input vectors of the left ear and the right ear into a feature space of a specific dimension, and stacking to obtain embedded feature vectors of the left ear and the right ear; the embedded feature vectors of the left ear and the right ear are sequentially input into a time modeling module and a space modeling module to obtain output after interaction of two channels, then the output of the two channels is spliced and mapped into a unified space-time fusion feature, and a feature sequence is output after all time steps are stacked; and inputting the feature sequence into a de-noising diffusion probability model by adopting the de-noising diffusion probability model, and obtaining an output model after forward diffusion and backward diffusion. The frame can complete space-time modeling and reconstruction of human body postures only by depending on the two IMU sensors worn on the two ears.

Description

Technical Field

[0001] This invention belongs to the field of human pose recognition technology, and in particular to a human pose estimation method reconstructed using a diffusion model. Background Technology

[0002] Significant progress has been made in recent years in existing technologies for 3D motion reconstruction using only IMUs. Some technical solutions propose a method for human motion reconstruction using only six IMUs, achieving satisfactory accuracy; others introduce deep learning methods, using bidirectional recurrent neural networks to estimate human pose from six IMUs, but their methods do not estimate global translation in 3D space.

[0003] Given this challenge, several recent works have proposed innovative solutions to address the limitations of traditional methods: HMD-Poser proposes a scalable whole-body estimation method that combines HMD and wearable IMU, but its complex IMU+HMD setup makes data processing very difficult; DiffusionPoser can reconstruct human motion in real time from any IMU configuration at a rate of 25Hz for online motion reconstruction, however, as a generative model, its performance deteriorates with fewer IMUs, resulting in highly unstable results; DynaIP uses sparse IMUs for real-time human pose estimation, attempting to find the optimal IMU position on the body to achieve the best pose estimation, but it does not estimate the global translation of the body, making it unable to represent the body's motion process. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a human pose estimation method based on a diffusion model. This method successfully reconstructs natural, continuous, and structurally sound three-dimensional human movements by introducing a symmetric head motion representation, a binaural-driven spatiotemporal feature learning network, and a diffusion-based human pose generation model with a temporal optimization module.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In the first approach, the human pose estimation method reconstructed using a diffusion model includes: Obtain the dataset, integrate the dataset and transform it into the SMPL coordinate system to form a fused input vector, and construct the local input vectors for the left and right ears; The local input vectors of the left and right ears are embedded into a feature space of a specific dimension and stacked to obtain the embedded feature vectors of the left and right ears. The embedded feature vectors of the left and right ears are input into the temporal modeling and spatial modeling modules respectively to obtain the output after the interaction of the two channels. Then, the outputs of the two channels are concatenated and mapped into a unified spatiotemporal fusion feature. After stacking all time steps, the feature sequence is output. A denoising diffusion probability model is adopted. The feature sequence is input into the denoising diffusion probability model, and the output model is obtained after forward diffusion and reverse diffusion.

[0006] Preferably, the acquired dataset includes head rotation posture data, head rotation angular velocity data, and acceleration data of the left and right ears.

[0007] Preferably, embedding the local input vectors of the left and right ears into a feature space of a specific dimension includes: Head rotation and rotational angular velocity are embedded into a 64-dimensional representation space through two independent linear layers; the linear accelerations of the left and right ears are projected into a 128-dimensional feature space. The embedded features of the left and right ears are stacked according to the ear channel dimension to obtain a tensor-form dual-channel local feature representation F. t .

[0008] Preferably, the step of inputting the embedded feature vectors of the left and right ears into the temporal modeling and spatial modeling modules sequentially includes: Customized modules using dual-channel sparse input for the left and right ears are used, with each module including a temporal modeling submodule and a spatial modeling submodule.

[0009] As a preferred option, Using a separate LSTM module, the dual-channel local feature representation F t The data are fed into an LSTM module to obtain the output time feature sequence H. t ; The time feature sequence H output by the LSTM module t The spatial modeling process, fed into a Transformer-based encoder, is as follows: ; Z t This represents the spatial interaction between the two channels in frame t. The two channel outputs of each frame are concatenated and mapped to a unified spatiotemporal fusion feature: ; After stacking all time steps, the final TSFL output feature sequence is obtained. .

[0010] Preferably, in the Transformer-based encoder, the sequence length of the Transformer is reduced to 2 to reduce computational overhead.

[0011] Preferably, after obtaining the output model through forward and reverse diffusion, the method further includes: In both forward and backward diffusion, a GRU-based RefineNet module is introduced after each diffusion step to enhance the temporal correlation between frames.

[0012] In the second embodiment, a computer-readable storage medium stores a computer program, characterized in that, when executed by a processor, the computer program implements the human pose estimation method reconstructed using a diffusion model as described in the first embodiment.

[0013] In the third embodiment, a human pose estimation device based on diffusion model reconstruction includes a memory, a processor, and a computer program stored in the memory and running on the processor. The feature is that when the processor executes the computer program, it implements the human pose estimation method based on diffusion model reconstruction described in the first embodiment.

[0014] The beneficial effects of using this invention are: This invention proposes a fusion mechanism that unifies binaural IMU information, solves the problem of inconsistency between left and right ear data in spatial coordinates, and enhances the stability of head motion expression.

[0015] Secondly, a lightweight binaural spatiotemporal feature learning network was constructed, which combined LSTM and Transformer structures to achieve effective modeling of the dynamic posture evolution process.

[0016] Finally, this embodiment introduces an MLP-based diffusion pose generation framework and embeds a GRU RefineNet module, which significantly improves the continuity and naturalness of the generated actions. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating the overall framework of the human pose estimation method reconstructed using a diffusion model, as described in this invention.

[0018] Figure 2 This is a visualization comparison of the method of the present invention with AvatarPoser, HMDPoser, and real data. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this technical solution clearer, the following detailed description, in conjunction with specific embodiments, further illustrates this technical solution. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of this technical solution.

[0020] This invention uses BaseSet as the experimental dataset, which integrates multiple subsets of the AMASS framework, including CMU, BMLrub, and HDM05, covering a rich variety of daily actions and expressive behaviors. BaseSet is divided into training and validation sets in a 9:1 ratio to ensure the diversity and generalization ability of the evaluation.

[0021] Unlike traditional methods that rely on multiple sensors or visual information, this embodiment proposes the Ear2DiffP framework. This framework only requires two IMU sensors worn in both ears to complete the spatiotemporal modeling and reconstruction of human posture, significantly reducing the invasiveness and deployment complexity of the system. The overall framework of this patent is as follows: Figure 1 As shown. Specifically: I. Data Fusion Within this framework, this embodiment proposes an efficient input representation method to construct a spatiotemporally consistent human motion input vector Xt from sparse inertial signals acquired by a binaural IMU, which drives the subsequent full-body posture reconstruction model. Specifically, this embodiment acquires rotation angle, rotation angular velocity, and linear acceleration signals from binaural IMU sensors. The head rotation posture...

[0022] and rotational angular velocity

[0023] The average results of the two-ear rotations are integrated and uniformly transformed into the SMPL coordinate system. This embodiment uses 6D representation to ensure the continuity of the rotation representation and its ease of learning.

[0024] For acceleration signals, this embodiment first performs gravity component removal processing on the raw data to improve its ability to represent the user's actual motion trends. Subsequently, this embodiment aligns the accelerations of the left and right ears to the SMPL coordinate system, denoted as... .

[0025] This embodiment uses vector concatenation to construct the fused representation. The final fused input vector is as follows:

[0026] Furthermore, to further model the collaborative relationship between the local dynamics of the left and right ears and the overall head posture, local feature representations of the left and right ears were constructed. The head rotation and angular velocity information were concatenated with the acceleration of each ear to form the local input vectors of the left and right ears.

[0027] The core motivation behind this design is that while overall body posture is primarily determined by the overall movement trend, local areas (such as the location of the left and right ears) exhibit fine-grained differences when performing actions in different directions (such as turning the head, tilting slightly, or trembling). By jointly modeling the global dynamic information of the head with the local linear acceleration features of the ears, the network can not only learn the global spatiotemporal structure of the whole-body movement but also retain the local micro-motion changes of key sensory points, thereby improving the model's sensitivity to detailed postures and the accuracy of generation.

[0028] Furthermore, this architecture enhances the system's ability to model asymmetric movements (such as unilateral offsets and single-arm motion drives), enabling the capture of diverse and natural motion patterns even with sparse sensor configurations. This global-local fusion feature modeling strategy provides a more expressive input representation foundation for subsequent pose reconstruction and motion generation.

[0029] II. Spatiotemporal Feature Fusion Feature embedding encoding: To enhance the model's ability to express spatiotemporal dynamic features, the local input vector is first processed. and Embedding encoding is performed to construct feature vectors with high-dimensional semantic representation, which serve as the basic input for subsequent spatiotemporal modeling modules.

[0030] Specifically, for left ear input , First, rotate the head angle. and rotational angular velocity It is embedded into a 64-dimensional representation space through two independent linear layers; while the linear acceleration of the left ear The features are then projected into a 128-dimensional feature space. The three sets of features are concatenated to form the embedded feature vector of the left ear input: Similarly, for right ear input The right ear embedding feature vector is obtained by processing it using an embedding module with the same structure: .

[0031] Finally, the embedded features from the left and right ears are stacked according to the ear channel dimension to form a tensor-based two-channel local feature representation, which is used for subsequent spatiotemporal fusion modeling.

[0032] The reason for mapping rotation angle and rotation angular velocity to 64 dimensions is that this information is mainly used to characterize the global dynamic trend of the head, providing structural and directional support in full-body posture modeling. Therefore, a medium dimension can be used to represent its continuous changes. Linear acceleration, on the other hand, reflects the motion intensity and fine-grained variation characteristics of the local area of ​​the ear. This has a stronger information density for characterizing transient movements (such as head tilting, shaking, and asymmetrical postures). Therefore, a 128-dimensional embedding is used to capture more potential spatiotemporal semantics, which helps to enhance the model's micro-motion perception ability and posture generation accuracy.

[0033] This structure not only preserves the independence of local perceptual features of the left and right ears, but also provides a good structural foundation for subsequent cross-channel information interaction and fusion, providing a more distinctive and expressive high-order input representation for whole-body posture modeling.

[0034] Binaural spatiotemporal feature network: After high-dimensional feature embedding, the original input remains isolated in the temporal dimension and lacks explicit modeling of different components in the spatial dimension. To simultaneously model temporal dynamics and spatial structure, a lightweight binaural spatiotemporal feature fusion network (binaural spatiotemporal feature network) is proposed to efficiently capture the spatiotemporal correlation of input features. This embodiment features customized structural optimization for only dual-channel sparse input (i.e., the left and right ear channels), avoiding redundant computation for inputs with more than 8 channels, greatly improving computational efficiency and deployment flexibility, and significantly enhancing the collaborative modeling capability for ear micro-movements and overall head dynamics. Specifically, the network consists of N = 2 stacked identical modules, each containing two sub-modules: a temporal modeling sub-module and a spatial modeling sub-module.

[0035] In the temporal modeling stage, two independent LSTM modules are introduced to learn the motion evolution of the left and right channels in the time dimension. For the temporal modeling submodule in each TSFL block, two independent unidirectional LSTM networks (corresponding to the input feature sequences of the left and right ears, respectively) are used, effectively preserving key temporal dependencies and facilitating the capture of motion changes over longer time scales. In this scheme, from F... t Acquire features from the left and right ears and send them to the relevant systems respectively. and The LSTM output is:

[0036] In the spatial modeling stage, a Transformer-based encoder structure is used to model the spatial interaction between the left and right ears. Thanks to the LSTM module's preprocessing of temporal information, the Transformer encoder only needs to model the spatial relationship between two channels within each frame, thus reducing the Transformer sequence length M to 2, significantly reducing computational overhead and making it more suitable for resource-constrained deployment environments. As input to the Transformer encoder, the spatial modeling process is as follows:

[0037] Z t This represents the spatial representation of the two channels in frame t.

[0038] Finally, the two channel outputs of each frame are concatenated and mapped to a unified spatiotemporal fusion feature:

[0039] After stacking all time steps, the final TSFL output feature sequence is obtained:

[0040] III. Ear2DiffP Diffusion model: A denoised diffusion probability model (DDPM) is used as the generation framework, modeling the pose generation task as a backward diffusion process that gradually recovers the target sequence from Gaussian noise. Let the target's full-body pose sequence be... Its data comes from real data distribution The forward diffusion process can be represented as:

[0041] in, The retention rate hyperparameter of the t-th diffusion non-diffusion determines the degree to which attitude information is covered by Gaussian noise. After... After step diffusion, the noise posture samples are finally obtained according to the standard normal distribution. .

[0042] Backdiffusion model The learning process generates realistic poses stepwise from Gaussian noise, in the following form:

[0043] in, The spatiotemporal features from the TSFL network output are shown as conditional information used to guide generation. It is a fixed variance scheduling function, following the derivation of DDPM, and uses a denoised mean function. Expanded to:

[0044] Among them, αt = α1·α2...·αt, Represents a noise prediction network. It is the cumulative retention rate. The model is trained following the given noise pose. Predicting clean human pose in steps t Instead of residual noise The loss function is:

[0045] in, Representation Model The output.

[0046] MLP architecture integrating GRU RefineNet: In this embodiment, a GRU-based RefineNet module is introduced after each diffusion step to enhance the temporal correlation between frames. Specifically, the basic MLP network contains 12 diffusion stages, each consisting of multiple fully connected layers, SiLU activation, LayerNorm, and 1D convolutions with a kernel size of 1. Temporal embeddings are repeatedly injected into each layer through linear transformations to preserve temporal dependencies.

[0047] To improve the accuracy of the generated sequences, a lightweight GRU module is appended after each stage of the MLP to perform short-term time modeling based on the output of that stage. This design allows the model to maintain computational efficiency while exhibiting stronger generative coherence. The final denoised output is:

[0048] in, motion sequence The corresponding latent feature representation obtained after the fully connected layer, i.e.

[0049] This is an embedded representation of the output features of the binaural spatiotemporal feature network.

[0050] Example 1 and Comparative Example

[0051] This embodiment evaluates the model’s performance in three aspects: posture accuracy, position recovery capability, and motion smoothness. The evaluation criteria refer to mainstream work in the field of human body weight [14, 10, 15].

[0052] Attitude-related metrics: Mean Joint Rotation Error (MPJRE): Measures the average angular deviation between the predicted result and the actual joint rotation, in degrees.

[0053] Location-related indicators: Mean Joint Position Error (MPJPE): The average Euclidean distance between the predicted and actual joint positions, in centimeters.

[0054] Hand position error (H-PE): The average position error of the left and right hands, used to assess the accuracy of upper limb movement.

[0055] Upper body (U-PE) / lower body (L-PE) position error: These measures the prediction error for the torso and legs, respectively.

[0056] Foot-PE position error: measures the average position error of the entire foot.

[0057] Heel-PE position error: measures the error in predicting the position of the heel.

[0058] Toe-PE position error: measures the error in predicting the position of the toe.

[0059] Smoothness-related indicators: Mean Joint Velocity Error (MPJVE): The difference between the predicted and actual velocities.

[0060] Jitter: The average value of the first derivative of joint acceleration, measured in units of... This reflects the smoothness of the movement.

[0061] Skate error: used to evaluate gait continuity, representing the lateral drift of the supporting foot in a continuous gait. The smaller the value, the more natural the movement.

[0062] To comprehensively evaluate the performance of this patent, this embodiment was compared with existing methods HMDPoser[6], AvatarPoser

[14] , Agol

[10] , and IMPPoser[5], covering multiple key position errors and motion dynamics related indicators. Table 1 shows the performance of each method on each indicator.

[0063] Table 1 Comparison with other methods

[0064] The results show that the method in this embodiment achieves the best performance in terms of metrics. It achieves the lowest error in the two core metrics for measuring the localization accuracy of key points in the upper and lower body, U-PE and L-PE, which are significantly better than other methods. At the same time, it also achieves the best performance in foot key point metrics such as Foot-PE, Heel-PE and Toe-PE. This indicates that although it only relies on the sparse input of binaural IMUs, the model in this embodiment can still effectively capture the dynamic structure of the human lower limbs and shows strong stability and generalization ability in complex movements such as walking and squatting.

[0065] To further evaluate the actual performance of this patent, in Figure 2 The document presents a visual comparison of the method (Ours) in this embodiment with AvatarPoser, HMDPoser, and real data (GT), covering multiple typical action sequences. All methods were trained on the same dataset and evaluated on the same test samples.

[0066] Compared to existing solutions, this approach firstly designs a unified binaural IMU information fusion mechanism to address the inconsistency between left and right ear data in spatial coordinates, thereby enhancing the stability of head motion representation. Secondly, it constructs a lightweight binaural spatiotemporal feature learning network, combining LSTM and Transformer structures to effectively model the dynamic pose evolution process. Finally, this approach introduces an MLP-based diffusion-style pose generation framework and embeds a GRU RefineNet module, significantly improving the continuity and naturalness of generated actions.

[0067] The above content is only a preferred embodiment of the present invention. For those skilled in the art, many changes can be made in the specific implementation and application scope based on the ideas of the present invention. As long as these changes do not depart from the concept of the present invention, they all fall within the protection scope of this patent.

Claims

1. A method for estimating human pose using a diffusion model, characterized by: include: Obtain the dataset, integrate the dataset and transform it into the SMPL coordinate system to form a fused input vector, and construct the local input vectors for the left and right ears; The local input vectors of the left and right ears are embedded into a feature space of a specific dimension and stacked to obtain the embedded feature vectors of the left and right ears. The embedded feature vectors of the left and right ears are input into the temporal modeling and spatial modeling modules respectively to obtain the output after the interaction of the two channels. Then, the outputs of the two channels are concatenated and mapped into a unified spatiotemporal fusion feature. After stacking all time steps, the feature sequence is output. A denoising diffusion probability model is adopted. The feature sequence is input into the denoising diffusion probability model, and the output model is obtained after forward diffusion and reverse diffusion.

2. The human pose estimation method reconstructed using a diffusion model according to claim 1, characterized in that: The acquired dataset includes head rotation posture data, head rotation angular velocity data, and acceleration data of the left and right ears.

3. The human pose estimation method reconstructed using a diffusion model according to claim 2, characterized in that: The process of embedding the local input vectors of the left and right ears into a feature space of a specific dimension includes: Head rotation and rotational angular velocity are embedded into a 64-dimensional representation space through two independent linear layers; the linear accelerations of the left and right ears are projected into a 128-dimensional feature space. The embedded features of the left and right ears are stacked according to the ear channel dimension to obtain a tensor-form dual-channel local feature representation F. t .

4. The human pose estimation method reconstructed using a diffusion model according to claim 1, characterized in that: The step of inputting the embedded feature vectors of the left and right ears into the temporal modeling and spatial modeling modules sequentially includes: Customized modules using dual-channel sparse input for the left and right ears are used, with each module including a temporal modeling submodule and a spatial modeling submodule.

5. The human pose estimation method reconstructed using a diffusion model according to claim 1, characterized in that: Using a separate LSTM module, the dual-channel local feature representation F t The data are fed into an LSTM module to obtain the output time feature sequence H. t ; The time feature sequence H output by the LSTM module t The spatial modeling process, fed into a Transformer-based encoder, is as follows: ； Z t This represents the spatial interaction between the two channels in frame t. The two channel outputs of each frame are concatenated and mapped to a unified spatiotemporal fusion feature: ； After stacking all time steps, the final TSFL output feature sequence is obtained. 。 6. The human pose estimation method reconstructed using a diffusion model according to claim 5, characterized in that: In the Transformer-based encoder, the sequence length of the Transformer is reduced to 2 to reduce computational overhead.

7. The human pose estimation method reconstructed using a diffusion model according to claim 1, characterized in that: After obtaining the output model through forward and reverse diffusion, the process further includes: In both forward and backward diffusion, a GRU-based RefineNet module is introduced after each diffusion step to enhance the temporal correlation between frames.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the human pose estimation method reconstructed using a diffusion model according to any one of claims 1 to 7.

9. A human pose estimation device based on diffusion model reconstruction, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the human pose estimation method reconstructed using a diffusion model according to any one of claims 1 to 7.

Images