Whole body motion generation method based on sparse tracking

Abstract

The present application relates to a kind of whole body motion generation method based on sparse tracking input, comprising: (1) collection sparse tracking input, and acquisition Gaussian noise conforming to human motion distribution;(2) obtain pre-trained human anatomy diffusion model, the human anatomy diffusion model, including for carrying out smooth prediction smooth human anatomy network, and for accurate prediction accurate human anatomy network;(3) based on the sparse tracking input and the Gaussian noise conforming to human motion distribution, by the smooth human anatomy network and the accurate human anatomy network carry out parallel motion regression;(4) based on the sparse tracking input and the result of parallel motion regression, by the smooth human anatomy network and the accurate human anatomy network carry out alternate motion refinement, generate the output data of whole body motion.The present application is with higher accuracy and smoothness while processing sparse tracking input and output corresponding whole body motion.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and in particular to a method for generating whole-body motion based on sparse tracking. Background Technology

[0002] In contemporary artificial intelligence (AI) research, a key challenge lies in the nuanced understanding and accurate simulation of human movement and behavior. Specifically, in the field of motion generation, research on deriving human motion from various patterns, such as text-to-motion methods like T2M-GPT, has attracted considerable interest. However, the human motion generated by these methods often fails to meet user expectations, particularly in terms of accuracy. Methods using RGB cameras or single head-mounted fisheye cameras are especially susceptible to occlusion and truncation, which compromise the smoothness and realism of the generated human motion. Therefore, these methods may not perfectly match the natural movement expected by people.

[0003] Recently, a growing number of researchers have focused on using sparse tracking input to synthesize full-body motion poses. In the field of human-centered artificial intelligence, particularly in cognition and interaction, the use of head-mounted displays and other spatial computing devices (such as AR and VR technologies) to generate motion is gaining increasing attention. Compared to traditional inertial measurement unit (IMU) motion capture and six-IMU-based methods, capturing full-body poses using fewer trackers offers a less invasive and more cost-effective solution. While spatial computing devices can track the rotation and position of the body and hands, they often lack comprehensive capture of full-body motion, especially the lower body.

[0004] Advances in motion generation based on sparse tracking, such as the diffusion-based model developed by AGLOL, have made significant contributions to the field. However, these methods cannot fully capture the complexity of human motion. In particular, they can lead to accumulated errors at joints, reducing the accuracy and smoothness of the generated motion.

[0005] It is particularly important to study how to overcome the problems of joint errors and unnatural motion that have occurred in previous methods. Therefore, based on sparse tracking, there is an urgent need for a model to improve the accuracy and smoothness of motion generation. Summary of the Invention

[0006] To address the aforementioned problems, the purpose of this invention is to provide a method for generating full-body motion based on sparse tracking input. This method can process sparse tracking input and output corresponding full-body motion, while also exhibiting higher accuracy and smoothness.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, this application provides a method for generating whole-body motion based on sparse tracking input, the method comprising:

[0009] (1) Collect sparse tracking input and collect Gaussian noise that matches the distribution of human motion;

[0010] (2) Obtain a pre-trained human anatomy diffusion model, the human anatomy diffusion model including a smooth human anatomy network for smooth prediction and an accurate human anatomy network for accurate prediction.

[0011] (3) Based on the sparse tracking input and the Gaussian noise that conforms to the human motion distribution, parallel motion regression is performed through the smooth human anatomy network and the accurate human anatomy network.

[0012] (4) Based on the results of the sparse tracking input and parallel motion regression, the smooth human anatomy network and the accurate human anatomy network are used to perform alternating motion refinement to generate output data of whole-body motion.

[0013] In one implementation of this application, sparse tracking input is obtained based on a head-mounted display and left and right hand tracking.

[0014] In one implementation of this application, the human anatomy diffusion model includes a latent space mapping module, an iterative feature enhancement module, a temporal feature pyramid module, and a hierarchical motion refinement module.

[0015] The latent space mapping module extracts input features and maps the input features to a unified latent space;

[0016] The iterative feature enhancement module enhances the mapped features;

[0017] The temporal feature pyramid module integrates multi-scale spatiotemporal motion features into the enhanced features and outputs a coarse prediction.

[0018] The layered motion refinement module performs layered processing on the coarse prediction and outputs the final prediction result.

[0019] In one implementation of this application, the iterative feature enhancement module uses an iterative method to gradually enhance motion features; the number of iterations is denoted as M, and the output of the j-th iteration is denoted as... Its iterative formula is expressed as:

[0020]

[0021] Where, f I 0 The initial value is f L After all iterations are completed, the final output is: This represents enhanced motion characteristics.

[0022] In one implementation of this application, during the processing of the time-series feature pyramid module, S = {s1, ..., s...} L} is the set of L downsampling factors, for The i-th scaling factor s in i First, for f I Press s i The scaling factor is used to downsample; then these downsampled features are processed, and intermediate features are output. Subsequently, using MLP to When the sequence length increases to N, the corresponding features are denoted as... And by aggregating these multi-scale outputs, the final output f is obtained. T It is the normalized sum of features at all scales:

[0023]

[0024] In one implementation of this application, the layered motion refinement module refines the coarse prediction values. Divided into three different parts: Where K = K t +K a +K l Each part corresponds to a different location, namely the torso, arms, and legs;

[0025] Then these three parts are encoded:

[0026]

[0027]

[0028]

[0029] Subsequently, a layered approach was used to refine the three body parts separately, as follows:

[0030]

[0031]

[0032]

[0033] Wherein, FC represents a fully connected layer;

[0034] Finally, and By merging, a refined full-body pose can be obtained. t-1 .

[0035] In one implementation of this application, the parallel motion regression processing includes:

[0036] The noise motion posture x t The observed joint features c and time step t are respectively input into the smooth human anatomy network m. s and accurate human anatomy network m a This yields two corresponding outputs. and Their formulas are:

[0037]

[0038]

[0039] in and All are full-body poses; then, the poses of the upper body and lower body are extracted separately. and They were then combined into a full-body pose. t-1 .

[0040] In one implementation of this application, the alternating refinement process is as follows:

[0041] p(x t-1 |x t ,c):=N(x t-1 ;m a (x t ,c,t),σ t 2 I),

[0042]

[0043] The present invention has the following advantages due to the adoption of the above technical solutions:

[0044] (1) The invention innovatively combines human anatomy and diffusion model, which makes the model focus on the causal relationship of movement and one-dimensional time properties, effectively alleviating the problem of cumulative error in joint position.

[0045] (2) The invention can handle complex motions well, accurately capture body angles and balance, and show stronger alignment in motion prediction. The proposed human anatomy diffusion significantly improves the performance of motion generation, with significantly improved smoothness and accuracy.

[0046] (3) The method proposed in the invention can run in real time, providing an important advance for creating realistic and responsive virtual avatars in real-world applications. Attached Figure Description

[0047] Figure 1 Overall flowchart of an embodiment of the present invention;

[0048] Figure 2 This is the basic network structure diagram of the present invention;

[0049] Figure 3 This is a diagram of the human anatomical network structure of the present invention;

[0050] Figure 4 This is a network structure diagram of the hierarchical motion refinement module of the present invention;

[0051] Figure 5 This is a schematic diagram of the detection results in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0053] To address the problem that existing technologies suffer from reduced accuracy and smoothness in generated motion due to accumulated errors at joints, this application provides a method for generating full-body motion based on sparse tracking input. The method includes:

[0054] (1) Collect sparse tracking input and collect Gaussian noise that matches the distribution of human motion;

[0055] (2) Obtain a pre-trained human anatomy diffusion model, the human anatomy diffusion model including a smooth human anatomy network for smooth prediction and an accurate human anatomy network for accurate prediction.

[0056] (3) Based on the sparse tracking input and the Gaussian noise that conforms to the human motion distribution, parallel motion regression is performed through the smooth human anatomy network and the accurate human anatomy network.

[0057] (4) Based on the results of the sparse tracking input and parallel motion regression, the smooth human anatomy network and the accurate human anatomy network are used to perform alternating motion refinement to generate output data of whole-body motion.

[0058] In step 1), we treat the motion prediction task as a conditional generation problem, with sparse tracking being the condition. Assume the whole-body pose distribution in the dataset is p. motion During the forward diffusion process, we will find clean data distribution p motionGaussian noise is gradually added until the output distribution approximates an isotropic Gaussian distribution. Taking the sample data pair (x0, c) as an example, from clean data x0 to x... T The diffusion process is defined as

[0059]

[0060] in hyperparameter α t It is a predefined positive constant. As T→∞, x T It tends towards an isotropic Gaussian distribution. The back-diffusion process is conditioned on sparsely tracking the input c. We train a diffusion model p. θ To predict the results after noise is eliminated during the diffusion process.

[0061] In step (2), the human anatomy network consists of four parts: latent space mapping, iterative feature enhancement, temporal feature pyramid, and hierarchical motion refinement. Latent space mapping maps input features to a unified latent space for subsequent processing. Iterative feature enhancement then takes the coarse features from the latent space mapping as input and enhances them to obtain a better representation. To further improve the smoothness and consistency of the generated motion, a temporal feature pyramid is used to integrate multi-scale spatiotemporal motion features and output a coarse prediction. Finally, inspired by human anatomy, hierarchical motion refinement employs a hierarchical architecture to refine the obtained predictions, making the motion more natural and accurate. The detailed architecture of each module is as follows.

[0062] Latent space mapping module

[0063] At time step t, first apply a fully connected layer to Noise-induced motion posture x t The observed joint features c are mapped to a unified latent space. Then, we can obtain the corresponding latent features. and Where D represents the dimension of the latent feature. Subsequently, the diffusion step size t is converted into a step-size embedding vector f. t , and f x and f c After connection, input the basic network structure for further processing. Note that f... t This process involves repeatedly injecting the information into each underlying network structure to prevent information loss during time-step embedding. We call the output of this step f. L For a detailed description of the basic network structure, please refer to [link / reference]. Figure 2 .

[0064] Iterative Feature Enhancement Module

[0065] Since the feature interactions between each frame in the latent space mapping are insufficient, we employ iterative feature enhancement to augment the motion features f.L To obtain a more robust representation, denoted as f I Specifically, we use an iterative method to gradually enhance the motion features. The number of iterations is denoted as M, and the output of the j-th iteration is denoted as... Taking the j-th iteration as an example, the refinement result of this iteration can be expressed by the mathematical formula as follows:

[0066]

[0067] in, The initial value is f L After all iterations are completed, the final output is: This represents enhanced motion characteristics. This iterative improvement process, achieved through the underlying network structure, is crucial for realizing accurate and reliable whole-body motion prediction.

[0068] Temporal Feature Pyramid Module

[0069] Considering that different actions correspond to different time scales, we apply a temporal feature pyramid to further improve the smoothness of the prediction. Let S = {s1, ..., s2} L Let} be a set of L downsampling factors. For The i-th scaling factor s in i First, for f I Press s i The scaling factor is used for downsampling. Then, the basic network structure is applied to process these downsampled features and output intermediate features. Subsequently, we used MLP to When the sequence length increases to N, the corresponding features are denoted as... To aggregate these multi-scale outputs, the final output f T It is the normalized sum of features at all scales:

[0070]

[0071] Using this multi-scale refinement process, we will f I Transform into more robust motion characteristics f T This effectively captures motion dynamics at different temporal resolutions and enhances overall smoothness. We will directly use f T The prediction method used is called smooth prediction. However, in our experiments, we found that smooth prediction tends to over-smooth the upper body posture, leading to unsatisfactory accuracy. Therefore, we propose precise prediction to ensure its accuracy. Figure 3 As shown, we use f I and f T To predict upper body posture separately and lower body posture Then we will and Combined, the complete body posture is obtained. Given f I It contains accurate upper body information, while f T This captures information at different time scales, thus the separate prediction method considers stationarity while ensuring accuracy.

[0072] Layered motion refinement module

[0073] Inspired by human anatomy, we employed a layered approach to refine motion. For example... Figure 4 As shown, we first present the coarse predicted values. Divided into three different parts: Where K = K t +K a +K l Each part corresponds to a different location: the torso, arms, and legs. We then encode each of these three parts separately:

[0074]

[0075]

[0076]

[0077] The three encoders have the same structure. Specifically, we use linear layers, a basic network structure, and another linear layer sequentially to extract motion features. Then, we employ a layered approach to refine the three body parts separately. The specific process is as follows:

[0078]

[0079]

[0080]

[0081] Here, FC stands for fully connected layer. Finally, we will and By combining these elements, a precise full-body posture can be obtained. t-1 This hierarchical and anatomically-based approach allows for a more detailed representation of human movement. By processing each body part individually and then recombining them, we can provide a more comprehensive and realistic description of whole-body movement. During training, the loss function used in the human anatomy network consists of two parts: intermediate supervision and final supervision.

[0082] L total =L coarse +L final .

[0083] Among them, L coarse It is the introduced intermediate supervision, and L final It is the L2 distance between the final prediction and the actual ground condition.

[0084] In step (3), as Figure 1 As shown, smooth prediction is beneficial for predicting smooth motions, but it leads to less than ideal prediction accuracy for the upper body. Conversely, accurate prediction can generate more precise predictions, but it cannot guarantee the smoothness of motion, especially for the lower body. To address the limitations of single methods, we propose a novel architecture based on human anatomical networks, namely human anatomical diffusion. Figure 1 As shown, it utilizes both accurate and smooth predictions to generate accurate and smooth forecasts. Specifically, the proposed architecture comprises two stages: parallel motion regression and alternating motion refinement.

[0085] In parallel motion regression, we refer to the human anatomy networks used for prediction using smooth prediction and precise prediction as smooth human anatomy networks (m). s ) and accurate human anatomy network (m a Considering the advantages of each method, we employ a dual-path approach to predict the upper and lower halves of the human body separately. Specifically, the noisy motion pose x... t The observed joint features c and time step t are respectively input into m s and m a Then, we can get two corresponding outputs. and Their formulas are:

[0086]

[0087]

[0088] in and All of these are full-body poses. Then, we extract the poses of the upper body and lower body separately. and They were then combined into a full-body pose. t-1 This stage is crucial for generating accurate and fluid predictions, paving the way for more detailed and realistic synthesis of human motion.

[0089] In step (4), to further improve the network performance, we adopted a sequential method to improve x. t Unlike the parallel method used in parallel motion regression, in this stage we apply m sequentially. a and m sThis is to gradually improve motion prediction. Specifically, the alternating refinement process is defined as follows:

[0090] p(x t-1 |x t ,c):=N(x t-1 ;m a (x t ,c,t),σ t 2 I),

[0091]

[0092] The alternating refinement process meticulously refines motion prediction through a sequential process. In experiments, this stage is crucial for improving the accuracy and naturalness of human motion synthesis.

[0093] In one application scenario, a multi-view 3D human pose estimation and tracking algorithm based on skeleton cluster tracking obtains a 2D human pose skeleton. Then, for each viewpoint, a planar stereo scanning method is used to perform depth regression on the 2D human skeleton. The depth information of each 2D joint is regressed using a human symmetric hole network and a joint symmetric hole network. Based on the depth information from each viewpoint, multiple 3D human poses are obtained, forming a skeleton pool. Skeleton clusters are obtained through a skeleton clustering algorithm, and a skeleton cluster tracking algorithm is applied to these skeleton clusters to obtain the final 3D human pose and ID information.

[0094] The specific implementation steps are as follows:

[0095] 1) In this embodiment, we used the CMU, BMLrub, and HDM05 datasets from the AMASS human motion dataset. The training and test set splitting criteria were consistent with those used in Avatarposer. Human pose was represented using the SMPL model, focusing on the global orientation of the root joints and the relative rotation of other joints.

[0096] 2) We trained the fluent and accurate human anatomy networks using Adam as the optimizer with a learning rate of 0.0003 and L2 loss as the loss function. We trained the networks for approximately 250,000 iterations using subsets of the AMASS dataset, including CMU, BMLrub, and HDM05.

[0097] 3) Using sparse tracking input and Gaussian noise that conforms to the distribution of human motion, parallel motion regression is performed through a smooth human anatomy network and an accurate human anatomy network.

[0098] 4) Using sparse tracking input and the output after parallel motion regression, the final three-dimensional human pose estimation points can be obtained by alternating motion refinement through a smooth human anatomy network and an accurate human anatomy network.

[0099] 5) In this embodiment, the computer executing the algorithm has an Intel i9 CPU, runs Ubuntu 18.04, uses CUDA version 11.2, and employs PyTorch, version 1.10.2, as the neural network framework. Parameters can be adjusted appropriately based on different performance requirements to achieve a balance between accuracy and speed, resulting in the final output.

[0100] In summary, this invention achieves real-time motion generation based on human anatomy diffusion and sparse tracking through the above steps. To verify the effectiveness and practicality of the proposed method, the overall results and cross-dataset results on the AMASS dataset are presented below. Table 1 shows the results of the example on the test set and AGROL.

[0101] Table 1 shows the overall results of each application method on the AMASS dataset, evaluated using the MPJRE, MPJPE, MPJVE, and Jitter metrics.

[0102] AvatarPoser 3.08 4.18 27.70 14.49 AGRoL 2.66 3.71 18.59 7.26 Fluent Human Anatomy Network 2.41 3.31 16.59 4.69 Accurate human anatomy network 2.40 3.18 16.42 7.35 Human Anatomy Diffusion 2.29 3.03 15.45 4.61

[0103] Table 1

[0104] Table 2 shows the cross-dataset results of each application method on AMASS, evaluated using the MPJRE, MPJPE, MPJVE, and Jitter metrics.

[0105]

[0106] Table 2

[0107] As can be seen from Table 1, compared with other methods for motion generation based on sparse tracking input, the improved algorithm proposed in this invention has better performance for human motion generation.

[0108] As shown in Table 2, this embodiment provides a significant improvement for the vast majority of test categories, demonstrating the effectiveness of the invention. Furthermore, Figure 5 The visual detection results also demonstrate the superior performance of the present invention.

[0109] In this application embodiment, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a computer device, implements the method described in this application embodiment.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0111] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0112] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating whole-body motion based on sparse tracking input, characterized in that, The method includes: (1) Collect sparse tracking input and collect Gaussian noise that conforms to the distribution of human motion; (2) Obtain a pre-trained human anatomy diffusion model, which includes a smooth human anatomy network for smooth prediction and an accurate human anatomy network for accurate prediction; wherein, the human anatomy diffusion model includes a latent space mapping module, an iterative feature enhancement module, a temporal feature pyramid module and a hierarchical motion refinement module; the latent space mapping module extracts input features and maps the input features to a unified latent space; the iterative feature enhancement module enhances the mapped features; the temporal feature pyramid module integrates multi-scale spatiotemporal motion features of the enhanced features and outputs a coarse prediction; the hierarchical motion refinement module performs hierarchical processing on the coarse prediction and outputs the final prediction result; During the processing, the time-series feature pyramid module records... yes The set of downsampling factors, for The first in Each proportional factor First of all according to The scaling factor is used to downsample; then these downsampled features are processed, and intermediate features are output. , The dimensions representing latent features; subsequently, MLP is used to... The sequence length increases to The corresponding features are denoted as And by aggregating these multi-scale outputs, the final output is... It is the normalized sum of features at all scales: ； The layered motion refinement module will refine the coarse prediction values. Divided into three different parts: , , ,in Each part corresponds to a different location, namely the torso, arms, and legs; then these three parts are encoded: Subsequently, a layered approach was used to refine the three body parts separately, as follows: in, Indicates a fully connected layer; Finally, , and By combining these elements, a refined full-body posture can be achieved. ; (3) Based on the sparse tracking input and the Gaussian noise that conforms to the human motion distribution, parallel motion regression is performed through the smooth human anatomy network and the accurate human anatomy network; (4) Based on the results of the sparse tracking input and parallel motion regression, the alternating motion refinement is performed through the smooth human anatomy network and the accurate human anatomy network to generate output data of whole-body motion.

2. The method for generating whole-body motion based on sparse tracking input according to claim 1, characterized in that, Sparse tracking input is obtained based on the head-mounted display and left and right hand tracking.

3. The method for generating whole-body motion based on sparse tracking input according to claim 1, characterized in that, The iterative feature enhancement module uses an iterative method to gradually enhance motion features; the number of iterations is denoted as... , No. The output of the next iteration is denoted as Its iterative formula is expressed as in, The initial value is After all iterations are completed, the final output is: , representing enhanced motion characteristics.

4. The method for generating whole-body motion based on sparse tracking input according to claim 1, characterized in that, The parallel motion regression processing procedure includes: Noise movement posture Observed joint features and time step Input into the fluent human anatomy network respectively and accurate human anatomy network This yields two corresponding outputs. and Their formulas are: in and All are full-body poses; then, the poses of the upper body and lower body are extracted separately. and Then they were combined into a full-body pose. .

5. The method for generating whole-body motion based on sparse tracking input according to claim 4, characterized in that, The process of refining the alternating motion is as follows: 。

Images