A holistic human motion generation method for manipulation perception

Abstract

The present invention discloses a method for generating overall human motion for manipulation perception, which relates to the field of virtual reality technology, and comprises the following steps: step 1, performing latent coding representation on human body and hand motions encoded from sparse tracking signals by constructing a conditional variational autoencoder; step 2, gradually reconstructing the noisy motion coding in combination with a sequence diffusion model to obtain the initial body and hand motion latent coding; step 3, extracting object manipulation features using a sequence control network, guiding DDPM to generate motion latent coding that integrates manipulation intentions, and finally generating the final overall human motion sequence that integrates manipulation semantics through a decoding module of CVAE. By introducing specific object manipulation representations, the present invention significantly improves the naturalness and coordination of the overall motion generated by virtual characters in a VR environment, and exhibits higher motion quality and interaction consistency during object interaction.

Description

Technical Field

[0001] The present invention belongs to the field of virtual reality technology, and in particular relates to a method for generating overall human body motions oriented to manipulation perception. Background Art

[0002] Virtual reality (VR) technology, a crucial bridge between the real and digital worlds, is driving the human-computer interaction experience towards a more natural and immersive experience. Generating realistic whole-body human motion, particularly those that encompass both body and hand movements, is key to simulating real-world user behavior in virtual environments. However, despite rapid advancements in visual display and spatial positioning, the motion control signals provided by current mainstream VR devices (such as the Meta Quest Pro, Apple Vision Pro, and PICO 4 Pro) are still very limited, making it impossible to fully track the user's entire body. This results in a significant information gap in whole-body motion generation.

[0003] At the same time, the different behaviors displayed by users when manipulating objects in VR scenes will significantly affect the overall human motion, further increasing the complexity of motion modeling, making the generation of high-quality overall human motion based on sparse tracking signals a challenging task. Currently, researchers mainly use various data-driven models to solve this problem, ranging from regression methods to probabilistic generation models. Although these methods have proposed a variety of feasible optimal motion estimation strategies, they ignore the constraints on the distribution of the latent motion space under the conditions of joint sparse motion control and manipulation content, resulting in the generated overall motion estimation range being too wide, making it difficult to accurately and efficiently estimate the body and hand movements simultaneously during manipulation. The resulting problems such as insufficient rationality of body posture and low frame rate of overall motion generation have, to a certain extent, limited the user's immersion and interaction quality in the virtual environment. Summary of the Invention

[0004] To achieve the above-mentioned object, the technical solution of the present invention is as follows: a method for generating holistic human body motions for manipulation perception, comprising the following steps:

[0005] Step 1: construct a conditional variational autoencoder to represent the human body and hand movements encoded from the sparse tracking signal.

[0006] Step 2: Combined with the sequential diffusion model, the noisy motion code is gradually reconstructed to obtain the initial body and hand motion latent code;

[0007] Step 3: Use the sequence control network to extract the object manipulation features, guide DDPM to generate the action latent code that integrates the manipulation intention, and finally generate the overall human action sequence that integrates the manipulation semantics through the decoding module of CVAE.

[0008] Preferably, a conditional variational autoencoder is constructed in step one, including: a body motion encoding module, a hand motion encoding module and a corresponding motion decoding module. The body and hand motion encoding modules encode human body and hand motions through sparse tracking signals, and output body and hand motion codes in the latent motion space respectively.

[0009] Preferably, a sequential diffusion model is constructed in step 2, including: body DDPM and hand DDPM. Given a sparse tracking signal and a noisy body latent code as input, the body DDPM reconstructs the initial body motion latent code. Given the initial body motion latent code, the sparse tracking signal and the noisy hand latent code, the hand DDPM reconstructs the initial hand motion latent code.

[0010] Preferably, in step three, a sequential control network is constructed, including a body control module and a hand control module. Given the object manipulation representation information, the body control module extracts the manipulation features and inputs them into the body DDPM; the output body latent code is input into the linear layer together with the initial body motion latent code to obtain the body motion latent code that integrates the manipulation information; similarly, the hand control module extracts the manipulation features, and the manipulation features are input into the hand control module together with the initial hand motion latent code to generate the hand motion latent code that integrates the manipulation information; finally, the optimized body and hand motion codes are input into the corresponding action decoding module constructed in step one to generate the final overall action that integrates the manipulation intention.

[0011] Preferably, step three includes object manipulation features to guide overall motion encoding of manipulation perception, including:

[0012] The user actively controls the action through the VR device's controller buttons. Once confirmed, the body and hand movements will be further refined.

[0013] The standard SMPL-X human skeleton model is used to represent the body and hands, which consists of 10 real values. For all joints A in SMPL-X, the local rotation of joint j is Defined in timestamp Aggregate functions on In the example, A(j) represents the ordered set of ancestor joints of joint j, and the global rotation of the joint is By multiplying the local rotation of all its ancestor joints i from the current joint j to the root joint To calculate, it is expressed by the following formula:

[0014] (1),

[0015] Sampling objects in left-handed and right-handed coordinate systems vertices, rather than directly using all vertices or uniform sampling, for interactive objects Any vertex on , its projection on the unit sphere for:

[0016] (2),

[0017] By making the interactive object All vertices in are projected onto the surface of the unit sphere for normalization, where is an interactive object The set of all vertices of represents the i-th point in V, represents the average value of the vertices in V;

[0018] Then sample uniformly on the unit sphere Vertices, and each sampling point is converted to the left / right-handed coordinate system, expressed as the polar coordinate form of the left-handed system and the right-handed system respectively ;

[0019] Define the distance feature vector , which provides continuous information about the proximity between the hand and the surface of the interaction object at timestamp t, and is uniformly sampled on the left and right hands at each timestamp t. points, and calculate the minimum distance between each sampling point and the nearest point on the surface of the interactive object, and use this to construct a distance feature vector .

[0020] Preferably, after constructing effective manipulation features, the manipulation features are used to generate overall actions with manipulation intentions through progressive manipulation guidance. The optimization process includes:

[0021] 1) Latent motion learning optimization: The optimization object is the conditional variational autoencoder obtained from the first stage of training. The body and hand motion codes are embedded in the latent space of the CVAE and serve as an important reference for generating the overall motion code in steps 2 and 3. In addition, the body and hand motion decoders in the CVAE rely on the overall motion code information to reconstruct the overall motion sequence;

[0022] 2) Train the sequential diffusion model from step 2 to learn complex body postures and hand details;

[0023] 3) Leverage the manipulation representation to train a sequence control network to optimize the overall action encoding.

[0024] Preferably, in 1), the object of optimization is the conditional variational autoencoder trained in step 1, wherein the motion codes of the human body and hands are embedded in the latent space of CVAE and serve as an important reference for generating the overall motion code in steps 2 and 3. In addition, the body and hand motion decoders in CVAE rely on this code information to reconstruct the overall motion sequence.

[0025] The loss function of CVAE is:

[0026] (3),

[0027] The first term is the KL divergence loss , which is used to constrain the potential space to move closer to the standard normal distribution, where represents the weight of KL divergence loss, represents the latent distribution predicted by the encoder With standard normal distribution The KL divergence between

[0028] The second term is the joint rotation reconstruction loss , used to constrain the predicted joint rotation to be closer to the real joint rotation, where represents the weight of the loss, represents the smooth-L1 loss function, and are the predicted and true joint rotation values, respectively;

[0029] The third term is the joint position loss calculated based on forward kinematics. , used to constrain the predicted joint positions to be closer to the true joint positions, where represents the weight of the loss, and is and Joint positions calculated via forward kinematics.

[0030] Preferably, the sequential diffusion model in step 2 is trained to learn complex body postures and hand details. When the input is a noisy body motion latent code and a sparse tracking signal, the loss function of the body diffusion model is:

[0031] (4),

[0032] in and Represent the predicted and real body movement latent codes respectively;

[0033] Next, we fix the parameters in the Body DDPM and train the hand diffusion model with the loss function:

[0034] (5),

[0035] in and Represent the predicted and true hand motion latent codes, respectively.

[0036] Preferably, in 3), the manipulation representation is used to train the sequence control network to optimize the overall motion encoding. After fixing the parameters of the body diffusion model and the hand diffusion model, the body control module is optimized using the following formula:

[0037] (6),

[0038] This loss function includes the rotation loss of the body joints and position loss of body joints Two items, including is the weight of the loss at that position, where and Represent the potential encoding of body movements with predicted and real fusion manipulation information, and Based on and The calculated joint positions;

[0039] The loss function of the hand control module is as follows:

[0040] (7),

[0041] in, and They represent the potential codes of hand movements that are the result of predicted and real fusion manipulation information, and Based on and The calculated hand joint positions. In addition to the hand joint rotation loss Loss of hand joint position In addition, it also includes the displacement loss of the key vertices of the hand, where represents the weight of the displacement loss, To manipulate the sampled hand vertex positions in the representation, Output of this method is the predicted vertex position of the hand motion.

[0042] Preferably, a lightweight online network is trained to fit the sequential diffusion model constructed in steps 2 and 3 and guided by the control network, so that the model can reason with fewer steps and predict a clear latent action representation.

[0043] In steps 2 and 3, for each DDPM, a 24-layer DiT denoiser is trained as the teacher model and distilled into a target 6-layer denoiser model. Since the teacher model and the target model share the same network architecture and feature dimensions, the target denoiser learns not only the final output of the teacher model but also the outputs of some of its intermediate layers. During the diffusion inference process in steps 2 and 3, five denoising steps are initially used; subsequently, a distillation process is used to reduce the number of denoising steps, and ultimately only one denoising step is required to complete the inference process.

[0044] Compared with the existing technology, the beneficial effects of the present invention are: by introducing specific object operation representations, the present invention significantly improves the naturalness and coordination of the overall movements generated by virtual characters in the VR environment, and exhibits higher movement quality and interaction consistency during object interaction. While ensuring movement quality, the generation speed is greatly improved, laying a solid foundation for the generation of semantically consistent interactive movements in more complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The execution process of the overall human body motion generation system of the present invention;

[0046] Figure 2 This is a snapshot of the overall movements of virtual humans of different body shapes manipulating a camera according to the present invention;

[0047] Figure 3 This is a distillation flow chart of the present invention;

[0048] Figure 4 This is a diagram showing the object manipulation features of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0050] Example: This example proposes a method for generating holistic human motion for manipulation perception. It constructs a conditional variational autoencoder (CVAE) to represent the body and hand motions encoded from sparse tracking signals as latent codes. It then uses a sequential diffusion model (DDPM) to gradually reconstruct the noisy motion codes to obtain the initial body and hand motion latent codes.

[0051] Furthermore, a sequence control network is used to extract object manipulation features, guiding the DDPM to generate latent encodings of actions that incorporate manipulation intent. Finally, the decoding module of the CVAE generates a holistic human action sequence that incorporates manipulation semantics. This method enables manipulation-aware holistic human action generation based on sparse tracking signals and object manipulation representations, improving the physical plausibility and temporal consistency of generated actions in virtual reality environments.

[0052] like Figure 1 As shown, the processing flow of this method mainly includes three steps:

[0053] Step 1: Construct a conditional variational autoencoder (CVAE), which includes a body motion encoding module, a hand motion encoding module, and a corresponding motion decoding module. The body and hand motion encoding modules are designed to accurately encode human body and hand motions using sparse tracking signals and output body and hand motion encodings in the latent motion space, respectively. The body and hand motion decoding modules are responsible for generating accurate human body and hand motions based on these latent codes.

[0054] Step 2: Construct a sequential diffusion model, including a body denoising diffusion probability model (DDPM) and a hand DDPM. Given the sparse tracking signal and the noisy body latent code as input, the body DDPM reconstructs the initial body motion latent code. Given the initial body motion latent code, the sparse tracking signal, and the noisy hand latent code, the hand DDPM reconstructs the initial hand motion latent code.

[0055] Step three: Construct a sequential control network, including a body control module and a hand control module. Given the object manipulation representation information, the body control module extracts manipulation features and inputs them into the body DDPM. The output body latent code is input into the linear layer together with the initial body motion latent code to obtain the body motion latent code that integrates the manipulation information. Similarly, the hand control module extracts manipulation features and inputs the manipulation features together with the initial hand motion latent code into the hand control module to generate the hand motion latent code that integrates the manipulation information. Finally, the optimized body and hand motion codes are input into the corresponding action decoding modules constructed in step one to generate the final overall action that integrates the manipulation intention.

[0056] According to the three steps above, the sparse tracking signal of the VR device and the object manipulation representation information in the virtual scene can be used to generate the final overall action that integrates the manipulation intention. To better describe the process of this method, the following are some definitions used in the article:

[0057] Conditional Variational Autoencoder (CVAE): A variational autoencoder (VAE) is a generative model that combines an autoencoder with a probabilistic graphical model. The encoder maps the input to a normal distribution in a latent space. After sampling, the decoder reconstructs the data and optimizes it by maximizing the Evidence Lower Bound (ELBO). CVAE introduces conditional information to the VAE, making the generation process controllable and suitable for tasks that require specific conditions.

[0058] Sparse Tracking Signals: refers to the limited tracking signals provided by the headset and controllers in VR devices. Compared with full-body joint sensors, they only capture the position of the head and hands.

[0059] Encoder / Decoder: The encoder compresses the high-dimensional input into a low-dimensional latent representation and extracts key features; the decoder reconstructs the latent representation into the original data space. The two constitute the autoencoder framework.

[0060] Latent Code: In an autoencoder or generative model, the low-dimensional vector output by the encoder embodies the key features and structure of the data, serving as an intermediate representation of the generation process. Its distribution characteristics influence the model's interpolation capabilities and generation quality, and are a key parameter controlling the generated content.

[0061] Diffusion Probabilistic Model (DDPM): A generative model based on Markov chains that gradually adds noise to the data through a forward process and then learns to gradually remove the noise through a backward process to restore the data distribution. DDPM generates high-quality samples by optimizing the noise prediction task and has become a mainstream generative method due to its stable training and excellent performance.

[0062] Model Distillation: A technology that optimizes model deployment through knowledge transfer. Its core is to transfer the knowledge of a complex teacher model to a lightweight student model. During training, the student model minimizes the output difference and task loss function with the teacher model, achieving model compression and performance approximation while significantly reducing computational complexity.

[0063] Object manipulation representation:

[0064] The goal of this method is to use the sparse signals provided by VR devices to generate overall human body movements that are consistent with manipulation perception. To achieve this goal, we first need an efficient method that can quickly extract and accurately represent the manipulation content. Therefore, this method proposes an effective object manipulation feature to guide the overall motion encoding of manipulation perception, such as Figure 4 As shown, it consists of the following contents:

[0065] Control state label at timestamp t , shape features of the avatar model SMPL-X , characteristics of the manipulated object And control action characteristics .

[0066] Control status label:

[0067] is a binary parameter; at timestamp t, the user is either in the controlling or non-controlling state. This method allows the user to actively control the VR device through the controller buttons. Once confirmed, the other three features mentioned above are used to further refine the body and hand movements, making the generated movements more natural and realistic during the control process.

[0068] Shape characteristics:

[0069] This method uses the standard SMPL-X human skeleton model to represent the body and hands, where is the body shape parameter of SMPL-X, which consists of 10 real values. For all joints A in SMPL-X, the local rotation of joint j is Defined in timestamp Aggregate functions on In the example, A(j) represents the ordered set of ancestor joints of joint j. The global rotation of the joint is By multiplying the local rotation of all its ancestor joints i from the current joint j to the root joint To calculate, it is expressed by the following formula:

[0070] (1),

[0071] In physically reasonable overall movements, virtual humans of different sizes will have different local rotations under the same control behavior and sparse tracking signals. Figure 2 It shows the overall action moments of virtual humans of different sizes manipulating the camera. Figure 2 Compared with the body shape of part (a), Figure 2 The thinner virtual human in part (b) has problems with hand and foot penetration.

[0072] Characteristics of the controlled object:

[0073] In order to generate accurate actions during the interaction, this method introduces object sampling points to represent the geometric shape and spatial position of the manipulated object. In order to reduce feature complexity and enhance the expression effect, this method samples the object in the left and right hand coordinate systems respectively. vertices, rather than using all vertices or uniform sampling. Any vertex on , its projection on the unit sphere for:

[0074] (2),

[0075] This step aims to All vertices in are projected onto the surface of the unit sphere for normalization, where is an interactive object The set of all vertices of represents the i-th point in V, Represents the average value of the vertices in V, that is, the geometric center.

[0076] Then sample uniformly on the unit sphere Vertices, and each sampling point is converted to the left / right-handed coordinate system, expressed as the polar coordinate form of the left-handed system and the right-handed system respectively .

[0077] Control action features:

[0078] In order to generate responsive and stable overall motion and promote smooth transition of motion during manipulation, this method defines the distance feature vector , which provides continuous information about the proximity between the hand and the surface of the object being interacted with at time stamp t. At each time stamp t, the vector is uniformly sampled on the left and right hands. points, and calculate the minimum distance between each sampling point and the nearest point on the surface of the interactive object, and use this to construct a distance feature vector .

[0079] Progressive control guidance optimization:

[0080] After constructing effective manipulation features, this method uses the manipulation features to generate overall actions with manipulation intentions through progressive manipulation guidance.

[0081] The optimization process consists of three steps:

[0082] Latent Motion Learning Optimization; Initial Holistic Motion Code Optimization; Manipulation-Aware Holistic Motion Code Optimization.

[0083] Latent motion learning optimization steps:

[0084] In this step, the conditional variational autoencoder (CVAE) trained in the first stage is optimized. The body and hand motion encodings are embedded in the CVAE's latent space and serve as an important reference for generating the overall motion encoding in the second and third stages. Furthermore, the body and hand motion decoders in the CVAE rely on this encoded information to reconstruct the overall motion sequence. To ensure the accuracy and naturalness of the motions, the optimization goal in this stage is to minimize the reconstruction error caused by inaccurate encodings.

[0085] The loss function of CVAE is:

[0086] (3),

[0087] The first term is the KL divergence loss , which is used to constrain the potential space to move closer to the standard normal distribution. represents the weight of KL divergence loss, represents the latent distribution predicted by the encoder With standard normal distribution The KL divergence between .

[0088] The second term is the joint rotation reconstruction loss , used to constrain the predicted joint rotation to be closer to the real joint rotation. represents the weight of the loss, represents the smooth-L1 loss function, and are the predicted and true joint rotation values, respectively.

[0089] The third term is the joint position loss calculated based on forward kinematics. , used to constrain the predicted joint positions to be closer to the true joint positions. represents the weight of the loss, and is and Joint positions calculated via forward kinematics.

[0090] Latent motion learning optimization steps:

[0091] This step trains the Sequential Diffusion Model from the second stage, enabling it to learn complex body postures and hand details. Given the input of noisy body motion latent codes and sparse tracking signals, the loss function of the Body Diffusion Model (BodyDDPM) is:

[0092] (4),

[0093] in and Represent the predicted and real body movement latent codes respectively.

[0094] Next, we fix the parameters in the Body DDPM and train the Hand Diffusion Model (Hand DDPM) with the loss function:

[0095] (5),

[0096] in and Represent the predicted and true hand motion latent codes, respectively.

[0097] Overall action optimization steps for control perception:

[0098] This step uses the manipulation representation to train the sequence control network to optimize the overall motion encoding. After fixing the parameters of the body diffusion model and the hand diffusion model (DDPM), the body control module is optimized using the following formula:

[0099] (6),

[0100] This loss function includes the rotation loss of the body joints and position loss Two items. and Represent the potential encoding of body movements with predicted and real fusion manipulation information, and Based on and The calculated joint positions.

[0101] Similarly, the loss function of the hand control module is as follows:

[0102] (7),

[0103] in, and They represent the potential codes of hand movements that are the result of predicted and real fusion manipulation information, and Based on and The calculated hand joint positions. In addition to the hand joint rotation loss Loss of hand joint position In addition, it also includes the displacement loss of the key vertices of the hand .in represents the weight of the loss, To manipulate the sampled hand vertex positions in the representation, Output of this method is the predicted vertex position of the hand motion.

[0104] Model Distillation: This step trains a lightweight online network to fit the sequential diffusion model constructed in stages two and three and guided by the control network. This allows the model to reason in fewer steps and predict clear latent action representations.

[0105] Specifically, in the second and third stages, for each DDPM, we train a 24-layer DiT denoiser as the teacher model and distill it into a 6-layer target denoiser model. Since the teacher model and target model share the same network architecture and feature dimensions, the target denoiser learns not only the final output of the teacher model but also the outputs of some of its intermediate layers.

[0106] In the second and third stages of diffusion inference, five denoising steps are initially used; then, Figure 3 The distillation process reduces the denoising step, and ultimately only one denoising step is required to complete the inference process.

[0107] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A method for generating overall human motion for manipulation perception, characterized in that: The following steps are involved: Step 1: construct a conditional variational autoencoder to represent the human body and hand movements encoded from the sparse tracking signal. Step 2: Combined with the sequential diffusion model, the noisy motion code is gradually reconstructed to obtain the initial body and hand motion latent code; Step 3: Use the sequence control network to extract the object manipulation features, guide DDPM to generate the action potential code that integrates the manipulation intention, and finally generate the overall human action sequence that integrates the manipulation semantics through the decoding module of CVAE. Step three includes object manipulation features to guide the overall motion encoding of manipulation perception, including: The user actively controls the action through the VR device's controller buttons. Once confirmed, the body and hand movements will be further refined. The standard SMPL-X human skeleton model is used to represent the body and hands, which consists of 10 real values. For all joints A in SMPL-X, the local rotation of joint j is Defined in timestamp Aggregate functions on In the example, A(j) represents the ordered set of ancestor joints of joint j, and the global rotation of joint j is By multiplying the local rotation of all its ancestor joints i from the current joint j to the root joint To calculate, it is expressed by the following formula: （1）， Sampling objects in left-handed and right-handed coordinate systems vertices, rather than directly using all vertices or uniform sampling, for interactive objects Any vertex on , its projection on the unit sphere for: （2）， By making the interactive object All vertices in are projected onto the surface of the unit sphere for normalization, where is an interactive object The set of all vertices of represents the i-th point in V, represents the average value of the vertices in V; Then sample uniformly on the unit sphere Vertices, and each sampling point is converted to the left / right-handed coordinate system, expressed as the polar coordinate form of the left-handed system and the right-handed system respectively ; Define the distance feature vector , which provides continuous information about the proximity between the hand and the surface of the interaction object at timestamp t, and is uniformly sampled on the left and right hands at each timestamp t. points, and calculate the minimum distance between each sampling point and the nearest point on the surface of the interactive object, and use this to construct a distance feature vector .

2. The method for generating overall human body motion for manipulation perception according to claim 1, characterized in that: In step 1, a conditional variational autoencoder is constructed, including: a body motion encoding module, a hand motion encoding module and a corresponding motion decoding module. The body motion encoding module and the hand motion encoding module encode human body and hand motions through sparse tracking signals, and output body and hand motion codes in the latent motion space respectively.

3. The method for generating overall human body motion for manipulation perception according to claim 1, characterized in that: In step 2, a sequential diffusion model is constructed, including body DDPM and hand DDPM. Given the sparse tracking signal and the noisy body latent code as input, the body DDPM reconstructs the initial body motion latent code. Given the initial body motion latent code, the sparse tracking signal and the noisy hand latent code, the hand DDPM reconstructs the initial hand motion latent code.

4. The method for generating overall human body motion for manipulation perception according to claim 1, characterized in that: In step 3, a sequential control network is constructed, including a body control module and a hand control module. Given the object manipulation representation information, the body control module extracts the manipulation features and inputs them into the body DDPM. The output body latent code is input into a linear layer together with the initial body motion latent code to obtain the body motion latent code that integrates the manipulation information. Similarly, the hand control module extracts manipulation features and inputs the manipulation features together with the initial hand motion latent code into the hand control module to generate a hand motion latent code that integrates the manipulation information; finally, the optimized body motion code and hand motion code are input into the corresponding motion decoding module constructed in step one to generate the final overall motion that integrates the manipulation intention.

5. The method for generating overall human body motion for manipulation perception according to claim 1, characterized in that: After constructing effective control features, progressive control guidance is used to use these features to generate overall actions with control intent. The optimization process includes: 1) Latent motion learning optimization: The optimization object is the conditional variational autoencoder obtained from the first stage of training. The body and hand motion codes are embedded in the latent space of the CVAE and serve as an important reference for generating the overall motion code in steps 2 and 3. In addition, the body and hand motion decoders in the CVAE rely on the overall motion code information to reconstruct the overall motion sequence; 2) Train the sequential diffusion model from step 2 to learn complex body postures and hand details; 3) Leverage the manipulation representation to train a sequence control network to optimize the overall action encoding.

6. The method for generating whole-body motion for manipulation perception according to claim 5, characterized in that: In step 1), the object of optimization is the conditional variational autoencoder trained in step 1. The motion codes of the human body and hands are embedded in the latent space of CVAE and serve as an important reference for generating the overall motion code in steps 2 and 3. In addition, the body and hand motion decoder in CVAE relies on the motion codes of the human body and hands to reconstruct the overall motion sequence. The loss function of CVAE is: （3）， The first term is the KL divergence loss , which is used to constrain the potential space to move closer to the standard normal distribution, where represents the weight of KL divergence loss, represents the latent distribution predicted by the encoder With standard normal distribution The KL divergence between The second term is the joint rotation reconstruction loss , used to constrain the predicted joint rotation to be closer to the real joint rotation, where represents the weight of the loss, represents the smooth-L1 loss function, and are the predicted and true joint rotation values, respectively; The third term is the joint position loss calculated based on forward kinematics. , used to constrain the predicted joint positions to be closer to the true joint positions, where represents the weight of the loss, and is and Joint positions calculated via forward kinematics.

7. The method for generating whole-body motion for manipulation perception according to claim 5, characterized in that: In step 2, the sequential diffusion model from step 2 is trained to learn complex body postures and hand details. When the input is a noisy body motion latent code and a sparse tracking signal, the loss function of the body diffusion model is: （4）， in and Represent the predicted and real body movement latent codes respectively; Next, we fix the parameters in the Body DDPM and train the hand diffusion model with the loss function: （5）， in and Represent the predicted and true hand motion latent codes, respectively.

8. The method for generating whole-body motion for manipulation perception according to claim 5, characterized in that: 3) In this paper, we use the manipulation representation to train the sequence control network to optimize the overall motion encoding. After fixing the parameters of the body diffusion model and the hand diffusion model, we optimize the body control module using the following formula: （6）， This loss function includes the rotation loss of the body joints and position loss of body joints Two items, including is the weight of the loss at that position, where and Represent the potential encoding of body movements with predicted and real fusion manipulation information, and Based on and The calculated joint positions; The loss function of the hand control module is as follows: （7）， in, and They represent the potential codes of hand movements that are the result of predicted and real fusion manipulation information, and Based on and The calculated hand joint positions, excluding the hand joint rotation loss Loss of hand joint position In addition, it also includes the displacement loss of the key vertices of the hand ,in represents the weight of the displacement loss, To manipulate the sampled hand vertex positions in the representation, Outputs the predicted vertex positions for hand movements.

9. The method for generating whole-body motion for manipulation perception according to claim 1, characterized in that: By training a lightweight online network to fit the sequential diffusion model constructed in steps 2 and 3 and guided by the control network, the model can reason in fewer steps and predict clear potential action representations. In steps 2 and 3, for each DDPM, a 24-layer DiT denoiser is trained as the teacher model and distilled into a target 6-layer denoiser model. Since the teacher model and the target model share the same network architecture and feature dimensions, the target denoiser learns not only the final output of the teacher model but also the outputs of some of its intermediate layers. During the diffusion inference process in steps 2 and 3, five denoising steps are initially used; subsequently, a distillation process is used to reduce the number of denoising steps, and ultimately only one denoising step is required to complete the inference process.

Images