Diffusion model-based three-dimensional digital human and object interactive motion synthesis method and system

Abstract

The invention belongs to the field of computer vision, computer graphics and robots, and relates to a three-dimensional digital human and object interactive motion synthesis method and system based on a diffusion model. The method comprises the following steps: acquiring three types of feature vectors: a three-dimensional grid shape of an object, a frame-by-frame motion sequence of the object and a body type feature vector of a digital human; performing conditional feature coding on the obtained three types of feature vectors to obtain conditional vectors; on the basis of the diffusion model, predicting noiseless estimation of the current time step by using a denoising device in a condition vector iteration manner, and optimizing the generated interactive motion sequence of the three-dimensional digital human and the object according to a generation guide strategy; and obtaining a three-dimensional grid of the human body based on the interactive motion sequence of the three-dimensional digital human and the object, and obtaining an interactive motion synthesis result of the three-dimensional digital human and the object through three-dimensional modeling software. According to the invention, a three-dimensional digital human and object interactive motion sequence with strong sense of reality can be synthesized, and the synthesized three-dimensional digital human has the motion of the trunk and the two hands at the same time.

Description

Technical Field

[0001] This invention specifically relates to the fields of computer vision, computer graphics, and robotics, and more specifically to a method and system for synthesizing the interactive motion of a three-dimensional digital human and object based on a diffusion model. Background Technology

[0002] Virtual 3D digital humans have entered people's daily lives, making it crucial to study how to improve the realism of digital humans. Researching the interactive motion of synthesized digital humans and objects represents a major breakthrough in the field. However, due to the uncertainties in human joint control, the complexity of hand-to-object interaction, and the lack of relevant datasets, existing algorithms for synthesizing 3D digital human-object interaction cannot produce highly realistic results.

[0003] Existing algorithms for synthesizing 3D digital human-object interaction motion can be categorized into methods for synthesizing single-frame digital human grasping objects and methods for synthesizing digital human-object interaction motion sequences. The single-frame synthesis method is relatively simple, with ample existing work and a well-developed field, and will not be discussed in this invention. This invention primarily focuses on methods for synthesizing digital human-object interaction motion sequences.

[0004] Existing deep learning-based algorithms for synthesizing 3D digital human-object interaction motion sequences can be categorized into reinforcement learning-based methods and generative algorithms. Reinforcement learning-based methods, through carefully designed reward and penalty functions for human-object interaction, encourage the human hand to grasp the object and generate subsequent interactive actions. These methods can produce relatively realistic results based on limited data. Generative algorithms, on the other hand, learn statistical patterns from massive datasets using deep learning generative models, such as diffusion models, to generate results that meet certain conditions within the data distribution. The synthesized results of generative algorithms are highly dependent on large amounts of high-quality data.

[0005] Existing reinforcement learning-based methods are still in their early stages of development. Due to the complexity of reward and penalty function design and the high degree of freedom of digital humans manipulating objects with both hands, the algorithm can only produce the action of a digital human grasping an object with one hand and making simple movements, but cannot produce full-body movements involving both hands, resulting in a weak sense of realism in the synthesized results.

[0006] Existing generative algorithm-based methods are limited by the generative models' ability to fit complex data, and are mainly divided into methods that do not include hand movements and methods that only use one or two hands. Methods that do not include hand movements can only synthesize the movements of the human torso joints, mainly for interactive motion scenarios between digital humans and larger objects; methods that only use one or two hands only synthesize hand movements, not the human body, and are suitable for interactive motion scenarios involving the manipulation of smaller objects. Neither of these methods can generate full-body motion in a digital human, resulting in a lack of realism in the synthesized results. Existing generative models include those based on variational autoencoders (VAEs) and diffusion models. Because diffusion models are relatively new, current research and application of them in the field of digital human-object interactive motion synthesis are insufficient. Summary of the Invention

[0007] The purpose of this invention is to use a diffusion model to synthesize highly realistic 3D digital human and object interaction motion sequences, and the synthesized 3D digital human has the movement of both the torso and the hands.

[0008] The technical solution adopted in this invention is as follows: A method for synthesizing the interactive motion of a 3D digital human and object based on a diffusion model includes the following steps: Three types of feature vectors are obtained: the 3D mesh shape of the object, the frame-by-frame motion sequence of the object, and the body shape feature vector of the digital human; Conditional feature encoding is performed on the three types of feature vectors obtained to obtain conditional vectors; Based on the diffusion model, the noise-free estimate of the current time step is predicted by the denoiser using conditional vector iteratively, and the generated 3D digital human and object interaction motion sequence is optimized according to the generation guidance strategy. Based on the interaction motion sequence of the 3D digital human and objects output by the diffusion model, the 3D mesh of the human body is obtained according to the reasoning process of the human body parameterization model, and the 3D digital human and object interaction motion synthesis result is obtained through 3D modeling software.

[0009] Furthermore, the conditional feature encoding of the acquired three types of feature vectors includes: The three types of feature vectors are subjected to two levels of positional encoding to obtain the primary conditional vector; the two levels of positional encoding include feature-level positional encoding and sequence frame-level positional encoding. The primary conditional vector is input into the conditional encoder for encoding to obtain the conditional vector.

[0010] Furthermore, the loss function used in the training phase of the diffusion model is: in, Let be the loss function of the diffusion model. For contact perception reconstruction loss, For contact perception interaction loss, , , This is the balance coefficient.

[0011] Furthermore, the aforementioned , and The following steps are used for calculation: First calculate : in For noise reduction This indicates that the diffusion process is uniformly sampled from time steps 1 to N, and the expected value is taken; the noise denoiser accepts noise steps. The following data and the condition vector generated by the condition encoder Output clean data after noise reduction ; After generating denoised and clean human pose data, all joints of the left and right hands are inferred using SMPL-X. and Simultaneously, the relative position of the generated wrist will be... relative to the position of the object's center of mass Add them together to get the wrist positions of the left and right hands. and Then calculate : in and These are contact tags for the left and right hands; It is the balance coefficient; Then calculate using the following formula. : in, This represents the joints of the hand inferred by SMPL-X. This represents the truth value of the hand's joints, and k represents the iteration over the left and right hands. Introduce exponentially decaying distance perception weights for each hand joint.

[0012] Furthermore, the generation guidance strategy includes grip stability guidance, hand-object contact guidance, and foot-floor penetration guidance.

[0013] Furthermore, the grip stability guidance, hand-object contact guidance, and foot-floor penetration guidance are calculated using the following steps: 1) Calculate grip stability guidance: in, This indicates the grasping stability guiding function. This represents the clean human posture data after noise reduction. The wrist position of the human body is obtained through the inference process of SMPL-X. This indicates the corrected wrist position in the world coordinate system. This indicates the optimized upper body parameters. This represents the initial upper body parameters. This represents the learning rate in gradient descent, used to control the step size for each parameter update. This represents the upper body parameters containing noise at step n. This represents the gradient corresponding to the noisy upper body parameters at step n. This represents the human body parameters after removing all noise. It is the contact tag for frame 1-T; 2) Calculate hand-object contact guidance: in, The hyperparameters representing the balance between contact and penetration. This represents the optimized hand parameters. Indicates the initial hand parameters. This represents the noisy hand parameters at step n. This represents the gradient corresponding to the noisy hand parameters at step n. This represents the learning rate in gradient descent, used to control the step size for each parameter update. This represents the hand-object contact guiding function. Indicates the penetration distance. Indicates the contact distance; 3) Calculate the penetration guidance between the foot and the floor: in, This represents the guiding function for the foot's penetration through the floor. This represents the optimized whole-body parameters. Indicates the initial whole-body parameters. This represents the noisy whole-body parameters at step n. This represents the gradient corresponding to the noisy whole-body parameters at step n. This represents the learning rate in gradient descent, used to control the step size for each parameter update. This represents all the vertices of the human body.

[0014] Furthermore, the process of obtaining the 3D digital human-object interaction motion synthesis result through 3D modeling software is to view the 3D digital human-object interaction motion synthesis result in 3D form or rendered as a 2D video.

[0015] A three-dimensional digital human-object interaction motion synthesis system based on a diffusion model, characterized in that it includes: The feature vector acquisition module is used to acquire three types of feature vectors: the 3D mesh shape of the object, the frame-by-frame motion sequence of the object, and the body shape feature vector of the digital human. The conditional feature encoding module is used to perform conditional feature encoding on the three types of feature vectors obtained to obtain conditional vectors; The diffusion model generation module is used to predict the noise-free estimate of the current time step based on the diffusion model and the denoiser using conditional vectors in an iterative manner, and to optimize the generated 3D digital human and object interaction motion sequence according to the generation guidance strategy. The interactive motion result acquisition module is used to obtain the interactive motion sequence of the 3D digital human and objects based on the output of the diffusion model, obtain the 3D mesh of the human body according to the reasoning process of the human body parameterization model, and obtain the 3D digital human and object interactive motion synthesis result through 3D modeling software.

[0016] The key points and corresponding beneficial effects of this invention are as follows: Key Point 1: A method for generating 3D digital human-object interaction motion using a diffusion model; Beneficial technical effect: A highly realistic 3D digital human-object interaction motion is generated using a single diffusion model. The generated digital human's full-body joint movements are coordinated and consistent, with natural hand grasping. Existing methods for generating 3D digital human-object interaction motion using diffusion models do not simultaneously generate both the digital human's torso and hand movements using a single diffusion model. Often, only one aspect can be selected for generation, or two diffusion models are used separately and then stitched together to create the full-body motion. This invention, however, generates the full-body motion of the digital human using a single diffusion model. Compared to existing methods like OMOMO, which only generate digital human torso motion and not hand motion, this invention significantly improves the realism of the results.

[0017] Key Point 2: A conditional injection method for diffusion models utilizing a Transformer encoder-decoder architecture; Beneficial technical effects: Using the Transformer encoder as a conditional encoder to encode the conditional vector, and using the Transformer decoder as a denoiser for the diffusion model to generate denoised results, this method completes the task in a form suitable for Transformer, from input sequence to output sequence, and can effectively guide the diffusion model to generate results that meet the input conditions.

[0018] Key Point 3: A positional encoding method that simultaneously indicates the location of network features; Beneficial technical effect: By using positional encoding, different positional codes are added to different features that are concatenated together. When used in conjunction with conventional sequence frame-level positional encoding, it can effectively indicate the concatenation relationship of different network features.

[0019] Key Point 4: A contact perception reconstruction loss; Beneficial technical effect: This loss can help the network better learn the posture of a human hand grasping an object, and provides feature cues in Euclidean space outside of rotational space, thereby enhancing the spatial perception ability of the 3D digital human. Introducing this constraint-trained diffusion model during the training phase enables the generated 3D digital human to better perceive the spatial position of objects.

[0020] Key Point 5: A contact-aware interaction loss; Beneficial technical effects: This loss can help the network better perceive the position of objects in the air, helping the generated human's hand to better approach the object, and enhancing the spatial perception ability of the 3D digital human. Introducing this constraint to train the diffusion model during the training phase enables the generated 3D digital human to produce more realistic interaction postures with objects.

[0021] Key Point 6: A generation guidance method in the inference phase of the diffusion model. Beneficial technical effects: The diffusion model employs an iterative process for gradual denoising during the inference phase. Each step of the iterative denoising process optimizes the generated digital human, guiding the diffusion model to produce more realistic results. By continuously optimizing hand penetration, encouraging hand-object contact, improving hand grasping stability, and optimizing foot contact with the ground in the denoised results from the diffusion model during the generation phase, the network is guided to generate more realistic results. Attached Figure Description

[0022] Figure 1 This is a flowchart of a three-dimensional digital human-object interaction motion synthesis method based on a diffusion model.

[0023] Figure 2 This is a schematic diagram of the processing procedure for a three-dimensional digital human-object interaction motion synthesis method based on a diffusion model.

[0024] Figure 3 This is a comparison chart of the work results.

[0025] Figure 4 This is a graph showing the results of an ablation experiment.

[0026] Figure 5 This is a block diagram of the modular composition of a 3D digital human-object interaction motion synthesis system based on a diffusion model. Detailed Implementation

[0027] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0028] The flowchart of a three-dimensional digital human-object interaction motion synthesis method based on a diffusion model according to the present invention is as follows: Figure 1 As shown, it is mainly divided into four stages: input stage, conditional feature encoding stage, diffusion model generation stage, and output stage.

[0029] 1) Input stage: First, the input is obtained, which mainly includes three types of feature vectors: the three-dimensional mesh shape of the object, the frame-by-frame motion sequence of the object, and the body feature vector of the digital human. These three types of features are used as conditions for the generative model to guide the generative model to produce results that meet the conditions.

[0030] 2) Conditional Feature Encoding Stage: After receiving the input, three types of features are encoded. First, the three types of features undergo two levels of positional encoding to obtain primary conditional vectors: feature-level positional encoding to prompt the network to distinguish different feature blocks, and sequence-level positional encoding to prompt the network to distinguish different frames. Then, the primary conditional vectors are input into the conditional vector encoder for encoding to obtain the conditional vectors.

[0031] 3) Diffusion Model Generation Stage: This stage mainly uses the diffusion model principle and conditional vectors to iteratively predict the noise-free estimate of the current time step using a denoiser. It also optimizes the generated 3D digital human-object interaction motion results according to a pre-designed generation guidance strategy. Then, weaker noise is added back to the result for the next iteration, progressing step by step according to a preset time step until the final time t=0, at which point the fully denoised final result is output, i.e., the denoised 3D digital human-object interaction motion sequence.

[0032] 4) Output stage: The diffusion model outputs a three-dimensional digital human and object interaction motion sequence. Then, based on the reasoning process of the human body parameterization model, the three-dimensional mesh of the human body is obtained. The generated result can be viewed in three-dimensional form or rendered as a two-dimensional video using three-dimensional modeling software.

[0033] like Figure 2As shown, this invention generates 3D digital human-object interactive motion based on a diffusion model. Since the generation result of this diffusion model is based on three conditions—object shape representation, object motion sequence, and body shape features of the human parametric model—it is named a conditional diffusion model. During the training phase of the conditional diffusion model, after each input condition undergoes MLP feature alignment, feature-level position encoding, sequence frame-level position encoding, and a conditional encoder, a conditional vector is obtained. This conditional vector is input to a denoiser to guide the output of a denoised result that meets the conditions. This invention follows the training strategy of the diffusion model to predict the denoised result; during the training phase, this invention does not optimize the result, and the generation guidance part on the right has no training parameters and does not participate in training. In the inference generation phase, this invention follows the inference strategy of the diffusion model, iteratively outputting the denoised result from the noisiest data, then guiding and optimizing the result before adding less noise in the next stage, iterating until the noise is completely eliminated, and finally generating the denoised 3D digital human-object interactive motion sequence.

[0034] The present invention provides a method for synthesizing the interactive motion of a three-dimensional digital human and object based on a diffusion model, which specifically includes the following steps: 1. Input conditions The input criteria include an object shape representation. Object motion sequence representation and SMPL-X body type characteristics The three types of features are aligned to the same dimension using three multilayer perceptrons (MLPs) and concatenated along the feature dimension, then encoded with the feature-level position. Blocks with identified features are combined and encoded with sequence frame-level location. The sequence position information of the identified features is added to obtain the primary conditional vector. The primary conditional vector is then used by the input conditional encoder to calculate the final conditional vector. .

[0035] 1.1 Input Feature Representation: 1.1.1 Representation of object shape The shape of the object is initially input as a point cloud. In obtaining the object's... After obtaining the point cloud coordinates of the frame, this invention uses BPS encoding to encode the point cloud of the object in that frame into a feature vector of length 1024*3. .

[0036] 1.1.2 Representation of Object Motion : in the Frame, the motion of an object It is composed of three pieces of information: the object's translation, the object's rotation (represented using 6D rotation), and the object's center position. .

[0037] 1.1.3 Human SMPL-X Body Type Characteristics Body shape features include body shape and gender. The body shape feature is a 10-dimensional vector used to control the shape characteristics of the human body, such as height and weight. The gender feature is a 1-dimensional vector used to represent the sex of the human body. .

[0038] 1.2 Feature Alignment and Stitching The three types of features obtained in Section 1.1 are aligned using MLPs with the same structure but independent features, and the dimensions of each feature are changed to 256. Then, they are concatenated along the feature dimensions.

[0039] 1.3 Location Coding: This invention incorporates positional encoding into the features. Besides the common positional encoding that indicates the frame number of a sequence, this invention refers to it as sequence-frame-level positional encoding. This invention also features a specially designed feature-level positional encoding. To better indicate three different types of features in the network: object shape, object motion, and human body shape. Expressed as a formula, in the... Frame, the obtained primary condition vector for: in It is a vector concatenation operation.

[0040] In existing technologies, positional coding is typically used to describe the temporal order of sequence frames, i.e., sequence frame-level positional coding. The proposed feature-level positional encoding does not assign structural positional information to different categories of features in the network input. This involves structured cue encoding designed for different modalities or semantic types (such as object shape, object motion, and human body size) in the input features, thereby significantly improving the model's ability to perceive differences in feature types. The feature-level positional encoding of this invention includes three categories: Used for static shape class features ; Features for dynamic motion ; Body shape characteristics related to individual human differences .

[0041] 1.4 Conditional Encoder: The conditional encoder consists of the encoder of the Transformer. It converts the primary conditional vector... Encode as feature vector That is, the condition vector.

[0042] 2. Conditional Diffusion Model This invention uses DDPM in the diffusion model, and uses the feature vector (i.e., condition vector) output by the condition encoder to inject generation conditions to guide the generation results, hence it is called the conditional diffusion model.

[0043] 2.1 Variable Representation 2.1.1 Generated Content. The generated human motion sequence H is represented using the pose parameters of the SMPL-X model. Specifically, the human pose of each frame is composed of the global translation and global rotation of the root joints, as well as the 6D rotation parameters of all joints in the body. Furthermore, this invention also generates a displacement sequence relative to the object's center of mass. It is used to describe the spatial position of a digital human's wrist in another way.

[0044] 2.1.2 Contact Tag. To independently represent the contact relationship between each hand and the object, this invention designs a binary mask. This is called a contact tag. These are the contact tags for frames 1-T, with dimensions (T, 2). The contact tags for frame t are... The two components are represented by 1 or 0 respectively, indicating that in the frame Whether the left or right hand is in contact with the object. Contact labels are determined by whether the minimum distance between each hand and a given object grid vertex is less than a given threshold (e.g., ...). The calculation was performed to determine the relative distance between the wrist and the object during the reasoning process. Whether the value is less than a given threshold to obtain a contact tag.

[0045] 2.2 Diffusion Model This embodiment uses the DDPM diffusion model, and the denoiser uses a Transformer decoder. The denoiser accepts a specific number of noise steps. and data at that noise level and the condition vector generated by the condition encoder Output clean data after noise reduction The noise steps are also called the noise reduction steps. It represents the overall representation vector at noise step n, which includes a joint representation of multimodal features such as digital human pose and object state.

[0046] Loss function of diffusion model It can be written as: in For noise reduction This represents uniform sampling of the diffusion process across time steps 1 to N, with the expected value taken. The loss function uses mean squared error (MSE) to constrain the clean data after noise reduction. and real data The differences.

[0047] 2.3 Other Loss Functions 2.3.1 Contact Perception Reconstruction Loss Clean, denoised human pose data generated by the network Subsequently, this invention uses SMPL-X to deduce all the joints of the left and right hands. and Simultaneously, the relative wrist position generated by the network will be... relative to the position of the object's center of mass Add them together to get the wrist positions of the left and right hands. and The present invention then calculates the contact sensing reconstruction loss. : in and These are contact tags for the left and right hands; It is the balance coefficient.

[0048] 2.3.2 Contact Perception Interaction Loss To make the model more sensitive to the proximity of the hands to objects, this invention introduces exponentially decaying distance-aware weights for each hand joint. : in It is the balance coefficient. This represents the Euclidean distance at each joint.

[0049] Then calculate the contact perception interaction loss. as follows: in, This represents the joints of the hand inferred by SMPL-X. represents the truth value of the hand's joints, and k represents traversing the left and right hands.

[0050] In summary, all loss functions during the training phase can be written as: All of them All are balance coefficients.

[0051] 3. Generate bootstrapping strategy This invention incorporates a generation guidance strategy during the inference phase. The generation process of the DDPM diffusion model is iterative, starting from the first... The data under the noise step is processed by a denoiser to generate clean data for step 0, and then noise is added to it. The process continues until clean data for step 0 is obtained from the noisy data of step 1. This generation guidance involves fine-tuning the clean data before each addition of noise to guide the direction of the generated result.

[0052] 3.1 Grasp stability guidance This invention uses the relative distance generated above at this stage. Complete the grip stability guidance. Given a series of relative distances of the left wrist. By calculating the relative distance To obtain its contact tag sequence, the value must be less than a certain threshold. Assuming the contact period starts from... arrive ( The objective of this invention is to ensure that in the first... At frame rate, the position of the wrist relative to the object in the next... Intra-frame remains unchanged. For the next... Any frame in the frame The corrected wrist position in the world coordinate system is calculated using the following formula: in and Is the object in the th order? Frame rotation and translation and Is the object in the th order? Frame rotation and translation. The invention will then generate... The wrist position of the human body is obtained through the inference process of SMPL-X, and then the wrist position is optimized to improve... : in, This indicates the grasping stability guiding function. express The wrist position of the human body is obtained through the inference process of SMPL-X. This indicates the calculated, corrected wrist position in the world coordinate system. This indicates the optimized upper body parameters. This represents the initial upper body parameters. This represents the learning rate in gradient descent, used to control the step size for each parameter update. This represents the upper body parameters containing noise at step n. This represents the gradient corresponding to the noisy upper body parameters at step n. This represents the human body parameters after removing all noise.

[0053] This invention uses contact tags All that's needed is to optimize the hands that make contact.

[0054] 3.2 Hand-object contact guidance This invention uses hand-object contact guidance to reduce potential penetration between the human body and object meshes inferred by SMPL-X, and encourages contact between the hand and the object. For hand vertices... The penetration distance and contact distance are calculated as follows: in, Represents the set of hand sampling points. Indicates signed distance. Indicates the first One hand sampling point.

[0055] This invention calculates the vertices of the hand. The signed distance (SDF) between the object and the mesh is used to calculate the distance between them, while also calculating the penetration distance and the contact distance. This invention defines hand-object contact guidance as follows: in, The hyperparameters representing the balance between contact and penetration. This represents the optimized hand parameters. Indicates the initial hand parameters. This represents the noisy hand parameters at step n. This represents the gradient corresponding to the noisy hand parameters at step n. This represents the learning rate in gradient descent, used to control the step size for each parameter update. This represents the hand-object contact guiding function.

[0056] 3.3 Foot-to-floor penetration guidance This invention employs foot-to-floor penetration guidance to encourage contact between the body and the ground. This applies to all vertebrae of the body. : in, This represents the guiding function for the foot's penetration through the floor. This represents the optimized whole-body parameters. Indicates the initial whole-body parameters. This represents the noisy whole-body parameters at step n. This represents the gradient corresponding to the noisy whole-body parameters at step n. This represents the learning rate in gradient descent, used to control the step size for each parameter update.

[0057] This invention uses the three guiding functions described above to explicitly guide the diffusion model generation process, thereby generating results that better match the conditions of object motion. The objective function can be defined as: like Figure 3 As shown in the comparison results, compared to the existing method OMOMO, this invention can generate the hand grasping posture during the interaction between a 3D digital human and an object. Figure 4 As shown in the ablation experiment results, the contact perception reconstruction loss and contact perception interaction loss proposed in this invention, as well as the generation guidance method, can effectively generate realistic 3D digital human and object interaction motion.

[0058] Other embodiments of the present invention: In the input stage of this invention, the number or type of input information can be increased or adjusted; In the conditional encoder of the present invention, the number of layers of the Transformer encoder can be increased or decreased, and the number of layers of the denoiser can be increased or decreased. In the conditional diffusion model of this invention, loss functions can be added or removed, or the weight values ​​of the loss functions can be modified; In the generation guide of this invention, guide items can be added or deleted, or the calculation order between items can be modified; The application of this invention can be extended to objects similar to digital humans, such as humanoid robots, virtual animals, robotic arms, and robotic hands.

[0059] Another embodiment of the present invention provides a three-dimensional digital human-object interaction motion synthesis system based on a diffusion model, such as... Figure 5 As shown, it includes: The feature vector acquisition module is used to acquire three types of feature vectors: the 3D mesh shape of the object, the frame-by-frame motion sequence of the object, and the body shape feature vector of the digital human. The conditional feature encoding module is used to perform conditional feature encoding on the three types of feature vectors obtained to obtain conditional vectors; The diffusion model generation module is used to predict the noise-free estimate of the current time step based on the diffusion model and the denoiser using conditional vectors in an iterative manner, and to optimize the generated 3D digital human and object interaction motion sequence according to the generation guidance strategy. The interactive motion result acquisition module is used to obtain the interactive motion sequence of the 3D digital human and objects based on the output of the diffusion model, obtain the 3D mesh of the human body according to the reasoning process of the human body parameterization model, and obtain the 3D digital human and object interactive motion synthesis result through 3D modeling software.

[0060] The specific working process of each of the above modules can be found in the corresponding process in the aforementioned method embodiments.

[0061] It should be understood that the methods and systems disclosed in the above embodiments of the present invention can be implemented in other ways. For example, the above module division can be implemented in other ways, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Each step and module in the present invention can be implemented in the form of software functional units and can be stored in a computer-readable storage medium, including several instructions to cause a computer device to execute some or all of the steps of the method described in the present invention. For example, one embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for executing each step of the method of the present invention. For example, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk, etc.) storing a computer program, which, when executed by a computer, implements each step of the method of the present invention. For example, another embodiment of the present invention provides a computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0062] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for synthesizing the interactive motion of a three-dimensional digital human and object based on a diffusion model, characterized in that, Includes the following steps: Three types of feature vectors are obtained: the 3D mesh shape of the object, the frame-by-frame motion sequence of the object, and the body shape feature vector of the digital human; Conditional feature encoding is performed on the three types of feature vectors obtained to obtain conditional vectors; Based on the diffusion model, the noise-free estimate of the current time step is predicted by the denoiser using conditional vector iteratively, and the generated 3D digital human and object interaction motion sequence is optimized according to the generation guidance strategy. Based on the interaction motion sequence of the 3D digital human and objects output by the diffusion model, the 3D mesh of the human body is obtained according to the reasoning process of the human body parameterization model, and the 3D digital human and object interaction motion synthesis result is obtained through 3D modeling software.

2. The method according to claim 1, characterized in that, The conditional feature encoding of the acquired three types of feature vectors includes: The three types of feature vectors are subjected to two levels of positional encoding to obtain the primary conditional vector; the two levels of positional encoding include feature-level positional encoding and sequence frame-level positional encoding. The primary conditional vector is input into the conditional encoder for encoding to obtain the conditional vector.

3. The method according to claim 1, characterized in that, The loss function used during the training phase of the diffusion model is: in, Let be the loss function of the diffusion model. For contact perception reconstruction loss, For contact perception interaction loss, , , This is the balance coefficient.

4. The method according to claim 3, characterized in that, The , and The following steps are used for calculation: First calculate : in For noise reduction This indicates that the expected value is obtained by uniformly sampling the diffusion process at time steps 1 to N. Noise denoiser accepts noise steps The following data and the condition vector generated by the condition encoder Output clean data after noise reduction ; After generating denoised and clean human pose data, all joints of the left and right hands are inferred using SMPL-X. and Simultaneously, the relative position of the generated wrist will be... relative to the position of the object's center of mass Add them together to get the wrist positions of the left and right hands. and Then calculate : in and These are contact tags for both left and right hands; It is the balance coefficient; Then calculate using the following formula. : in, This represents the joints of the hand inferred by SMPL-X. This represents the truth value of the hand's joints, and k represents the iteration over the left and right hands. Introduce exponentially decaying distance perception weights for each hand joint.

5. The method according to claim 1, characterized in that, The generation guidance strategy includes grip stability guidance, hand-object contact guidance, and foot-floor penetration guidance.

6. The method according to claim 5, characterized in that, The grip stability guidance, hand-object contact guidance, and foot-floor penetration guidance are calculated using the following steps: 1) Calculate grip stability guidance: in, This indicates the grasping stability guiding function. This represents the clean human posture data after noise reduction. The wrist position of the human body is obtained through the inference process of SMPL-X. This indicates the corrected wrist position in the world coordinate system. This indicates the optimized upper body parameters. This represents the initial upper body parameters. This represents the learning rate in gradient descent, used to control the step size for each parameter update. This represents the upper body parameters containing noise at step n. This represents the gradient corresponding to the noisy upper body parameters at step n. This represents the human body parameters after removing all noise. It is the contact tag for frame 1-T; 2) Calculate hand-object contact guidance: in, The hyperparameters representing the balance between contact and penetration. This represents the optimized hand parameters. Indicates the initial hand parameters. This represents the noisy hand parameters at step n. This represents the gradient corresponding to the noisy hand parameters at step n. This represents the learning rate in gradient descent, used to control the step size for each parameter update. This represents the hand-object contact guiding function. Indicates the penetration distance. Indicates the contact distance; 3) Calculate the penetration guidance between the foot and the floor: in, This represents the guiding function for the foot's penetration through the floor. This represents the optimized whole-body parameters. Indicates the initial whole-body parameters. This represents the noisy whole-body parameters at step n. This represents the gradient corresponding to the noisy whole-body parameters at step n. This represents the learning rate in gradient descent, used to control the step size for each parameter update. This represents all the vertices of the human body.

7. The method according to claim 1, characterized in that, The process of obtaining the 3D digital human-object interaction motion synthesis result through 3D modeling software is to view the 3D digital human-object interaction motion synthesis result in 3D form or rendered as a 2D video.

8. A three-dimensional digital human-object interaction motion synthesis system based on a diffusion model, characterized in that, include: The feature vector acquisition module is used to acquire three types of feature vectors: the 3D mesh shape of the object, the frame-by-frame motion sequence of the object, and the body shape feature vector of the digital human. The conditional feature encoding module is used to perform conditional feature encoding on the three types of feature vectors obtained to obtain conditional vectors; The diffusion model generation module is used to predict the noise-free estimate of the current time step based on the diffusion model and the denoiser using conditional vectors in an iterative manner, and to optimize the generated 3D digital human and object interaction motion sequence according to the generation guidance strategy. The interactive motion result acquisition module is used to obtain the interactive motion sequence of the 3D digital human and objects based on the output of the diffusion model, obtain the 3D mesh of the human body according to the reasoning process of the human body parameterization model, and obtain the 3D digital human and object interactive motion synthesis result through 3D modeling software.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images