A text-driven three-dimensional human-object interaction action generation method and system based on physical simulation

Abstract

The application discloses a kind of text-driven three-dimensional man-object interaction action generation method and system based on physical simulation, and relates to computer animation and artificial intelligence technical field.The method comprises: obtaining text description and the object geometric information of interactive object;The above information is input into pre-trained generation model, and the preliminary three-dimensional man-object interaction action sequence based on kinematics is obtained;The control strategy learned by the physical simulation correction model is used to calculate the real force and collision of the preliminary action sequence in the physical simulation environment, and the physical real state trajectory is obtained;The target action sequence is output based on the trajectory.In addition, the application uses the physical simulation correction model as the differentiable physical agent, and fine-tunes the generation model parameters through the dynamics consistency loss, to realize the bottom internalization of physical law.The application effectively eliminates the penetration and drift phenomenon in generated action, and significantly improves the reality and physical consistency of action in complex interaction task.

Description

Technical Field

[0001] This invention relates to the fields of computer animation and artificial intelligence technology, and more specifically, to a method and system for generating text-driven 3D human-object interactive actions based on physical simulation. Background Technology

[0002] With the rapid development of virtual reality, digital entertainment, and robot simulation, the technology for generating natural and realistic interactive actions between 3D digital humans and objects in virtual environments has become a research hotspot. This technology not only enhances the expressiveness and immersion of virtual characters but also has significant application value in areas such as human-computer interaction and intelligent simulation training.

[0003] Currently, text-driven 3D motion generation technology has made some progress, capable of generating human pose sequences based on semantic descriptions and initially supporting the synthesis of object interaction actions. However, existing methods still face significant problems of insufficient physical plausibility when generating complex human-object interaction actions. Specifically, the generated motion sequences often exhibit "clipping," where the human model and object model intersect in a non-physical way; simultaneously, object movements are prone to "drifting," such as suspension or sliding, which do not conform to the laws of realistic dynamics. These problems are particularly prominent in interactive tasks that require continuous force transmission and contact maintenance, such as pushing, pulling, carrying, and using tools.

[0004] To address the aforementioned clipping and drifting issues, existing technologies have proposed some optimization schemes, but fundamental flaws still exist:

[0005] On the one hand, some existing technologies attempt to post-process and correct the generated pose by introducing neural networks (such as intervention networks) after the diffusion model, using the predicted human-object relative spatial relationship and distance field loss. However, these methods essentially remain at the level of fitting kinematic and geometric data, lacking modeling of the underlying physical mechanisms of the real physical world (such as mass, friction, rigid body collisions, and reaction forces). This purely data-driven spatial geometry correction cannot fundamentally guarantee the authenticity of mechanical transmission and remains rigid and prone to failure when dealing with complex dynamic interactions.

[0006] On the other hand, some existing technologies employ feedback mechanisms based on spatial error indicators (such as contact distance deviation, trajectory collision probability, etc.). When an error exceeds the limit, the diffusion model is triggered to generate a correction frame. However, this type of open-loop or shallow closed-loop strategy of "detecting errors and generating correction frames" mainly relies on surface spatial rules for frame-by-frame repair feedback. This not only easily leads to the accumulation of temporal errors and motion jitter, but also the diffusion model itself does not internalize the underlying physical laws, resulting in poor performance when generalized to complex physical scenes that have never been seen before.

[0007] In summary, existing text-driven action generation methods are often disconnected from realistic physics simulation systems. Current solutions, whether through post-processing fine-tuning of geometric features or frame-by-frame regeneration based on spatial errors, fail to address the physical inconsistency problem at the generator level.

[0008] Therefore, a novel framework is urgently needed that not only enables physical correction based on text-driven generation, but more importantly, truly "internalizes" physical constraints into the diffusion model. That is, a mechanism is needed that can treat the physical simulation environment as a "differentiable proxy," calculating the dynamic consistency loss between the action sequence and the physical reality, and using gradient backpropagation to penetrate the physical simulator to directly fine-tune the internal parameters of the text-driven diffusion model. Only by forcing the generator's feature distribution to converge directly in a direction conforming to Newtonian mechanics can we generate 3D human-object interaction actions that conform to physical laws and completely eliminate clipping and drift phenomena from the source, without relying on complex control strategies. This is precisely the technical problem that this invention aims to solve. Summary of the Invention

[0009] The technical problem this invention aims to solve is to address the shortcomings of existing technologies by providing a text-driven 3D human-object interaction action generation method and system based on physical simulation. This invention not only performs dynamic correction on the initial action sequence in a physical simulation environment, but also uses the physical simulation model as a differentiable proxy, directly fine-tuning the internal parameters of the diffusion model by calculating the dynamic consistency loss and backpropagating it. This method corrects the generation defects of the kinematic model from the model's underlying layer, effectively eliminating clipping and drift phenomena in the generated action sequence, and significantly improving the realism and physical consistency of 3D digital human actions in complex interactive tasks.

[0010] The present invention discloses a text-driven 3D human-object interaction action generation method based on physical simulation, which includes the following steps: S1. Obtain the input text description information and the object geometry information of the interactive object, wherein the object geometry information includes point cloud data for guiding the diffusion model conditions and a set of convex hull geometry for force calculation of the physics simulation engine. S2. Input the text description information and point cloud data into a pre-trained text-driven interactive action generation model to generate a preliminary three-dimensional human-object interaction action sequence based on kinematics; S3. The preliminary three-dimensional human-object interaction sequence is used as a reference trajectory and input together with the convex hull geometry set into the pre-trained physical simulation correction model; S4. Using the policy network learned by the physical simulation correction model, the human joints are driven by the physics engine in the physical simulation environment to perform realistic force and collision calculations and obtain the physical real state trajectory. S5. Extract the target's three-dimensional human-object interaction sequence based on the physical real-state trajectory; S6. Using the physical simulation correction model as a differentiable physical proxy, calculate the dynamic consistency loss between the target 3D human-object interaction action sequence and the physical real state trajectory, and update the internal parameters of the text-driven interaction action generation model by penetrating the differentiable physical proxy through the gradient backpropagation mechanism. .

[0011] Preferably, step S1 specifically includes: S1.1. Obtain natural language text describing the intention of human-object interaction as the text description information; S1.2. Obtain the 3D model of the interactive object; S1.3. Randomly sample N key points on the surface of the object's 3D model to generate point cloud data; simultaneously, use the V-HACD algorithm to perform convex hull decomposition on the object's 3D model to generate a set of convex hull geometries.

[0012] Preferably, step S2 specifically includes: S2.1. The text description information is encoded into a text semantic feature vector by the text encoder in the text-driven interactive action generation model, and the object geometric feature vector of the point cloud data is extracted by the point cloud geometric encoder in the text-driven interactive action generation model. S2.2. Parallel denoising processing is performed on the text semantic feature vector and the object geometric feature vector using a dual-branch diffusion action generation model; S2.3. A cross-attention communication module is introduced to exchange spatiotemporal information between two feature vectors after denoising, outputting a kinematic-based... Preliminary 3D human-object interaction sequence.

[0013] Preferably, in step S3, the physical simulation correction model is obtained through pre-training using a teacher-student distillation framework.

[0014] Preferably, step S4 specifically includes: S4.1. Initialize the physical simulation environment and construct a rigid body physical collision body based on the convex hull geometry set; S4.2. Construct the current simulation state based on the current simulation parameters of the simulated human body, the rigid body physical collision body, and the preliminary three-dimensional human-object interaction action sequence; S4.3. The strategy network directly maps and calculates the target instruction of the PD controller that drives the simulated human body according to the current simulation state; S4.4. The physics simulation engine receives the target instruction from the PD controller, converts it into driving torque, and performs rigid body dynamics integral calculation by combining the environmental gravity and the friction of the colliding body to generate the next frame of physical reality that conforms to Newtonian mechanics. S4.5. Repeat steps S4.3 to S4.4 to perform closed-loop integral simulation until the entire length of the reference trajectory is covered, generating a complete physical real-state trajectory.

[0015] Preferably, in step S4.3, the calculation expression for the target instruction of the PD controller is:

[0016] In the formula, For PD controller target instructions; Represents the Transformer policy network; current simulation state. This includes the perceived state of the simulated human body and object poses, the nearest distance vector from the human body joints to the convex hull of the object surface, and the force-sensing contact markers. The target state for future frames extracted from the initial action sequence; This represents the policy network weights.

[0017] Preferably, step S5 specifically includes: Human joint rotation data and object 6DoF pose data are extracted frame by frame from the physical real-state trajectory and converted into a target 3D human-object interaction action sequence in standard animation format.

[0018] Preferably, in step S6, the internal parameters The update logic is as follows:

[0019] In the formula, Generate internal parameters for the model of the text-driven interactive action; This represents the loss of dynamic consistency. This is a preliminary 3D human-object interaction sequence; This is the learning rate.

[0020] A system for implementing the aforementioned text-driven 3D human-object interaction action generation method based on physical simulation, the system comprising: Multimodal geometry preprocessing module: used to acquire input text description information and object geometry information of interactive objects, wherein the object geometry information includes point cloud data and a set of convex hull geometry; Text-driven action generation module: used to input the text description information and point cloud data into a pre-trained text-driven interactive action generation model to generate a preliminary three-dimensional human-object interaction action sequence based on kinematics; Rigid body dynamics solution module: used to input the preliminary 3D human-object interaction action sequence as a reference trajectory, together with the convex hull geometry set, into the pre-trained physical simulation correction model; using the policy network learned by the physical simulation correction model, the physical engine drives the human joints to perform realistic force and collision calculations in the physical simulation environment to obtain the physical real state trajectory. Action sequence output module: used to extract the target's three-dimensional human-object interaction action sequence based on the physical real state trajectory; Differentiable Agent Evolution Module: This module uses the physical simulation correction model as a differentiable physical agent to calculate the dynamic consistency loss between the target 3D human-object interaction action sequence and the physical reality trajectory. It then uses a gradient backpropagation mechanism to penetrate the differentiable physical agent and update the internal parameters of the text-driven interaction action generation model. .

[0021] Beneficial effects The advantages of this invention are: by combining text-driven motion generation with physical simulation correction, clipping and drift phenomena in generated actions are effectively eliminated while maintaining semantic accuracy. This invention not only utilizes a pre-trained physical simulation model and its control strategy to perform dynamic adjustments and collision optimization on initial actions in a simulation environment, but also uses this physical simulation model as a differentiable proxy to fine-tune the internal parameters of the generation model (diffusion model) by calculating the dynamic consistency loss. This mechanism endows the generation model with the ability to perceive physical laws from the ground up, significantly improving the physical realism and motion coherence of 3D digital human-object interaction actions, and is particularly suitable for scenarios such as virtual interaction, simulation training, and digital content generation where high motion realism is required. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the steps of the text-driven 3D human-object interaction action generation method based on physical simulation of the present invention.

[0023] Figure 2 This is a schematic diagram of the overall technical route of the text-driven 3D human-object interaction action generation method based on physical simulation of the present invention.

[0024] Figure 3 This is a schematic diagram of the closed-loop execution flow of the physical simulation correction module in the text-driven 3D human-object interaction action generation method based on physical simulation of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to embodiments, but this does not constitute any limitation on the present invention. Any limited modifications made by any person within the scope of the claims of the present invention are still within the scope of the claims of the present invention.

[0026] See Figures 1-3 The present invention discloses a text-driven 3D human-object interaction action generation method based on physical simulation, the method comprising the following steps: S1. Obtain the input text description information and the object geometry information of the interactive object.

[0027] The text description information is obtained from natural language text describing the intention of human-object interaction. Specifically, natural language text describing the intention of human-object interaction is obtained as text description information through the user text input interface or data loading module, such as "pick up the water glass on the table".

[0028] The geometric information of an object can be obtained from the 3D model of the interactive object. Specifically, this 3D model can be derived from a standard 3D model library or a custom modeling file. For the object's 3D model, point cloud data can be generated by randomly sampling N=512 key points on its surface. This is used for the diffusion model condition introduction. Furthermore, to adapt to the efficient collision detection of the physics simulation engine, the 3D model of the object needs to be preprocessed with convex hull decomposition to generate a set of convex hull geometries for force calculation in the physics simulation engine. Point cloud data and the set of convex hull geometries are used as the object's geometric information. Specifically, regarding the set of convex hull geometries, the V-HACD algorithm can be used to decompose the complex model into a set of simplified geometries. , where m is the number of convex hulls after decomposition. This set will serve as a simplified geometric representation for collision detection in subsequent physics simulations.

[0029] S2. Input the point cloud data from the text description information and the object's geometric information into a pre-trained text-driven interactive action generation model to generate a preliminary 3D human-object interaction action sequence based on kinematics. ,in For human movement sequences, This is a 6DoF pose sequence for an object. The text-driven interactive action generation model is a mature action synthesis model in this field based on generative architectures (such as diffusion models or variational autoencoders, VAEs), aiming to map logically consistent preliminary motion trajectories from semantic descriptions. It is particularly noteworthy that the text-driven interactive action generation model in this invention has undergone closed-loop physical alignment fine-tuning through pre-executed step S6, internalizing physical laws into its underlying feature distribution, rather than simply performing pure kinematic fitting. This type of model can generate corresponding 3D human pose and object pose sequences based on text descriptions and object geometric features. In this invention, this model serves as the front end for kinematic generation, providing a preliminary reference for subsequent physical simulation corrections. HOI-Diff (Human-Object InteractionDiffusion) is a typical representative of this type of model.

[0030] The specific action flow for step S2 is as follows: S2.1. Input the text description information into the text encoder of the text-driven interactive action generation model to obtain the text semantic feature vector. The object's geometric information (or the overall representation of its convex hull set) is input into the point cloud geometric encoder of the text-driven interactive action generation model to extract the object's geometric feature vector. .

[0031] S2.2. A dual-branch diffusion action generation model is used to perform parallel denoising on the feature vectors of the human body and objects. Specifically, the text semantic feature vector is denoised through the human body motion branch, and the object geometric feature vector is denoised through the object motion branch.

[0032] S2.3. A cross-attention communication module is introduced to exchange spatiotemporal information between the denoised feature vector representations, outputting a kinematic-based... A preliminary 3D human-object interaction sequence. In this preliminary 3D human-object interaction sequence, the features used to represent the human body include a 263-dimensional redundant representation of joint positions, rotations, joint velocities, and foot contact tags, to ensure the integrity of the initial velocity state in the dynamic tracking in step S4.

[0033] S3. The preliminary 3D human-object interaction sequence is used as a reference trajectory and input together with the convex hull geometry set into the pre-trained physical simulation correction model.

[0034] The physical simulation correction model is pre-trained using a teacher-student distillation framework. In practice, the pre-trained student policy network model and its corresponding physical simulation environment configuration file are directly loaded. Furthermore, the student policy network employs a Transformer architecture for sequence modeling, extracting the dynamic dependencies of human-object coupling through an attention mechanism.

[0035] S4. Perform closed-loop physical correction based on underlying torque control: Utilize the policy network learned by the physical simulation correction model to drive the human joints in the physical simulation environment to perform realistic force and collision calculations and obtain the physical real state trajectory.

[0036] This step is crucial for the present invention to achieve the transition from kinematics to dynamics in motion sequences, and its specific execution flow is as follows: S4.1. Initialization of the Physics Simulation Environment and Construction of Collisions First, initialize the simulation scene in a physics simulation engine (such as Isaac Gym or PhysX). Construct the corresponding physical colliders in the simulation space using a set of convex hull geometries, and set their dynamic properties such as mass and friction coefficient. Simultaneously, implement the preliminary 3D human-object interaction sequence generated by S2. Set as the reference trajectory (Target Stream) of the physical simulation correction model during the simulation process.

[0037] S4.2. Current Simulation Status Construction: The current simulation state is constructed based on the current simulation parameters of the simulated human body, the rigid body physical collision body, and the preliminary three-dimensional human-object interaction action sequence.

[0038] Among them, the current simulation state From the state of perception and target state Together constitute, that is : 1. Perception State : Reflects the physical reality of the simulated character at the current moment, including: the rotation angle of the simulated human body's joints. Root node position Joint angular velocity linear velocity Orientation of interactive objects Central position angular velocity and linear velocity In addition, it also includes the nearest distance vector from human joints to the object surface. and contact markers detected in real time by the simulation engine. .

[0039] 2. Target State Extracting future actions from preliminary 3D human-object interaction sequences The frame reference information is used to guide the simulated character in motion tracking.

[0040] S4.3. The strategy network of the physical simulation correction model directly maps and calculates the target commands, i.e., action commands, of the PD controllers that drive the 51 drive joints of the simulated human body based on the current simulation state. .

[0041] Among them, action instructions based on policy networks The calculation method is as follows: Policy network for physical simulation correction model Receive the above current simulation state As input, the motion commands that drive the simulated human body are calculated through forward propagation. Its mathematical expression is:

[0042] in, This represents the weight parameters in the policy network that have been frozen and pre-learned through a teacher-student distillation framework. The policy network... Output action vector This corresponds to the target angular displacement of the 51 drive joints in the simulated human body model at the next simulation step.

[0043] S4.4. The physics simulation engine receives the target instruction from the PD controller, converts it into driving torque, and performs rigid body dynamics integral calculation by combining the environmental gravity and the friction of the colliding body to generate the next frame of physical reality that conforms to Newtonian mechanics.

[0044] S4.5. Iteratively solve to generate the physical reality trajectory. Repeat steps S4.3 to S4.4 to perform closed-loop integral simulation until the entire reference trajectory is covered, generating a complete physical state trajectory. This trajectory constitutes the basis for calculating the dynamic consistency loss in the subsequent step S6. The benchmark (Ground Truth).

[0045] S5. Output the target's 3D human-object interaction sequence based on the physical reality trajectory.

[0046] Specifically, human joint rotation data and object 6DoF pose data are extracted frame by frame from the physical reality trajectory, and then converted into a target 3D human-object interaction motion sequence in a standard animation format. This sequence can be visualized and further applied using 3D animation software or rendering engines.

[0047] S6. Execute the closed-loop evolution of the generator using a differentiable physical proxy: Using the physical simulation correction model as a differentiable physical proxy, calculate the dynamic consistency loss between the target 3D human-object interaction action sequence and the physical reality trajectory, and update the internal parameters of the text-driven interaction action generation model by penetrating the differentiable physical proxy through a gradient backpropagation mechanism. .

[0048] To enable the text-driven interactive action generation model to have physical awareness during the generation stage, this embodiment executes physical alignment logic during the model training or fine-tuning stage. The specific implementation plan is as follows: S6.1. Constructing a differentiable physical proxy feedback loop The aforementioned physical simulation correction model is used as a differentiable physics proxy. In the fine-tuning iterations, the preliminary 3D human-object interaction sequence generated in step S2 is used... The input is sent to the physical agent. Because the physical simulation correction model employs a neural network-based policy control architecture, its correction process is sensitive to the input sequence. It is differentiable, thus allowing gradients to be fed back from the physical correction results to the front-end generative model.

[0049] S6.2. Calculate the dynamic consistency loss

[0050] Using the physical truth trajectory output in step S4 as a reference (Ground Truth), calculate the preliminary 3D human-object interaction sequence. loss of dynamic consistency with it This loss function is used to quantify the deviation between the resulting motion generation and the physical dynamic constraints, and its calculation formula is as follows:

[0051] in, For the first Preliminary pose data of the frame. This is the corresponding physically corrected pose data. For contact mechanics constraints, This is the balance coefficient.

[0052] S6.3. Perform gradient backpropagation and parameter update Using the chain rule, through the differentiable physical agent, The generated gradient is fed back to the text-driven interactive action generation model. Based on the calculated gradient, the internal weight parameters of the text-driven interactive action generation model are adjusted. The update is performed, and the update logic follows the formula below:

[0053] In the formula, Weight parameters for the model generated for the text-driven interactive action; The preset learning rate; This represents the derivative of the physical consistency deviation with respect to the generated sequence; This represents the derivative of the generated sequence with respect to the model parameters.

[0054] S6.4. Iterative optimization to achieve physical alignment Through multiple iterative training iterations using the aforementioned closed-loop feedback mechanism, the feature distribution of the text-driven interactive action generation model is forced to converge towards a space that conforms to dynamic consistency. After fine-tuning, the text-driven interactive action generation model can directly generate interactive actions that conform to physical constraints in subsequent inference processes (i.e., execution step S2), thereby reducing the correction pressure on the backend physics engine.

[0055] This invention also provides a text-driven 3D human-object interaction action generation system based on physical simulation, comprising the following modules: Multimodal geometry preprocessing module: This module receives text commands and object model files from the user and performs point cloud sampling and convex hull decomposition preprocessing on the object model.

[0056] Text-driven action generation module: This model integrates a pre-trained text-driven interactive action generation model, which is used to generate a preliminary 3D human-object interaction action sequence based on the input text description information and object geometric information.

[0057] Rigid body dynamics solution module: This is the core module of the system, integrating a pre-trained physical simulation correction model and a physical simulation engine. It is responsible for performing dynamic correction and collision optimization on the initial 3D human-object interaction sequence. It uses a policy network to output action commands and relies on the underlying physical engine to perform closed-loop integral calculations of forces and collisions, thus deconstructing the physical reality.

[0058] Action Sequence Output Module: This module receives data from the physical simulation correction module, converts it into standard 3D animation format data (i.e., the target 3D human-object interaction action sequence), and provides visualization preview and file export functions.

[0059] Differentiable Agent Evolution Module: Used to treat the physical simulation correction model as a differentiable physical agent, extract dynamic loss and calculate gradients through the physical agent, and backpropagate to update the internal weight parameters of the text-driven action generation module.

[0060] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention, and these will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A text-driven method for generating 3D human-object interaction actions based on physical simulation, characterized in that, The method includes the following steps: S1. Obtain the input text description information and the object geometry information of the interactive object, wherein the object geometry information includes point cloud data for guiding the diffusion model conditions and a set of convex hull geometry for force calculation of the physics simulation engine. S2. Input the text description information and point cloud data into a pre-trained text-driven interactive action generation model to generate a preliminary three-dimensional human-object interaction action sequence based on kinematics; S3. The preliminary three-dimensional human-object interaction sequence is used as a reference trajectory and input together with the convex hull geometry set into the pre-trained physical simulation correction model; S4. Using the policy network learned by the physical simulation correction model, the human joints are driven by the physics engine in the physical simulation environment to perform realistic force and collision calculations and obtain the physical real state trajectory. S5. Extract the target's three-dimensional human-object interaction sequence based on the physical real-state trajectory; S6. Using the physical simulation correction model as a differentiable physical proxy, calculate the dynamic consistency loss between the target 3D human-object interaction action sequence and the physical real state trajectory, and update the internal parameters of the text-driven interaction action generation model by penetrating the differentiable physical proxy through the gradient backpropagation mechanism. .

2. The method for generating text-driven 3D human-object interaction actions based on physical simulation according to claim 1, characterized in that, Step S1 specifically includes: S1.

1. Obtain natural language text describing the intention of human-object interaction as the text description information; S1.

2. Obtain the 3D model of the interactive object; S1.

3. Randomly sample N key points on the surface of the object's 3D model to generate point cloud data; simultaneously, use the V-HACD algorithm to perform convex hull decomposition on the object's 3D model to generate a set of convex hull geometries.

3. The method for generating text-driven 3D human-object interaction actions based on physical simulation according to claim 1, characterized in that, Step S2 specifically includes: S2.

1. The text description information is encoded into a text semantic feature vector by the text encoder in the text-driven interactive action generation model, and the object geometric feature vector of the point cloud data is extracted by the point cloud geometric encoder in the text-driven interactive action generation model. S2.

2. Parallel denoising processing is performed on the text semantic feature vector and the object geometric feature vector using a dual-branch diffusion action generation model; S2.

3. A cross-attention communication module is introduced to exchange spatiotemporal information between two feature vectors after denoising, outputting a kinematic-based... A preliminary 3D human-object interaction sequence.

4. The method for generating text-driven 3D human-object interaction actions based on physical simulation according to claim 1, characterized in that, In step S3, the physical simulation correction model is obtained through pre-training using a teacher-student distillation framework.

5. The method for generating text-driven 3D human-object interaction actions based on physical simulation according to claim 1, characterized in that, Step S4 specifically includes: S4.

1. Initialize the physical simulation environment and construct a rigid body physical collision body based on the convex hull geometry set; S4.

2. Construct the current simulation state based on the current simulation parameters of the simulated human body, the rigid body physical collision body, and the preliminary three-dimensional human-object interaction action sequence; S4.

3. The strategy network directly maps and calculates the target instruction of the PD controller that drives the simulated human body according to the current simulation state; S4.

4. The physics simulation engine receives the target instruction from the PD controller, converts it into driving torque, and performs rigid body dynamics integral calculation by combining the environmental gravity and the friction of the colliding body to generate the next frame of physical reality that conforms to Newtonian mechanics. S4.

5. Repeat steps S4.3 to S4.4 to perform closed-loop integral simulation until the entire length of the reference trajectory is covered, generating a complete physical real-state trajectory.

6. The method for generating text-driven 3D human-object interaction actions based on physical simulation according to claim 5, characterized in that, In step S4.3, the calculation expression for the target instruction of the PD controller is: In the formula, For PD controller target instructions; Represents the Transformer policy network; current simulation state. This includes the perceived state of the simulated human body and object poses, the nearest distance vector from the human body joints to the convex hull of the object surface, and the force-sensing contact markers. The target state for future frames extracted from the initial action sequence; This represents the policy network weights.

7. The method for generating text-driven 3D human-object interaction actions based on physical simulation according to claim 1, characterized in that, Step S5 is as follows: Human joint rotation data and object 6DoF pose data are extracted frame by frame from the physical real-state trajectory and converted into a target 3D human-object interaction action sequence in standard animation format.

8. The method for generating text-driven 3D human-object interaction actions based on physical simulation according to claim 1, characterized in that, In step S6, the internal parameters The update logic is as follows: In the formula, Generate internal parameters for the model of the text-driven interactive action; This represents the loss of dynamic consistency. This is a preliminary 3D human-object interaction sequence; This is the learning rate.

9. A system for implementing the text-driven 3D human-object interaction action generation method based on physical simulation as described in any one of claims 1-8, characterized in that, The system includes: Multimodal geometry preprocessing module: used to acquire input text description information and object geometry information of interactive objects, wherein the object geometry information includes point cloud data and a set of convex hull geometry; Text-driven action generation module: used to input the text description information and point cloud data into a pre-trained text-driven interactive action generation model to generate a preliminary three-dimensional human-object interaction action sequence based on kinematics; Rigid body dynamics solution module: used to input the preliminary 3D human-object interaction action sequence as a reference trajectory, together with the convex hull geometry set, into the pre-trained physical simulation correction model; using the policy network learned by the physical simulation correction model, the physical engine drives the human joints to perform realistic force and collision calculations in the physical simulation environment to obtain the physical real state trajectory. Action sequence output module: used to extract the target's three-dimensional human-object interaction action sequence based on the physical real state trajectory; Differentiable Agent Evolution Module: This module uses the physical simulation correction model as a differentiable physical agent to calculate the dynamic consistency loss between the target 3D human-object interaction action sequence and the physical reality trajectory. It then uses a gradient backpropagation mechanism to penetrate the differentiable physical agent and update the internal parameters of the text-driven interaction action generation model. .

Images