Text-driven human-object interaction action animation generation method and device and medium

Abstract

The invention discloses a text-driven human-object interaction action animation generation method and device and a medium, and the method comprises the steps: constructing an interaction distance field: representing a dynamic space relation between a human body joint point and an object key point in an interaction process through calculating an IDF matrix between the human body joint point and the object key point; constructing a human-object interaction action generation diffusion model; inputting a multi-modal condition into a Transform network after the multi-modal condition is embedded through MLP, and generating a semantic matching interaction action through space-time dependence modeling; constructing a human-object relationship diffusion model: taking the IDF matrix as input, introducing a space-time self-attention mechanism, capturing a local interaction relationship, time dependency and an inter-frame dependency relationship, and further modeling a dynamic interaction mode of a human body and an object; and guiding human-object interaction action generation based on the IDF matrix. According to the method, through joint modeling of the action and the interaction relation, the physical rationality and the motion space-time coherence of the interaction action are remarkably improved, and the method has the technical advantages in the aspects of animation generation precision and controllability.

Description

Technical Field

[0001] The present invention relates to the field of human and movable object motion generation, and particularly to a text-driven human-object interaction motion animation generation method, device, and medium. Background Art

[0002] Human-object interaction (HOI) synthesis is crucial for creating immersive and realistic experiences in applications such as virtual reality, computer animation, and robotics. A vivid HOI sequence typically consists of a series of continuous actions involving an object, such as picking up an object while walking, putting down an object, or adjusting its position with the foot. During this process, the actions of the human body and the object need to be precisely synchronized, and the contact points must match accurately. Achieving such a realistic HOI requires not only generating natural body movements but also deeply understanding the spatio-temporal relationship between the human body and the manipulated object to ensure the physical rationality and visual coherence of the interaction.

[0003] In recent years, the remarkable success of diffusion models has inspired numerous studies on applying them to HOI generation. Some studies have attempted to incorporate prior information into the motion generation model, such as fine-grained text descriptions, physical forces, hand joint positions, and possible contact maps. At the same time, some studies have introduced guiding mechanisms to impose additional constraints on the diffusion model, thereby generating more physically plausible actions. According to different representation methods of the human-object relationship, existing methods can be roughly divided into three categories: contact-based methods, motion-based methods, and distance-based methods. Contact-based methods usually select a fixed set of human joints and predict the contact probability between these joints and the corresponding vertices on the object through a generation module. Motion-based methods, on the other hand, focus on the translational and rotational changes between human joints and the object and use this information to correct the motion trajectory of the object. In contrast, distance-based methods calculate the distance between specific vertices of the human body and the object and use this to supervise or adjust the generated actions to make the interaction more precise.

[0004] During the inference process, most studies have adopted different forms of guiding strategies, such as distance-based constraints and spatial alignment techniques, to optimize the distribution of the output actions and make them closer to the expected interaction distance. However, although these methods have solved the problem of representing the human-object relationship to a certain extent, they mainly rely on a single perspective, that is, only focusing on the relationship between human joints and the nearest object vertices, and have not fully considered the overall human-object spatial relationship. This limitation may affect the model's ability to capture complex HOI details, thereby reducing the quality of the generated results.

[0005] In addition, it is very difficult to effectively represent the geometric shape of an object, which hinders the accurate calculation of the spatio-temporal relationship (such as spatial distance) between a person and an object. Previous methods usually simplify the object representation by the centroid of the object or the closest point to the human body, thus ignoring the overall geometric complexity of the object. And directly using all the points on the object surface is also impractical due to the substantial increase in computational cost. Secondly, due to the high-dimensional and dynamic characteristics of HOI, designing a model that can learn these relationships is inherently challenging. Capturing subtle and context-related interactions requires complex modeling techniques that can effectively combine spatial geometry with temporal dynamics. Summary of the Invention

[0006] To at least to some extent solve one of the technical problems existing in the prior art, an object of the present invention is to provide a text-driven human-object interaction action animation generation method, device and medium.

[0007] The first technical solution adopted by the present invention is as follows:

[0008] A text-driven human-object interaction action animation generation method includes the following steps:

[0009] Step 1, construct an interaction distance field: by calculating the IDF matrix between human body joint points and object key points, characterize the dynamic spatial relationship between the two during the interaction process;

[0010] Step 2, construct a human-object interaction action generation diffusion model: respectively input multi-modal conditions into the Transformer network after being embedded by MLP, and generate semantically matching interaction actions through spatio-temporal dependence modeling; the multi-modal conditions include a noisy motion sequence, a time step, a text description, object key points and human body joints;

[0011] Step 3, construct a human-object relationship diffusion model: use the IDF matrix as the input, introduce a spatio-temporal self-attention mechanism, capture local interaction relationships, time dependencies and inter-frame dependence relationships, and then model the dynamic interaction mode between the human body and the object;

[0012] Step 4, guide the generation of human-object interaction actions based on the IDF matrix: during the inference process, first generate a preliminary motion sequence and calculate its IDF matrix; input the IDF matrix into the distance matrix diffusion model to generate an optimized IDF matrix; adjust the IDF matrix and indirectly iteratively optimize the preliminary motion sequence, and finally output an optimized motion sequence that conforms to the characteristics of the real interaction space.

[0013] Furthermore, the sampling method of the object key point set in step 1 includes: calculating the axis-aligned bounding box of the object's three-dimensional mesh, and extracting the minimum distance points of the 8 vertices of the bounding box on the object's surface; using the Poisson disk sampling algorithm to uniformly select 16 supplementary points on the object's surface; finally, forming the object key point set with these 24 points.

[0014] Furthermore, the interactive distance field in step 1 is calculated from the Euclidean distance between the object key points and the human joint points, and a 24×24 spatio-temporal distance matrix is generated for each frame; the human joint points adopt 24 key points in the SMPL-X model, including 22 main joint points and the left and right palm center points.

[0015] Furthermore, the input of the human-object interaction action generation diffusion model in step 2 includes the noisy motion parameters, object motion parameters, text feature embeddings, and the object sampling point coordinates of the initial frame and the human joint point coordinates in the T-standard pose; among them, the noisy motion parameters include the global positions of the human joints and the 6D continuous rotation parameters, and the object motion parameters include the centroid position, relative rotation, and the global coordinates of the sampling points; the text feature embeddings are extracted by the CLIP text encoder; the model uses Transformer as the denoising network and outputs the denoised human motion data.

[0016] Furthermore, in the training process of the human-object interaction action generation diffusion model in step 2, the motion reconstruction loss and the IDF matrix matching loss are jointly optimized; the IDF matrix matching loss uses the mean square error to measure the generated IDF matrix D pr and the real IDF matrix D gt The difference between them:

[0017]

[0018] In the formula, and respectively represent the human joints and object key points obtained in the generated motion sequence, and represent the real human joints and object key points from the real motion sequence.

[0019] Furthermore, the overall structure of the human-object relationship diffusion model in step 3 is as follows: first, compress the 24×24 IDF matrix to 4×4 dimensions through a linear dimensionality reduction module; then this matrix enters the cascaded Transformer blocks; each Transformer block consists of a temporal attention layer, a spatial attention layer, and a feed-forward network layer, and layer normalization is performed before each layer's calculation; the noise embedding and the text embedding are added to form an embedding vector, which is incorporated into the layer normalization process of the Transformer block to enhance the model's spatio-temporal dependence modeling ability.

[0020] Furthermore, the human-object relationship diffusion model designs a dynamic modulation mechanism: the noise embedding vector at diffusion time step t is added to the CLIP text semantic embedding vector to form a joint conditional vector, and then two modulation parameters c scale and c shift are generated through a modulation network; these parameters are used to dynamically adjust the layer normalization operation, and its formula can be expressed as:

[0021] adaLN(h, c) = c scale LayerNorm(h) + c shift ,

[0022] where h is the input feature of the current layer, enabling the model to adaptively adjust the feature distribution according to the text semantics and the state of the diffusion process;

[0023] In the training stage, the model reconstructs the real IDF matrix D0 from the noise input D t , and its loss function is defined as follows:

[0024]

[0025] where R(·) is the human-object relationship denoising network, and c is the conditional vector that fuses text, object key point features, and human T-pose features.

[0026] Furthermore, in the motion generation inference process of the guidance optimization mechanism in step 4, by applying the IDF matrix guidance only in the last ten time steps of the denoising process and using the gradient optimization algorithm to iteratively adjust the preliminary motion sequence, the difference between the IDF matrix D corresponding to the generated motion sequence and the optimized IDF matrix output by the distance matrix diffusion model is minimized, and its guidance loss function is defined as:

[0027]

[0028] The second technical solution adopted by the present invention is:

[0029] An electronic device, the electronic device includes a processor and a memory, and at least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a text-driven human-object interaction action animation generation method as described above.

[0030] The third technical solution adopted by the present invention is:

[0031] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a text-driven human-object interaction action animation generation method as described above.

[0032] The fourth technical solution adopted by the present invention is:

[0033] A computer program product or a computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the above-mentioned text-driven human-object interaction action animation generation method.

[0034] The beneficial effects of the present invention are as follows: By jointly modeling actions and interaction relationships, the present invention significantly improves the physical rationality and motion spatio-temporal coherence of interaction actions, and has technical advantages in terms of animation generation accuracy and controllability. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present invention or the prior art. It should be understood that the drawings introduced below are only for conveniently and clearly presenting some embodiments of the technical solutions in the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative efforts.

[0036] Figure 1 For the FullBodyManipulation dataset objects and sampled object surface key points in the embodiments of the present invention.

[0037] Figure 2 For the structure diagram of the human-object relationship diffusion model in the embodiments of the present invention.

[0038] Figure 3 For the structure diagram of the text-driven human-object interaction generation method model in the embodiments of the present invention.

[0039] Figure 4 For the step flowchart of a text-driven human-object interaction action animation generation method in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] Embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application and should not be construed as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0041] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms “a”, “the” and “said” used in the embodiments of the present application and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. Additionally, unless otherwise clearly defined, words such as “set up”, “installed”, “connected” should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0042] In the description of the present application, it should be understood that for the orientation description, such as the orientation or positional relationship indicated by up, down, front, back, left, right, etc., is based on the orientation or positional relationship shown in the accompanying drawings. It is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present application.

[0043] In the description of the present application, the meaning of “a number of” is one or more, the meaning of “a plurality of” is two or more, “greater than”, “less than”, “exceeding” etc. are understood as not including the number itself, and “above”, “below”, “within” etc. are understood as including the number itself. If there is a description of “first” and “second”, it is only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or the sequence relationship of the indicated technical features.

[0044] In the description of the present application, “and / or” describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character “ / ” generally represents an “or” relationship between the preceding and following associated objects.

[0045] For the text-driven human-object interaction action generation task, the present invention proposes a text-driven "human-object interaction" (HOI) action generation framework ROG based on the diffusion model. This method improves the realism of action generation by modeling the inherent spatio-temporal relationships of HOI. To achieve efficient object representation, we select key points that are boundary-sensitive and rich in geometric details from the object mesh to ensure a comprehensive description of the object's geometric structure. We use this representation to construct an Interaction Distance Field (IDF) to capture the robust dynamic characteristics of HOI. In addition, we propose a relational diffusion model that integrates spatio-temporal attention mechanisms to enable it to deeply understand complex HOI associations. The model guides the motion generation process to produce relation-aware and semantically aligned actions by optimizing the IDF features of the generated actions. Experimental evaluations show that ROG significantly outperforms the existing state-of-the-art methods in terms of the realism and semantic accuracy of HOI generation.

[0046] Embodiment 1

[0047] As Figure 4 shown, this embodiment provides a text-driven human-object interaction action animation generation method, which specifically includes the following steps:

[0048] Step S1, construct an Interaction Distance Field (IDF). Based on the geometric features of the object's three-dimensional mesh, extract a set of key points that cover the overall contour and local structure of the object; establish a dynamic mapping relationship between the human body joint points and the object key points, and quantify the spatial evolution law of the interaction behavior of the human body contacting and approaching the object surface through the spatio-temporal distance matrix calculation of consecutive time steps; the IDF matrix represents the dynamic proximity degree of the human body and the object in the three-dimensional space, providing fine-grained spatial constraints for action generation.

[0049] See Figure 1, As an alternative implementation, the core objective of constructing the interaction distance field matrix in step S1 is to accurately represent the three-dimensional geometric shape and structural features of an object by efficiently sampling key points of the object, thereby providing fine-grained spatial constraints for generating human-object interaction actions. First, an axis-aligned bounding box (AABB) is defined based on the geometric distribution of the object's three-dimensional mesh. The vertex coordinates of this bounding box are determined by the minimum and maximum values of the object along the coordinate axes in three-dimensional space, which can quickly enclose the overall spatial range of the object. To capture the boundary features of the object, the minimum distance points from the object's surface to the eight vertices of the AABB are calculated, obtaining 8 initial key points. These points can effectively reflect the external contour and spatial extension characteristics of the object. To further cover the local details of the object, the Poisson disk sampling algorithm is used to generate 16 uniformly distributed supplementary key points on the object's surface. This algorithm ensures that the distance between any two points is not less than a preset threshold through probabilistic sampling, thus avoiding over-aggregation of key points. The final set of 24 key points (8 boundary points + 16 local points) represents the morphological characteristics of the object, denoted as P = {P1, P2,..., P 24}. Through the above method, for each object's three-dimensional mesh, we can obtain a set of 24 key points, which are not only representative of the object's shape and structure but also can provide accurate position information of the object in three-dimensional space. In terms of the human body, a bone topology is constructed based on 24 joint points of the SMPL-X parametric human model, including 22 main joint points such as the spine and limbs, as well as the left and right palm center points. These joint points completely cover the physical structure of the human motion chain, denoted as Q = {Q1, Q2,..., Q 24}. Then, for each frame of the motion sequence, a 24×24 IDF matrix is dynamically generated by calculating the three-dimensional Euclidean distance between the 24 key points of the object and the 24 joint points of the human body. The value of each element d ij in the matrix is determined by the following formula:

[0050]

[0051] In the formula, (x i , y i , z i ) are the three-dimensional coordinates of the object point P i , and (x j , y j , z j ) are the three-dimensional coordinates of the human joint point Q j . This matrix not only quantifies the spatial proximity between the human body and the object within a single frame but also reveals the spatio-temporal evolution laws of behaviors such as contact and separation during the interaction process through cumulative calculations over consecutive time steps, providing physical constraints for the subsequent diffusion model.

[0052] Step S2: Construct a human-object interaction action generation diffusion model. Use Transformer as the denoising network architecture. The input includes the fused vector of the noisy motion sequence features, the independent embedding vectors of the object key points and the human body joint points, the time-step noise embedding, and the text semantic embedding. The spatio-temporal dependence of the motion sequence is modeled through multi-modal feature concatenation and spatio-temporal attention mechanism; the output is restored to the original motion data dimension through a projection layer to generate interaction actions with semantic matching.

[0053] In one embodiment, the core objective of the human-object interaction action generation diffusion model constructed in step S2 is to predict the human body and object motion trajectories that conform to the text semantics through multi-modal feature fusion and spatio-temporal modeling. At the data representation level, the motion data of the human body and the object are uniformly encoded as a time series m0 ∈ R N×D , where N represents the total number of frames and D represents the data dimension. Specifically, the human body motion parameters are constructed based on the SMPL-X parameterization model, including the global joint positions (representing the positions of each joint in the world coordinate system) and the 6D continuous rotation parameters (avoiding the gimbal lock problem of traditional Euler angles), and can accurately reconstruct the three-dimensional human body mesh model from the pose and shape parameters. For the representation of object motion, three-part composite data is used: the global 3D position of the object centroid (reflecting the overall movement trajectory of the object), the relative rotation parameter (describing the pose change of the object relative to the initial geometric template), and the global coordinates of the object surface sampling points (dynamically capturing the morphological details of the object).

[0054] The text semantic information, as the key conditional signal for action generation, extracts the high-dimensional semantic embedding vector through the CLIP text encoder, mapping the natural language description to the latent space aligned with the motion data. To further strengthen the model's understanding of the human-object interaction relationship, two types of auxiliary conditional inputs are introduced: one is the sampling point coordinates of the object in the initial frame, as the static benchmark of the object shape, and the other is the joint point coordinates of the human body in the T-standard pose, defining the topological structure of the skeletal system.

[0055] At the model architecture level, a generation framework based on the diffusion probability model is adopted, including a forward noise diffusion and a backward denoising process. In the forward process, Gaussian noise is gradually added to the clean motion data m0 through a Markov chain, and the completely noised data m is obtained after T steps of diffusion T; In the reverse process, the original data distribution is gradually restored by training the denoising network. The core innovation of this model lies in using Transformer as the backbone of the denoising network. Its input is composed of concatenated multi-modal features, including the temporal feature vector obtained by linearly projecting the noisy motion sequence, the independent embedding vectors of object sampling points and human joint points, and the fused vector of the noise time step embedding and the CLIP text embedding. Through the multi-head self-attention mechanism, Transformer can simultaneously capture the spatio-temporal dependencies of the motion sequence, including the coherent action evolution in the time dimension and the joint co-motion in the space dimension, as well as cross-modal semantic associations, such as the correspondence between text keywords and specific joint motions. Finally, the network output is restored to the original motion data dimension through a linear projection layer, generating an interactive action that conforms to the semantic description.

[0056] In terms of the training strategy, in addition to the conventional reconstruction loss, the designer designs a human-object interaction distance field (IDF) matching loss function to strengthen the spatial rationality of the generated actions. This loss function measures the difference between the IDF matrix D calculated from the generated actions and the real IDF matrix D by using the mean square error: pr and gt respectively represent the predicted human joint and object key point coordinates obtained from the predicted motion sequence

[0057]

[0058]

[0059] where, and represent the predicted human joints and object key points obtained from the predicted motion sequence respectively, and represent the real human joints and object key points from the real motion sequence m. D(·) is the function for calculating the IDF matrix.

[0060] Step S3: Construct a human-object relationship diffusion model. Taking the noisy IDF matrix as the input, compress the data dimension through a linear dimensionality reduction module, and adopt cascaded Transformer blocks combined with time-space self-attention mechanism to capture the inter-frame dependency relationships and interaction patterns; fuse the noise embedding, text embedding, object features and human skeleton features, and output the denoised IDF matrix.

[0061] In one embodiment, the core function of the human-object relationship diffusion model constructed in step S3 is to reconstruct the real human-object interaction spatial relationship from the noisy interaction distance field (IDF) matrix through spatio-temporal feature learning and conditional guidance. As Figure 2 shown, Figure 2 is a schematic diagram of the network architecture. At the time step t of the diffusion process, the noisy IDF matrix D is input.t ∈R 24×24×N (where N is the number of frames in the sequence), the model's goal is to generate an IDF matrix D0 that matches the real interaction behavior through denoising. To balance computational efficiency and feature expression ability, the model first performs linear dimensionality reduction on the original IDF matrix: the 24×24 spatial dimension is compressed to 4×4 through two fully connected networks, reducing the total feature dimension from 576 to 16. This operation not only significantly reduces the computational complexity but also breaks through the prior constraints of traditional methods that rely on predefined human body parts or object region partitioning, enabling the model to adaptively focus on dynamically significant features related to interactions.

[0062] The dimensionality-reduced IDF tensor is concatenated with two types of static conditional information: one is the coordinates of the sampling points of the object's initial frame encoded by an MLP, and the other is the coordinates of the human body's T-standard pose joint points extracted by an encoding network. The concatenated feature sequence is input into a cascaded Transformer module for spatio-temporal joint modeling. Each Transformer block consists of three core components: First, the multi-head temporal attention layer calculates the inter-frame dependencies along the time dimension N to capture the dynamic evolution law of the contact state during the interaction process; second, the multi-head spatial attention layer models the correlation between the human body region and the local structure of the object on the compressed 4×4 spatial grid; finally, the feed-forward network layer further enhances the feature expression ability through non-linear transformation.

[0063] To effectively fuse multi-modal conditional signals, the model designs a dynamic modulation mechanism: the noise embedding vector at the diffusion time step t is added to the CLIP text semantic embedding vector to form a joint conditional vector, and then two modulation parameters c scale and c shift are generated through a modulation network (composed of a fully connected layer and an activation function). These parameters are used to dynamically adjust the layer normalization operation, and its formula can be expressed as:

[0064] adaLN(h,c)=c scale LayerNorm(h)+c shift ,

[0065] where h is the input feature of the current layer, enabling the model to adaptively adjust the feature distribution according to the text semantics and the state of the diffusion process.

[0066] During the training phase, the model reconstructs the real IDF matrix D0 from the noise input D t , and its loss function is defined as follows:

[0067]

[0068] Among them, R(·) is a human-object relationship denoising network, and c is a conditional vector that fuses text, object key point features, and human T-pose features. Through this design, the model utilizes multi-modal conditions during the diffusion process to model the human-object interaction distance field.

[0069] Step S4: Generate human-object interaction actions based on IDF. During the inference process, first generate a preliminary motion sequence and calculate its interaction distance field matrix; input the IDF matrix into the distance matrix diffusion model to generate an optimized IDF matrix; through a differentiable guidance loss function, directly adjust the IDF matrix and indirectly perform iterative optimization on the preliminary motion sequence, and finally output an optimized motion sequence that conforms to the characteristics of the real interaction space.

[0070] As an alternative implementation, the complete process of using the distance matrix to guide the generation of human-object interaction actions in step S4 is as Figure 3 shown. The core of this process is to strengthen the physical rationality of the output of the motion generation model through an iterative optimization mechanism: at each time step t of the diffusion denoising process, the motion generation model G first generates an initial motion sequence The sequence generated at this time may generate unrealistic motions, showing unreasonable contact and dynamic characteristics, and these problems will be reflected in the interaction distance field (IDF) during the inference process. To correct these problems, we perform forward kinematics calculations, deduce the three-dimensional coordinates of each joint point of the human body and the spatial positions of the key sampling points of the object, and based on these geometric data, calculate the corresponding interaction distance field matrix D in real time. Then, this IDF matrix D is input into the relationship model R(·), and the relationship model generates an optimized distance field matrix through spatio-temporal feature decoding whose spatial distribution is closer to the constraint conditions of real interaction behaviors.

[0071] To drive the motion sequence to align with the optimized distance field, the guidance loss is defined as follows:

[0072]

[0073] To minimize use a second-order L-BFGS optimizer to perform iterative optimization on the motion sequence The distance field optimization result provided by the relationship model is used as a constraint, and through backpropagation gradients, guide the motion generation model to adjust parameters such as joint rotation and translation, so that the human body posture and object motion gradually satisfy implicit physical laws such as contact stability and force transmission direction.

[0074] Considering that the motion sequences generated in the early stage of diffusion are severely disturbed by noise and it is difficult to guarantee geometric rationality, the guiding mechanism is only activated at the last ten key time steps. At the end of diffusion, the motion sequences already have basic rationality, and applying guidance at this time can significantly improve the detail quality. This selective application not only improves the computational efficiency but also significantly improves the overall quality of the synthesized motion.

[0075] Embodiment 2

[0076] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory. At least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement Figure 4 a text-driven human-object interaction action animation generation method as shown.

[0077] It can be understood that the memory may include a random access memory (RAM), and may also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0078] The processor may include one or more processing cores. The processor uses various interfaces and circuits to connect various parts within the entire server. By running or executing instructions, programs, code sets, or instruction sets stored in the memory, and by invoking data stored in the memory, it performs various functions of the server and processes data. Optionally, the processor may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor may integrate a combination of one or several of a central processing unit (CPU) and a modem, etc. Among them, the CPU mainly processes the operating system, application programs, etc.; the modem is used for processing wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor and may be implemented separately through a single chip.

[0079] Since this electronic device is the electronic device corresponding to a text-driven human-object interaction action animation generation method according to an embodiment of the present invention, and the principle of solving problems by this electronic device is similar to that of this method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and repeated parts will not be elaborated.

[0080] Embodiment 3

[0081] An embodiment of the present invention further provides a computer-readable storage medium, in which at least one instruction, at least one segment of program, code set, or instruction set is stored, and the at least one instruction, the at least one segment of program, the code set, or the instruction set is loaded and executed by a processor to implement Figure 4 a text-driven human-object interaction action animation generation method as shown.

[0082] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable storage medium, which includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), or other optical disc memories, magnetic disk memories, tape memories, or any other computer-readable medium capable of carrying or storing data.

[0083] Since this storage medium is the storage medium corresponding to a text-driven human-object interaction action animation generation method of an embodiment of the present invention, and the principle of solving problems by this storage medium is similar to that of this method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be elaborated.

[0084] Embodiment 4

[0085] In some possible implementation manners, various aspects of the method of the embodiment of the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to cause the computer device to execute the steps of a text-driven human-object interaction action animation generation method according to various exemplary implementation manners described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0086] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following technologies well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0087] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0088] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable those of ordinary skill in the art to understand the content of the present invention and implement it accordingly, and should not be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the essence of the content of the present invention should be covered within the protection scope of the present invention.

Claims

1. A text-driven method for generating human-object interaction action animations, characterized in that, It includes the following steps: Step 1, construct an interactive distance field: By calculating the IDF matrix between human joint points and object key points, the dynamic spatial relationship between the two during the interaction is characterized; Step 2, construct a human-object interaction action generation diffusion model: Embed multi-modal conditions through MLP respectively and then input them into the Transformer network, and generate semantically matched interaction actions through spatio-temporal dependence modeling; Step 3, construct a human-object relationship diffusion model: Using the IDF matrix as the input, introduce a spatio-temporal self-attention mechanism to capture local interaction relationships, time dependencies, and inter-frame dependencies, and then model the dynamic interaction patterns between the human body and the object; Step 4, guide the generation of human-object interaction actions based on the IDF matrix: During the inference process, first generate a preliminary motion sequence and calculate its IDF matrix; input the IDF matrix into the distance matrix diffusion model to generate an optimized IDF matrix; adjust the IDF matrix and indirectly perform iterative optimization on the preliminary motion sequence, and finally output an optimized motion sequence that conforms to the characteristics of the real interaction space.

2. A method for generating a text-driven human-object interaction action animation according to claim 1, wherein The sampling method of the object key point set in Step 1 includes: calculating the axis-aligned bounding box of the object's three-dimensional mesh, and extracting the minimum distance points of the 8 vertices of the bounding box on the object surface; using the Poisson disk sampling algorithm to uniformly select 16 supplementary points on the object surface; finally, these 24 points form the object key point set.

3. A method for generating a text-driven human-object interaction action animation according to claim 1, characterized in that, The interactive distance field in Step 1 is calculated from the Euclidean distance between object key points and human joint points, and a 24×24 spatio-temporal distance matrix is generated for each frame; 24 key points in the SMPL-X model are used for human joint points, including 22 main joint points and the palm center points of the left and right hands.

4. A method for generating a text-driven human-object interaction action animation according to claim 1, characterized in that, The input of the human-object interaction action generation diffusion model described in Step 2 includes noisy motion parameters, object motion parameters, text feature embeddings, and the coordinates of the object sampling points in the initial frame and the coordinates of human joint points in the T-standard pose; among them, the noisy motion parameters include the global position of human joints and 6D continuous rotation parameters, and the object motion parameters include the centroid position, relative rotation, and global coordinates of the sampling points; the text feature embeddings are extracted through the CLIP text encoder; the model uses Transformer as the denoising network and outputs the denoised human motion data.

5. A method for generating a text-driven human-object interaction action animation according to claim 1, characterized in that In step 2, during the training process of the human-object interaction action generation diffusion model, the motion reconstruction loss and the IDF matrix matching loss are jointly optimized; the IDF matrix matching loss uses the mean square error to measure the generated IDF matrix D pr and the real IDF matrix D gt The difference between them: In the formula, and respectively represent the human joints and object key points obtained in the generated motion sequence. and represent the true human joints and object key points from real motion sequences.

6. A method for generating a text-driven human-object interaction action animation according to claim 1, characterized in that, The overall structure of the human-object relationship diffusion model in Step 3 is as follows: First, compress the IDF matrix to 4×4 dimensions through a linear dimensionality reduction module; then this matrix enters the cascaded Transformer blocks; each Transformer block consists of a temporal attention layer, a spatial attention layer, and a feed-forward network layer, and layer normalization is performed before each layer's calculation; the noise embedding and the text embedding are added together to form an embedding vector, and it is merged into the layer normalization process of the Transformer block to enhance the model's spatio-temporal dependence modeling ability.

7. A method for generating a text-driven human-object interaction action animation according to claim 6, characterized in that, The human-object relationship diffusion model designs a dynamic modulation mechanism: the noise embedding vector at diffusion time step t is added to the CLIP text semantic embedding vector to form a joint conditional vector, and then two modulation parameters c scale and c shift are generated through a modulation network; these parameters are used to dynamically adjust the layer normalization operation, and its formula can be expressed as: adaLN(h, c) = c scale LayerNorm(h) + c shift , In the formula, h is the input feature of the current layer, enabling the model to adaptively adjust the feature distribution according to the text semantics and the state of the diffusion process; During the training phase, the model takes the noisy input D t Reconstruct the true IDF matrix D0, and its loss function is defined as follows: Among them, R(·) is the human-object relationship denoising network, and c is the conditional vector that fuses text, object key point features, and human T-pose features.

8. A method for generating a text-driven human-object interaction action animation according to claim 1, characterized in that In step 4, the guiding optimization mechanism, during the motion generation inference process, guides by applying the IDF matrix only in the last ten time steps of the denoising process and uses a gradient optimization algorithm to iteratively adjust the preliminary motion sequence, minimizing the difference between the IDF matrix D corresponding to the generated motion sequence and the optimized IDF matrix output by the distance matrix diffusion model. Its guiding loss function is defined as: between, and its guiding loss function is defined as:

9. An electronic device, characterized in that, The electronic device includes a processor and a memory. At least one instruction, at least one program, a code set, or an instruction set is stored in the memory. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, At least one instruction, at least one program, a code set, or an instruction set is stored in the storage medium. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 8.

Images