Method and apparatus for generating motion in two-person interactive modes

Abstract

This disclosure relates to a method and apparatus for generating two-person interactive motion. The method includes: semantically expanding a text description using a large language model, decomposing the text description into an agent description and a receiver description; determining a first text feature based on the agent description, receiver description, and text description; calculating the cosine similarity between the first text feature and text features in a database; determining the two-person interactive motion sequence corresponding to a second text feature with the highest cosine similarity to the first text feature; obtaining a latent space representation; and setting the latent space representation, the second text feature, and the first text feature as conditions for a diffusion model; fusing the conditions with time step information to obtain a modulation vector; denoising the modulation vector to obtain two-person motion features and concatenating them; and generating a two-person interactive motion sequence corresponding to the text description based on the concatenated two-person motion features. This effectively improves the realism and diversity of generated two-person interactive motion.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and 3D human motion generation, and more specifically, to a method and apparatus for generating motion in two-person interactive modes. Background Technology

[0002] Two-person interactive motion generation is a crucial research topic in computer vision and 3D human motion modeling, and a fundamental technology for applications such as human-computer interaction, virtual character animation, immersive games, and filmmaking. The goal of two-person interactive motion generation is to utilize deep learning methods to automatically generate natural, coherent, and physically sound sequences of two-person interactive motion based on given control conditions (e.g., text, actions, scenes). Compared to single-person motion generation, two-person interactive motion generation not only needs to ensure the reasonableness of individual character movements but also needs to capture the interaction relationships, temporal dependencies, and causal connections between the two characters to make the generated interactive actions more closely resemble real-world behavior.

[0003] Existing methods generally employ generative frameworks based on diffusion models or generative adversarial networks, using text prompts as conditional inputs to drive the model to generate corresponding two-person action sequences. However, these methods still have several shortcomings in practical applications. First, two-person interaction datasets are generally small in scale and have limited action categories, making it difficult for models to fully learn diverse interaction patterns, resulting in insufficient generalization ability and often distorted or disjointed results in new scenarios. Second, the text prompts are coarse-grained; existing methods typically treat the entire description as a single conditional input, making it difficult to distinguish the behavioral details of the actor and the receiver, easily leading to role confusion, misaligned actions, or unreasonable interactions. Furthermore, the identities of the actor and receiver often change dynamically over time during two-person interactions, while most existing methods use unidirectional interaction modeling strategies, failing to explicitly model identity switching, resulting in a lack of continuity and naturalness in the generated action sequences.

[0004] On the other hand, existing methods do not make sufficient use of contextual information and external knowledge, relying only on the limited conditions provided by the training set and input text, making it difficult to accurately reproduce complex interaction scenarios. For example, when the input description is semantically ambiguous or the scene exceeds the training distribution, the model often struggles to generate actions that meet semantic requirements. Furthermore, most current generation methods focus on the visual effects of the generated sequences, but lack sufficient constraints on physical consistency and interaction rationality, easily leading to situations that do not conform to physical laws, such as joint drift, bone stretching, interlacing, or disjointed movements, affecting the realism and usability of the generated sequences. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure proposes a method, apparatus, computing system, and computer-readable storage medium for generating two-person interactive motion by combining external retrieval enhancement, fine-grained text prompts, and identity-sensitive interaction modeling.

[0006] According to one aspect of this disclosure, a method for generating two-person interactive motion is provided. The method includes: semantically expanding an input text description using a large language model; decomposing the expanded text description into an agent description and a receiver description; determining a first text feature based on the agent description, receiver description, and the input text description using a text encoder; calculating the cosine similarity between the first text feature and text features in a two-person interactive motion database; determining a two-person interactive motion sequence corresponding to a second text feature with the highest cosine similarity to the first text feature from the two-person interactive motion database; obtaining a latent space representation of the two-person interactive motion based on the determined two-person interactive motion sequence; setting the latent space representation, the second text feature, and the first text feature as conditions for a diffusion model; fusing the conditions with time step information using a feature mapper to obtain a modulation vector; denoising the modulation vector using the diffusion model to obtain two-person motion features and concatenating them; and generating a two-person interactive motion sequence corresponding to the input text description using a Mamba interaction module based on the concatenated two-person motion features.

[0007] Optionally, the method for generating two-person interactive motion further includes: acquiring a two-person interactive video and a corresponding text description, and using a human pose estimation algorithm to extract the two-person interactive motion sequence from the two-person interactive video to construct the two-person interactive motion database.

[0008] Optionally, the step of determining the first text feature using a text encoder based on the agent description, the receiver description, and the input text description includes: concatenating the agent description, the receiver description, and the input text description into a text set, and determining the first text feature based on the text set.

[0009] Optionally, the diffusion model includes a Transformer-based denoising network, and the modulation vector is denoised using the Transformer-based denoising network.

[0010] Optionally, the Mamba interaction module uses a bidirectional scanning mechanism to input the spliced ​​two-person motion features in both forward and reverse order. The spliced ​​two-person motion features input in both forward and reverse order are then fused through multiple layers and decoded through a linear layer to generate the two-person interactive motion sequence corresponding to the input text description.

[0011] Optionally, the two-person interactive motion generation method further includes: determining reconstruction loss and multi-constraint loss based on the generated two-person interactive motion sequence corresponding to the input text description, and optimizing the diffusion model based on the reconstruction loss and multi-constraint loss.

[0012] Optionally, the multiple constraint loss includes bone length constraint, masked joint distance map constraint, relative orientation constraint, foot contact constraint, and joint velocity constraint.

[0013] According to another aspect of this disclosure, a two-person interactive motion generation device is provided, the device comprising: a text feature determination unit configured to semantically expand an input text description using a large language model, decompose the expanded text description into an agent description and a receiver description, and determine a first text feature using a text encoder based on the agent description, the receiver description, and the input text description; and a condition setting unit configured to calculate the cosine similarity between the first text feature and text features in a two-person interactive motion database, and determine the element with the highest cosine similarity to the first text feature from the two-person interactive motion database. The second text feature corresponds to the two-person interactive motion sequence. Based on the determined two-person interactive motion sequence, the latent space representation of the two-person interactive motion is obtained, and the latent space representation, the second text feature, and the first text feature are set as conditions for the diffusion model. The denoising unit is configured to fuse the conditions with time step information through a feature mapper to obtain a modulation vector, and use the diffusion model to denoise the modulation vector to obtain the two-person motion features and concatenate them. The motion sequence generation unit is configured to generate a two-person interactive motion sequence corresponding to the input text description based on the concatenated two-person motion features through the Mamba interaction module.

[0014] Optionally, the condition setting unit is further configured to: acquire a two-person interaction video and a corresponding text description, and extract the two-person interaction motion sequence from the two-person interaction video using a human pose estimation algorithm, so as to construct the two-person interaction motion database.

[0015] Optionally, the text feature determination unit is further configured to: concatenate the agent description, the receiver description, and the input text description into a text set, and determine a first text feature based on the text set.

[0016] Optionally, the diffusion model includes a Transformer-based denoising network, and the modulation vector is denoised using the Transformer-based denoising network.

[0017] Optionally, the Mamba interaction module uses a bidirectional scanning mechanism to input the spliced ​​two-person motion features in both forward and reverse order. The spliced ​​two-person motion features input in both forward and reverse order are then fused through multiple layers and decoded through a linear layer to generate the two-person interactive motion sequence corresponding to the input text description.

[0018] Optionally, the motion sequence generation unit is further configured to: determine reconstruction loss and multi-constraint loss based on the generated two-person interactive motion sequence corresponding to the input text description, and optimize the diffusion model based on the reconstruction loss and multi-constraint loss.

[0019] Optionally, the multiple constraint loss includes bone length constraint, masked joint distance map constraint, relative orientation constraint, foot contact constraint, and joint velocity constraint.

[0020] According to another aspect of this disclosure, a computing system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the two-person interactive motion generation method as described above.

[0021] According to another aspect of this disclosure, a computer-readable storage medium is provided for storing instructions, wherein when the instructions are executed by at least one computing device, the at least one computing device causes the at least one computing device to perform the two-person interactive motion generation method as described above.

[0022] By adopting this disclosure, external knowledge can be fully utilized to enhance the model's understanding of complex interactive semantics, and the relationship between individual actions and identity switching can be accurately modeled, ensuring the quality of the generated results in terms of action continuity, interaction rationality and physical consistency, thereby improving the generalization ability and reliability of two-person motion generation in practical application scenarios. Attached Figure Description

[0023] The above and / or other objects and advantages of this disclosure will become clearer from the following description of embodiments in conjunction with the accompanying drawings, wherein:

[0024] Figure 1 This is a flowchart illustrating a two-person interactive motion generation method according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 This is a network structure diagram according to an embodiment of the present disclosure;

[0026] Figure 3 This is a block diagram illustrating a two-person interactive motion generation device according to an exemplary embodiment of the present disclosure;

[0027] Figure 4This is a block diagram illustrating a computing system including at least one computing device and at least one storage device of storage instructions according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0028] The following description, in conjunction with the accompanying drawings, provides specific embodiments to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding this disclosure, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be altered as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0029] This disclosure proposes a method for generating two-person interactive motion based on retrieval enhancement and identity-sensitive interaction. Addressing challenges in two-person interactive motion generation tasks such as data scarcity, complex interactions, and identity swapping, this method designs a generation framework combining retrieval enhancement, fine-grained text prompts, and identity-sensitive interaction. By constructing a two-person interactive motion database and utilizing a large language model to expand tags and decompose the two-person descriptions into fine-grained descriptions of the agent and receiver, the model's understanding of interactive semantics and individual actions is improved. The prompt text, retrieved two-person interactive motion sequences, and corresponding text features are fused as conditions for a diffusion model, combined with a time-step driven denoising process to generate natural and realistic two-person interactive motion sequences. An identity-sensitive Mamba interaction module is used, and a bidirectional scanning mechanism is introduced to model identity interaction, ensuring the continuity and rationality of motion generation. Multiple constraint loss mechanisms are designed, including bone length constraints, masked joint distance map constraints, and relative orientation constraints, to improve the physical consistency and spatial rationality of interactive actions. Therefore, this method effectively enhances the realism and diversity of generated two-person interactive motion and improves generalization ability in complex interaction scenarios.

[0030] Figure 1 This is a flowchart illustrating a two-person interactive motion generation method according to an exemplary embodiment of the present disclosure. Figure 2 This is a network structure diagram according to an embodiment of the present disclosure.

[0031] like Figure 1As shown, in step S101, the input text description is semantically expanded using a large language model. The expanded text description is decomposed into agent description and receiver description (for example, the text description "one person hugs another person from behind, with arms wrapped around the waist" is decomposed into agent description "the person giving the hug wraps their arms around the waist of another person from behind, presenting an intimate and close posture" and receiver description "the person being hugged is hugged from behind, with their waist wrapped around the arms of another person, possibly giving a feeling of being hugged and close to the hugger"). A text encoder is then used to determine the first text feature based on the agent description, receiver description, and input text description. In the example, the agent description, receiver description, and input text description are concatenated into a text set, and the first text feature is determined based on the text set. For example, two-person interaction videos and corresponding text descriptions can be collected, and a human pose estimation algorithm can be used to extract the two-person motion sequences from the videos, forming a two-person interaction motion database with the training set. This enables the model to understand asymmetrical role relationships in interactions, achieving more precise generative control.

[0032] For example, using a pre-trained large language model to process the original interactive text. Expand the text to obtain a more detailed description of the scene and interaction, and break the text down into descriptions of the actors. Description of the recipient The expanded text set is obtained. :

[0033]

[0034] in, This indicates a splicing operation.

[0035] Will The text features are obtained by feeding them into a text encoder:

[0036]

[0037] in, For text encoders, for example, the Llama3 model (such as...) can be used. Figure 2 (As shown).

[0038] In step S102, the cosine similarity between the first text feature and the text features in the two-person interaction motion database is calculated. The two-person interaction motion sequence corresponding to the second text feature with the highest cosine similarity to the first text feature is determined from the database. Based on the determined two-person interaction motion sequence, the latent space representation of the two-person interaction motion is obtained, and the latent space representation, the second text feature, and the first text feature are set as conditions for the diffusion model. In the example, a two-person interaction video and its corresponding text description can be obtained. A human pose estimation algorithm is used to extract the two-person interaction motion sequence from the video to construct a two-person interaction motion database. Constructing this database reduces the generation difficulty and improves the rationality and richness of the actions. In the example, the database consists of a training set and externally collected two-person interaction videos processed by human pose estimation. The database interaction tags are manually or semi-automatically labeled, and a pre-trained large language model is used to expand and unify the style of the tags to improve the consistency of retrieval and alignment. For example, in addition to using the training set to build database D1, we can also collect videos of two-person interactions and use human pose estimation algorithms to obtain joint representations of the two-person motion sequences in the videos, forming a new database D2. We can manually or semi-automatically label the motion sequences with interactive behavior tags and use a large language model to expand the tags to make them conform to the language style of the training set.

[0039] For example, calculating the cosine similarity between text features and text features in a database:

[0040]

[0041] in, For the expanded set of prompt text, Cosine similarity is used to represent the similarity score. All calculated similarity scores are sorted, and the sample with the highest score is selected as the data for retrieval enhancement. A learnable mapping module is then used to obtain the latent space representation of the two-person motion. And, will Text features The splicing is used as a conditional input for the diffusion model.

[0042] In step S103, the set conditions and time step information are fused using a feature mapper to obtain a modulation vector. A diffusion model is then used to denoise the modulation vector to obtain the motion features of the two individuals, which are then concatenated. In this example, the diffusion model includes a Transformer-based denoising network, which is used to denoise the modulation vector. For example, as... Figure 2As shown, the obtained conditions and time step information can be fused after feature mapping, added to the noisy latent space representation of the two-person motion sequence through an adaptive normalization layer, fed into a denoising network, and then modeled by a Transformer module to obtain the two-person motion features and splice them together.

[0043] For example, by fusing conditions and time steps, a modulation vector can be obtained. :

[0044]

[0045] Modulation vector The final condition, after passing through an adaptive normalization layer, is the two-person motion sequence. The input sequence is mapped to a latent space representation through an input mapping layer. The features, along with the conditions, are fed into a Transformer-based denoising network to output dual-person features. Then splice them together to get .

[0046] In step S104, the Mamba interaction module generates a two-person interaction motion sequence corresponding to the input text description based on the concatenated two-person motion features. In the example, the Mamba interaction module uses a bidirectional scanning mechanism to input the concatenated two-person motion features in both forward and reverse order. The concatenated two-person motion features, input in both forward and reverse order, are then fused through multiple iterative layers and decoded through a linear layer to generate the two-person interaction motion sequence corresponding to the input text description. For example, an identity-sensitive Mamba interaction module may include units for cross-person interaction modeling and self-attention units for maintaining individual patterns. A bidirectional scanning mechanism is used to input the motion sequences of individuals a and b in both forward and reverse order. The module output is then fused through multiple iterative layers and decoded through a linear layer to obtain the final two-person motion sequence. For example, the concatenated features are input into the identity-sensitive Mamba interaction module, and a bidirectional scanning mechanism is used to model the two-person interaction motion sequence. The modeled features are then decomposed into... The above process, after N iterations, yields the final representation of the two-person interactive motion features, which is then mapped through a linear layer to obtain the generated two-person interactive motion sequence. .

[0047] Furthermore, based on the generated two-person interactive motion sequence corresponding to the input text description, reconstruction loss and multi-constraint loss can be determined, and the diffusion model can be optimized and trained based on these losses. For example, multi-constraint losses include bone length constraints, masked joint distance map constraints, relative orientation constraints, foot contact constraints, and joint velocity constraints. By introducing multiple constraints, it is ensured that the generated motion is not only visually natural but also physically believable (e.g., no slipping, no joint twisting), guaranteeing that the output motion sequence has good physical consistency and spatial plausibility.

[0048] For example, the reconstruction loss can be represented as follows:

[0049]

[0050] in, , , For noise reduction networks, , Let E represent the actual two-person interactive motion sequence, and λ represent the expected value. t This represents the weight value of the loss. This represents the noise coefficient that varies over time during forward noise addition. , This represents the noise added when two motions are subjected to noise. This indicates the conditions under which control is generated. Constraint adjustment losses include foot contact losses. Joint speed loss Bone length loss Weighted mask joint distance map loss and relative orientation loss etc., including foot contact loss and joint velocity loss Defined in single-person human motion generation work. Bone length loss. Used to constrain the length of the generated human skeleton:

[0051]

[0052] in, Represents the function for calculating bone length. Weighted masked joint distance map loss. Used to constrain the spatial distance between two interacting individuals:

[0053]

[0054] in, This represents a distance mapping between all points between two people. Represents a weighted matrix. It is an indicator function that masks the loss by applying a two-dimensional distance threshold on the XZ plane, activating the loss only when the horizontal distance between two people is small enough.

[0055] Relative orientation loss is used to constrain the relative orientation of the generated human body:

[0056]

[0057] in, This represents the inverse kinematic process, which outputs joint rotation, while This represents the two-dimensional relative orientation of two people around the Y-axis, obtained from rotation.

[0058] Finally, the overall function is represented as follows:

[0059]

[0060] Among them, the weighting coefficients of each loss , , , , This can be determined experimentally. During the inference phase, the two-person motion sequence can be initialized as random noise, and the denoising diffusion implicit model (DDIM) strategy can be used to recover the human motion sequence.

[0061] The dual-person interactive motion generation method according to the exemplary embodiments of this disclosure introduces a retrieval enhancement mechanism into the diffusion model conditional generation framework. It expands and decomposes the dual-person interactive motion text through a pre-trained large language model to provide fine-grained cues for the agent and the recipient. Combined with externally retrieved dual-person interactive motion sequences as prior conditions input to the diffusion model, it improves the model's understanding of interactive semantics and the diversity of generation. By adaptively normalizing and modulating conditional features, it guides the denoising process to obtain high-quality latent features. Then, through an identity-sensitive Mamba interaction module and a bidirectional scanning mechanism, it explicitly models the identity swapping relationship, so that the generated dual-person interactive motion remains natural and coherent in the time dimension. Furthermore, it introduces multiple physical constraint losses such as bone length, masked joint distance map, and relative orientation to improve the physical consistency and spatial rationality of the generated actions.

[0062] Figure 3 This is a block diagram illustrating a two-person interactive motion generation apparatus according to an exemplary embodiment of the present disclosure.

[0063] like Figure 3As shown, the dual-person interactive motion generation apparatus 300 according to an exemplary embodiment of the present disclosure includes: a text feature determination unit 301, configured to semantically expand the input text description using a large language model, decompose the expanded text description into an agent description and a receiver description, and determine a first text feature using a text encoder based on the agent description, the receiver description, and the input text description; and a condition setting unit 302, configured to calculate the cosine similarity between the first text feature and text features in a dual-person interactive motion database, and determine the first text feature with the highest cosine similarity from the dual-person interactive motion database. The two-person interactive motion sequence corresponding to the two text features is used to obtain the latent space representation of the two-person interactive motion based on the determined two-person interactive motion sequence, and the latent space representation, the second text feature, and the first text feature are set as conditions for the diffusion model; the denoising unit 303 is configured to fuse the conditions and time step information through the feature mapper to obtain the modulation vector, and use the diffusion model to denoise the modulation vector to obtain the two-person motion features and concatenate them; the motion sequence generation unit 304 is configured to generate a two-person interactive motion sequence corresponding to the input text description based on the concatenated two-person motion features through the Mamba interaction module.

[0064] In the example, the condition setting unit 302 is further configured to: acquire a two-person interaction video and the corresponding text description, and use a human pose estimation algorithm to extract the two-person interaction motion sequence in the two-person interaction video in order to construct a two-person interaction motion database.

[0065] In the example, the text feature determination unit 301 is further configured to: concatenate the agent description, the receiver description and the input text description into a text set, and determine the first text feature based on the text set.

[0066] In the example, the diffusion model includes a Transformer-based denoising network, and the modulation vector is denoised using the Transformer-based denoising network.

[0067] In the example, the Mamba interaction module uses a bidirectional scanning mechanism to input the spliced ​​two-person motion features in both forward and reverse order. The spliced ​​two-person motion features, input in both forward and reverse order, are then fused through multiple layers and decoded through a linear layer to generate a two-person interaction motion sequence corresponding to the input text description.

[0068] In the example, the motion sequence generation unit 304 is further configured to: determine the reconstruction loss and the multi-constraint loss based on the generated two-person interactive motion sequence corresponding to the input text description, and optimize the diffusion model based on the reconstruction loss and the multi-constraint loss.

[0069] In the example, the multi-constraint loss includes bone length constraint, masked joint distance map constraint, relative orientation constraint, foot contact constraint, and joint velocity constraint.

[0070] The above combination Figures 1 to 2 The specific operations shown are respectively by Figure 3 The corresponding unit in the illustrated two-person interactive motion generation device 300 performs the operation; specific operational details will not be elaborated here. By employing the two-person interactive motion generation device according to the exemplary embodiments of this disclosure, external knowledge can be fully utilized to enhance the model's understanding of complex interactive semantics, and the relationship between individual actions and identity switching can be accurately modeled, ensuring the quality of the generated results in terms of action continuity, interaction rationality, and physical consistency.

[0071] Figure 4 This is a block diagram illustrating a computing system including at least one computing device and at least one storage device of storage instructions according to an exemplary embodiment of the present disclosure.

[0072] like Figure 4 As shown, the computing system 400 according to an exemplary embodiment of the present invention includes a computing device 401 and a storage device 402. The storage device 402 stores computer-executable instructions. When the computer-executable instructions are executed by the computing device 401, the two-person interactive motion generation method described in any of the foregoing embodiments is executed.

[0073] The computing device 401 can be deployed in a server or client, or on a node device in a distributed network environment. Furthermore, the computing device 401 can be a PC, tablet, personal digital assistant, smartphone, web application, or other device capable of executing the aforementioned set of instructions. Here, the computing device is not necessarily a single computing device; it can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The computing device can also be part of an integrated control system or system manager, or can be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission). In the computing device, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor also includes analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0074] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores instructions, which, when executed by at least one computing device, cause the at least one computing device to perform the two-person interactive motion generation method described in any of the foregoing embodiments. The computer-readable storage medium includes magnetic media such as floppy disks and magnetic tapes, optical media (including optical disc (CD) ROMs and DVD ROMs), magneto-optical media such as floppy discs, hardware devices such as ROMs and RAMs designed for storing and executing program commands, and flash memory. The instructions may include language code executable by a computer using an interpreter and machine language code generated by a compiler.

[0075] By adopting this disclosure, external knowledge can be fully utilized to enhance the model's understanding of complex interactive semantics, and the relationship between individual actions and identity switching can be accurately modeled, ensuring the quality of the generated results in terms of action continuity, interaction rationality and physical consistency, thereby improving the generalization ability and reliability of two-person motion generation in practical application scenarios.

[0076] The processes, methods, or algorithms disclosed herein can be transmitted to, or implemented by, a processing device, controller, or computer, which may include any existing programmable electronic control unit or a dedicated electronic control unit. Similarly, the processes, methods, or algorithms can be stored in various forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media (such as ROM devices) and information variablely stored on writable storage media (such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media). The processes, methods, or algorithms can also be implemented in a software executable object. Optionally, the processes, methods, or algorithms can be implemented wholly or partially using suitable hardware components (such as ASICs, FPGAs, state machines, controllers, or other hardware components or devices) or a combination of hardware components, software components, and firmware components.

[0077] Although this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered merely for descriptive purposes and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner and / or if components in the described system, architecture, apparatus, or circuit are replaced or supplemented with other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents shall be construed as included in this disclosure.

Claims

1. A method for generating motion in two-person interactive interactions, characterized in that, The method for generating motion in two-person interactive interactions includes: The large language model is used to semantically expand the input text description. The expanded text description is decomposed into agent description and patient description. The text encoder is used to determine the first text feature based on the agent description, patient description and input text description. Calculate the cosine similarity between the first text feature and the text features in the two-person interactive motion database. Determine the two-person interactive motion sequence corresponding to the second text feature with the highest cosine similarity to the first text feature from the two-person interactive motion database. Obtain the latent space representation of the two-person interactive motion based on the determined two-person interactive motion sequence. Set the latent space representation, the second text feature, and the first text feature as conditions for the diffusion model. The condition and time step information are fused by a feature mapper to obtain a modulation vector. The modulation vector is then denoised using the diffusion model to obtain the motion features of the two people and then concatenated with them. The Mamba interaction module generates a two-person interaction motion sequence corresponding to the input text description based on the spliced ​​two-person motion features. The Mamba interaction module employs a bidirectional scanning mechanism to input the spliced ​​two-person motion features in both forward and reverse order. These spliced ​​motion features, input in both forward and reverse order, are then iteratively fused over multiple layers and decoded through a linear layer to generate the two-person interactive motion sequence corresponding to the input text description. The step of determining the first text feature using a text encoder based on the agent description, the receiver description, and the input text description includes: concatenating the agent description, the receiver description, and the input text description into a text set, and determining the first text feature based on the text set.

2. The dual-person interactive motion generation method according to claim 1, wherein The method for generating two-person interactive motion further includes: acquiring a two-person interactive video and a corresponding text description, and using a human pose estimation algorithm to extract the two-person interactive motion sequence from the two-person interactive video in order to construct the two-person interactive motion database.

3. The dual-person interactive motion generation method according to claim 1, wherein The diffusion model includes a Transformer-based denoising network, and the modulation vector is denoised using the Transformer-based denoising network.

4. The dual-person interactive motion generation method according to claim 1, wherein The method for generating two-person interactive motion further includes: determining reconstruction loss and multi-constraint loss based on the generated two-person interactive motion sequence corresponding to the input text description, and optimizing the diffusion model based on the reconstruction loss and multi-constraint loss.

5. The dual-person interactive motion generation method according to claim 4, wherein The multiple constraint loss includes bone length constraint, masked joint distance map constraint, relative orientation constraint, foot contact constraint, and joint velocity constraint.

6. A two-person interactive motion generation apparatus characterized by comprising: The dual-person interactive motion generation device includes: The text feature determination unit is configured to semantically expand the input text description using a large language model, decompose the expanded text description into agent description and receiver description, and use a text encoder to determine the first text feature based on the agent description, receiver description and input text description. a condition setting unit configured to calculate a cosine similarity between the first text feature and text features in a two-person interaction motion database, determine a second text feature corresponding to a two-person interaction motion sequence with the highest cosine similarity to the first text feature from the two-person interaction motion database, obtain a latent space representation of the two-person interaction motion based on the determined two-person interaction motion sequence, and set the latent space representation, the second text feature, and the first text feature as conditions of a diffusion model; a denoising unit configured to fuse the conditions and time step information through a feature mapper to obtain a modulation vector, denoise the modulation vector using the diffusion model to obtain two-person motion features, and concatenate the two-person motion features; a motion sequence generation unit configured to generate a two-person interaction motion sequence corresponding to the input text description based on the concatenated two-person motion features through a Mamba interaction module, wherein the Mamba interaction module uses a bidirectional scanning mechanism to input the concatenated two-person motion features in a forward and reverse order respectively, and the concatenated two-person motion features input in the forward and reverse order are fused through multiple layers of iteration and then decoded by a linear layer to generate the two-person interaction motion sequence corresponding to the input text description, wherein the text feature determination unit is further configured to concatenate the actor description, the recipient description, and the input text description into a text set, and determine the first text feature based on the text set.

7. A computing system comprising at least one computing device and at least one storage device storing instructions, wherein the computing system is configured to perform operations comprising: The instructions, when executed by the at least one computing device, cause the at least one computing device to perform the two-person interaction motion generation method according to any one of claims 1-5.

8. A computer-readable storage medium storing instructions, wherein, The instructions, when executed by the at least one computing device, cause the at least one computing device to perform the two-person interaction motion generation method according to any one of claims 1-5.

Images