A text-driven three-dimensional human motion generation method

Abstract

The present invention discloses a text-driven three-dimensional human motion generation method, which realizes fine-grained human motion generation through a diffusion model and a large language model. First, a text encoder is used to extract multi-layer semantic features of the text, and at the same time, a large language model is used to identify core and non-core body parts; then the human body is divided into five regions: the left arm, the right arm, the left leg, the right leg, and the torso, and the word-level text features and the motion features of each part are fused and refined through a multi-head cross-attention mechanism; an adaptive loss function is used to optimize the motion of the core parts and keep the non-core parts natural and smooth; finally, the diffusion model is used to denoise and generate full-body motion. In addition, this method supports parameter editing and motion completion functions, and can flexibly control the motion amplitude and complete part of the input motion. This method can be applied to the fields of virtual reality, animation production, and human-computer interaction, and the generated motions are natural, accurate, and conform to the text description.

Description

Technical Field

[0001] The present invention relates to a text-driven three-dimensional human motion generation method based on a diffusion model and a large language model, which is a fine-grained human motion generation method and belongs to the technical field of artificial intelligence-generated content. Background Art

[0002] With the continuous development of fields such as virtual reality, animation production, and human-computer interaction, generating fine-grained human motions from text descriptions has become an important requirement for these applications. Text-driven motion generation can significantly improve the efficiency of animation production, enhance the immersion in virtual reality, and make human-computer interaction more natural and intuitive. Existing text-to-motion generation methods are mainly based on generative adversarial networks (GANs), variational autoencoders (VAEs), and diffusion models (Diffusion). However, these methods have significant difficulties in generating complex, fine-grained multi-action motions.

[0003] The main problems of existing methods are, on the one hand, that the human body is usually regarded as a unified whole, ignoring the independent actions of individual body parts. Human motion often consists of multiple body parts performing different atomic actions. This simplified global perspective makes it difficult for the generation model to capture details, and the finally generated motion lacks naturalness and realism. On the other hand, existing models often ignore the details in the text description and only focus on the global semantics, unable to flexibly adjust to small changes in the text. For example, for subtle differences such as "waving the left hand" and "waving the right hand", existing models often generate similar actions, ignoring the key differences described in the text and resulting in a mismatch between the generated result and the input.

[0004] Therefore, the existing technology has limitations in generating fine-grained motions that conform to specific text descriptions, and it is difficult to balance the overall consistency of the whole body and the accuracy of details at the same time. Summary of the Invention

[0005] Object of the Invention: Aiming at the deficiencies of the existing technology in fine-grained human motion generation, the present invention proposes a fine-grained text-driven human motion generation method through the partition processing of the human body and the introduction of a large language model for text semantic analysis, so as to solve the deficiencies of the existing technology in capturing multi-action details and text alignment, and generate more natural, accurate, and text-description-conforming human motions.

[0006] Technical Solution: A text-driven three-dimensional human motion generation method based on a diffusion model and a large language model, which realizes the generation of complex and diverse precise motions through partition processing and a multi-level text-motion alignment strategy. The method of the present invention includes the following key steps:

[0007] Step 1: Construction of the text feature extraction module

[0008] In this step, a Text Feature Extraction Module (TEM) is constructed. This module uses a pre-trained text model (CLIP-ViT-B / 32) to process the input text to obtain word-level semantic representations and sentence-level features. In addition, a large language model (LLAMA2 or GPT-3.5 Turbo) is used to analyze the actions described in the text. Through structured prompts and example analysis, the large language model can deeply understand the text and accurately identify the actions of the core and non-core participating parts, ensuring that key actions receive high attention during the generation process.

[0009] Step 2: Human Motion Zoning Processing

[0010] In this step, human motion is divided into five independent regions: the left arm, the right arm, the left leg, the right leg, and the torso. Specifically, the motion features of each part, i.e., joint point information, are separated from the overall motion features, and human actions are divided into five regions to analyze the motion features of each part separately. This zoning processing helps to capture more detailed motion details, thereby improving the accuracy of motion generation.

[0011] Based on the "core body parts" extracted in Step 1, these five regions are further divided into core participating parts and non-core participating parts, making the zoning of each part more targeted. This allows for more targeted analysis and generation of the motion features of different parts in subsequent processing.

[0012] Step 3: Construct a Multi-Head Cross-Attention Fusion Refinement Layer

[0013] The refinement layer fuses word-level text features with local motion features through a multi-head cross-attention mechanism to achieve fine-grained adjustment of each body part.

[0014] The specific operations are as follows:

[0015] First, the motion features of each segment are mapped to the same latent space as the text word-level features through a linear mapping layer to ensure consistent dimensions and representation spaces between different feature types. Then, a multi-head cross-attention mechanism is applied to combine the word-level text features with the local motion features of each body part. The multi-head structure enables the model to learn from multiple attention points, thereby capturing information at different levels and achieving fine optimization of the actions of each part. Finally, an encoder is used to further process the fused features. The encoder consists of multiple self-attention modules and feed-forward neural networks to ensure that the generated motion is smooth and coherent in the time dimension while maintaining the naturalness and accuracy of the actions.

[0016] Step 4: Adaptive Optimization of Core and Non-Core Parts

[0017] An adaptive loss function is used to distinguish and optimize the core and non-core parts. In step 1, the core parts (such as the head, arms, and legs) are identified by a large language model and given higher optimization weights to ensure the accurate capture of the actions of key parts. First, by analyzing the action descriptions in the input text, the distribution of the core and non-core parts is determined, and the corresponding loss weights are dynamically adjusted. This approach focuses more on processing the core sub-actions to ensure the precise control of the core parts in the generated motion, while the non-core parts retain naturalness and flexibility.

[0018] Step 5: Integration and synthesis of full-body motion

[0019] In this step, the refined local action features are synthesized into a complete human motion through an integration layer. First, the action features of each body part after refinement are merged to generate a complete full-body motion representation. This merging process includes aligning the spatial and temporal dimension features of each body part to ensure seamless connection of the actions. Then, the merged full-body action is fused with the sentence-level text features to ensure the semantic consistency of the overall motion. The sentence-level text features provide the semantic background of the overall action to ensure that the generated full-body motion meets the context requirements. Finally, the full-body motion is further optimized through a deep encoder, which captures the detailed relationship between the dynamics of the full-body motion and the text description, thereby generating a natural motion sequence that conforms to the text description.

[0020] Step 6: Generation of human motion based on the diffusion model

[0021] The human motion is generated based on the diffusion model, and a conditional diffusion model is used to ensure that the generation process conforms to the text description. During the training process, noise is gradually added to the input motion data through a Markov noise chain to generate multi-level noise samples for training the denoising ability of the model, enabling the model to gradually restore a clear motion that conforms to the text description. In the generation stage, starting from completely Gaussian noise, the motion is generated by reverse denoising to conform to the text input. The entire denoising process contains multiple time steps, and the generated motion sequence at each step is based on the input text and optional control parameters to gradually approach the ideal motion output at each stage.

[0022] Step 7: Implement parameter editing and motion completion functions

[0023] In addition to generating corresponding three-dimensional human actions based on text descriptions, the present invention can also implement parameter editing functions and motion completion functions.

[0024] Parameter Editing Function: Fine-grained control of the generated motion is achieved by adjusting the scaling factor τ of the local motion. The scaling factor is used to adjust the amplitude of the actions of each body part, and users can flexibly adjust the details of the generated motion through the interactive interface. For example, increasing the scaling factor of the arm can generate a larger swing of the arm, while decreasing the scaling factor of the leg produces a more gentle leg motion.

[0025] Motion Completion Function: Support generating a complete human motion based on partial input. Users can provide partial motion input (such as only the actions of the upper body), and combine the existing motion input and text description to generate the actions of the unspecified body parts. First, feature extraction is performed on the existing motion input, and the remaining actions are generated in combination with the text description, so that the generated full-body motion seamlessly integrates with the input part, maintaining the naturalness and overall coordination of the actions.

[0026] Step 8: Loss Function Design and Optimization Objectives

[0027] Optimizing the quality of the generated motion through the adaptive loss functions of the refinement layer and the integration layer:

[0028] Refinement Layer Loss Function: Used to optimize the generation effect of local actions, distinguish the core and non-core parts, and ensure the fine capture of core actions. This loss function includes position loss and pose loss, which are used to ensure that the joint positions of the generated actions are consistent with the target, while maintaining the rationality of the motion pose. The optimization objective of this process is: where λ 核心 and λ 非核心 represent the parameter weights of the core part and the non-core part respectively, and are the losses calculated using the same loss function: represents the action features generated from the 1st to the Tth time step and the mean squared error loss with the true action feature X0.

[0029] Integration Layer Loss Function: Used to optimize the overall consistency of the full-body motion and its alignment with the text description. The loss function of this process is similar to that of the refinement layer.

[0030] Total Loss Function: By combining the losses of the refinement layer and the integration layer and assigning appropriate weights, ensure that the finally generated human motion sequence reaches the optimal effect. The total loss function introduces a regularization loss to prevent overfitting and improve the generalization ability of the model, so that the generated motion adapts to different text descriptions and application scenarios. The optimization objective of this process is: where, λ 细化层 and λ 集成层 are used to balance the roles of the refinement layer and the integration layer in the entire model during training.

[0031] A computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-mentioned text-driven three-dimensional human motion generation method based on the diffusion model and the large language model.

[0032] A computer-readable storage medium stores a computer program that executes the above-mentioned text-driven three-dimensional human motion generation method based on the diffusion model and the large language model.

[0033] The text-driven human fine-grained motion generation method based on the diffusion model and the large language model provided by the present invention realizes the accurate generation of complex multi-actions through the segmented processing of the human body and the multi-level text-motion alignment strategy. Compared with the prior art, it has the following advantages:

[0034] 1. Improvement in generation accuracy: By dividing the human body into five parts (left arm, right arm, left leg, right leg, and torso) and independently optimizing each part in combination with the multi-head cross-attention mechanism, the present invention significantly improves the accuracy of motion generation, effectively captures fine-grained multi-action details, and makes the generated motion more natural and in line with the text description.

[0035] 2. Enhancement of text alignment ability: The present invention uses the large language model to extract core and non-core body parts from the input text and adopts a multi-level text-motion alignment strategy to ensure the accurate mapping of key actions described in the text, achieving sensitivity and precision to subtle text differences, thereby improving the text alignment ability of the generated motion.

[0036] 3. Operational flexibility and control: By introducing a parameter editing function, the present invention realizes the parametric adjustment of the actions of each body part, enabling users to flexibly control the amplitude and intensity of the actions without retraining the model, and significantly improving the operational flexibility of the model in animation production and virtual reality applications.

[0037] 4. Generation of natural and consistent full-body motion: Through the combined processing of the refinement layer and the integration layer, the present invention effectively solves the deficiencies of the prior art in terms of full-body motion consistency, enabling the fine-grained actions of each body part to naturally fuse into an overall full-body motion, ensuring the coordination and naturalness of the actions.

[0038] 5. Reduction of additional training requirements: In the motion completion application, the method generates a complete motion sequence through the existing partial input and text description, and can achieve the natural completion of partial motions without additional training, significantly saving training time and computing resources.

[0039] The present invention can be widely applied to fields such as virtual reality, animation production, and human-computer interaction, providing a solution with higher precision, more naturalness, and easier operation for human motion generation in these application scenarios, and having broad application prospects and significant technical advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is the schematic diagram of the method principle of the embodiment of the present invention;

[0041] Figure 2 is the schematic diagram of the action refinement and parameter editing application of the embodiment of the present invention;

[0042] Figure 3 is the schematic diagram of the body partitioning of the embodiment of the present invention;

[0043] Figure 4 is the schematic diagram of the action sampling of the embodiment of the present invention;

[0044] Figure 5 is the schematic diagram of the action generation function of the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The following further clarifies the present invention in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent forms of modification by those skilled in the art fall within the scope defined by the appended claims of this application.

[0046] A text-driven three-dimensional human motion generation method based on a diffusion model and a large language model. This method generates complex multi-action motion sequences that conform to the text description through human body segmentation processing and a multi-level text-motion alignment strategy. The following are the specific embodiments and operation processes of the present invention.

[0047] As Figure 1 shown, the framework of the present invention includes core components such as a text feature extraction module, a refinement layer, an integration layer, and a diffusion model, which are used to achieve the precise generation of complex actions.

[0048] Step 1: Construction of the text feature extraction module

[0049] In this step, first, a text feature extraction module (TEM) is constructed. This module uses the pre-trained text model CLIP-ViT-B / 32 to process the input text to obtain word-level semantic representations and sentence-level features. The specific operations are as follows:

[0050] (111) The input text undergoes word segmentation processing, and each word is mapped to the word embedding space of the pre-trained model.

[0051] (112) Use the CLIP-ViT-B / 32 model to encode the word embeddings of each word to obtain word-level text features.

[0052] (113) Use the CLIP-ViT-B / 32 model to encode the complete sentence and extract sentence-level features, thereby obtaining the overall semantic information.

[0053] Next, use a large language model (such as LLAMA2 or GPT-3.5 Turbo) to understand and analyze the actions described in the text, and obtain the core and non-core body parts involved in the corresponding generated actions of the text. During the understanding and analysis process, adopt <Problem Description> Input and output introduction of the action generation work; Introduction of body partitioning, and indicate the joint chains included in each part.

[0054] <Example> Given two examples, for instance, the core parts involved in "a person is walking and waving with the right hand" are "left leg, right leg, right arm".

[0055] <Problem Details and Output Settings> Given the text description "<To be Input>", give the corresponding sequence of core participating parts, and the output format is required to be several items in "[left arm, right arm, left leg, right leg, torso]".

[0056] Thus, the large language model can analyze and obtain which are the core participating parts described in the text, and the remaining parts are non-core participating parts.

[0057] Through the formula During the training process, assign higher weights to the core body parts, thereby enhancing the attention to the action features of the core parts during the generation process. The specific operations are as follows:

[0058] (121) Input the text conditions into the large language model.

[0059] (122) The large language model conducts in-depth understanding and analysis of the text based on the preset structured prompts and examples.

[0060] (123) Output the list of core participating parts corresponding to the text conditions, providing input for subsequent processing.

[0061] Step 2: Human motion partitioning processing

[0062] In this step, divide human motion into five independent regions: left arm, right arm, left leg, right leg, and torso. The specific operations are as follows:

[0063] (211) As Figure 3 shown, define the joints and motion ranges of each region.

[0064] (212)Separate the joint information of each part from the overall motion characteristics.

[0065] (213)Divide the human motion into five regions for separately analyzing the motion characteristics of each part.

[0066] Based on the "core body parts" extracted in step 1, further divide these five regions into core participating parts and non-core participating parts to make the partitioning of each part more targeted. The specific operations are as follows:

[0067] (221)Determine whether each region is a core part according to the classification results of the large language model.

[0068] (222)Perform the same feature extraction and processing operations on the five independent body parts, and in subsequent step 3, adjust the weights through parameters λ 核心 and λ 非核心 to make the weights of the core participating parts greater than those of the non-core participating parts, thereby enhancing the model's attention to the core parts.

[0069] Step 3: Multi-Head Cross-Attention Fusion Refinement Layer

[0070] In this step, fuse the word-level text features extracted in step 1 with the motion characteristics of the five independent body parts in step 2 through the multi-head cross-attention mechanism to achieve fine-grained adjustment of each body part. The specific operations are as follows:

[0071] (311)Map the motion characteristics of each body part to the same latent space as the word-level text features through a linear mapping layer to obtain the motion characteristics in the latent space.

[0072] (312)Apply the multi-head cross-attention mechanism to combine the word-level text features with the motion characteristics of each body part in the latent space obtained in the above step (311) to obtain the fusion features of each body part.

[0073] (313)Use the encoder based on the Transformer architecture to further process the fusion features of each body part to obtain temporally continuous and fine-grained adjusted motion characteristics. This encoder can effectively learn time features, thereby ensuring that the generated motion is smooth and coherent in the time dimension.

[0074] In the refinement layer (refinement process), the present invention adopts a loss function During the training process, distinguish and optimize the core and non-core parts by adjusting the weights of parameters λ 核心 and λ 非核心 . The specific operations are as follows:

[0075] (321)Determine the distribution of the core and non-core parts by analyzing the action descriptions in the input text.

[0076] (322) Dynamically adjust the loss weights λ of the core part and the non-core part 核心 and λ 非核心 , so that the model gives priority to the atomic actions corresponding to the core part, ensuring precise control of the core part during the generated movement.

[0077] Step 4: Integration and synthesis of full-body movements

[0078] In this step, the refined local action features are synthesized into a complete human movement using an integration layer. The specific operations are as follows:

[0079] (411) Pass the fusion features of each body part obtained from the refinement layer in Step 3 through their respective independent mapping layers to restore the movement features in the latent space into the joint position information corresponding to different body parts, that is, the movement representations of each body part. Subsequently, according to the composition method corresponding to the body partition, the movement representations of the five independent parts are merged to generate a complete full-body movement representation.

[0080] (412) The complete full-body movement representation is converted into an embedding representation with the same dimension as the sentence-level features through a mapping layer, and fused with the sentence-level features to obtain the spliced full-body movement features, ensuring the semantic consistency of the overall movement.

[0081] (413) By using an encoder with a deeper layer than the Transformer architecture in the refinement layer, the spliced full-body movement features are further optimized and learned to obtain a diffusion model. The integration layer uses the full-body movement features as the training unit, so as to be able to learn the fluency and naturalness of the overall movement. Different from independently learning the movement features of the five parts in the aforementioned refinement layer, the refinement layer cannot achieve overall optimization due to the lack of global information. The design of the integration layer makes up for this defect. The reason for using a deeper encoder is that the integration layer processes the full-body overall movement features, and the amount of feature information to be learned is larger, while the refinement layer only processes the movement features of independent body parts.

[0082] Step 5: Movement generation process based on the diffusion model

[0083] As Figure 4 shown, this method generates human movements based on the diffusion model, and uses a conditional diffusion model to ensure that the generation process conforms to the text description.

[0084] The specific operations are as follows:

[0085] (511) In the training stage, noise is gradually added to the input movement data through a Markov noise chain to generate multi-level noise samples, and the model learns the features of the original input data through the denoising process.

[0086] (512) In the generation stage, starting from completely Gaussian noise, human motions that conform to the text input are generated through an inverse denoising process.

[0087] Step 6: Parameter Editing and Motion Completion Function

[0088] The present invention also provides a parameter editing function and a motion completion function. The specific operations are as follows:

[0089] (611) By adjusting the scaling factor τ of local actions, fine-grained control of the generated motion is achieved.

[0090] (612) Support generating a complete human motion based on partial input, and combining existing motion input and text descriptions to generate actions of unspecified body parts. Specifically, during the generation process, the existing part action input will replace the corresponding part's motion output in the refinement layer, directly participate in the splicing process of independent part motion features, and be transmitted as input to the integration layer. At the same time, text descriptions are combined to guide the generation of new actions.

[0091] Step 7: Loss Function and Optimization Objective

[0092] The quality of the generated motion is optimized by the adaptive loss functions of the refinement layer and the integration layer. The specific operations are as follows:

[0093] (711) Design the loss function of the refinement layer Among them and have the same calculation principle: First, calculate the loss function of each body part according to where x0 represents the original motion feature of the body part in the training set, represents the motion feature of the generated body part. Then, sum the loss functions of all core parts to obtain Sum the loss functions of all non-core parts to obtain This design is used to optimize the generation effect of local actions, and adjust the attention direction of the model targeted by distinguishing between core and non-core parts.

[0094] (712) Design the loss function of the integration layer: Where X0 represents the original motion feature of the complete body in the training data set, represents the motion feature of the complete body generated by the integration layer, which is used to optimize the overall consistency of the whole-body motion and the alignment with the text description.

[0095] (713) By combining the losses of the refinement layer and the integration layer and assigning the best weights obtained through grid analysis, ensure that the finally generated human motion sequence reaches the optimal effect.

[0096] Application Example

[0097] As Figure 5 shown, the present invention has wide applications in virtual reality scenarios. For example, when a user inputs the text "A person walks forward while waving their right hand", the present invention generates a human motion that conforms to the description, presenting precise walking and right hand waving motions while maintaining overall natural consistency.

[0098] In summary, the present invention provides a text-driven human fine-grained motion generation method based on diffusion models and large language models. Through human body segmentation processing and multi-level text-motion alignment strategies, the fine generation of complex human actions and accurate text-driven alignment are achieved. This method is applicable to multiple fields such as animation production, virtual reality, and human-computer interaction, significantly improving the accuracy and controllability of motion generation, and having broad application prospects and practical value.

[0099] Obviously, those skilled in the art should understand that each step of the above-described text-driven three-dimensional human motion generation method based on diffusion models and large language models of the embodiments of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed over a network composed of multiple computing devices. Optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. Thus, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A text-driven three-dimensional human motion generation method based on diffusion models and large language models, characterized in that It includes the following steps: Step 1: Construction of the text feature extraction module Construct a text feature extraction module that processes the input text using a pre-trained text model to obtain word-level text features and sentence-level features; Analyze the actions described in the text through a large language model to distinguish actions involving core and non-core body parts. The specific process is to enable the large language model to understand the text and extract the actions of the involved core and non-core participating parts through structured prompts and example analysis; Step 2: Human body movement partition processing Divide human body movements into multiple independent regions; separate the movement features of each part from the overall action features; divide the five independent regions into core participating parts and non-core participating parts; Step 3: Construction of the multi-head cross-attention fusion refinement layer The refinement layer fuses the word-level text features with the local action features through the multi-head cross-attention mechanism to achieve fine-grained adjustment of each body part; Step 4: Adaptive optimization of core and non-core parts Adopt an adaptive loss function to distinguish and optimize core and non-core parts; by analyzing the action descriptions in the input text, determine the distribution of core and non-core parts, and dynamically adjust the corresponding loss weights; Step 5: Integration and synthesis of whole-body movements Synthesize the refined local action features into a complete human body movement through the integration layer; first, merge the refined action features of each body part to generate a complete whole-body movement representation; the merging process includes aligning the spatial and temporal dimension features of each body part; then, fuse the merged whole-body action with the sentence-level text features; Finally, optimize the whole-body movement through the encoder to generate a natural movement sequence that conforms to the text description; Step 6: Human body movement generation based on the diffusion model Generate human body movements based on the conditional diffusion model. During the training process, gradually add noise to the input movement data through the Markov noise chain to generate multi-level noise samples for training the denoising ability of the model, so that the model can gradually restore a clear movement that conforms to the text description; in the generation stage, start from completely Gaussian noise and perform reverse denoising to generate a human body movement that conforms to the text input; the entire denoising process contains multiple time steps, and the generated movement sequence at each step is based on the input text and optional control parameters to obtain the output at each stage; Step 7: Loss function design and optimization objective; optimize the quality of the generated movement through the adaptive loss functions of the refinement layer and the integration layer: Refined layer loss function: used to optimize the generation effect of local actions, distinguish the core and non-core parts, and the optimization objective is: where λ 核心 and λ 非核心 represent the parameter weights of the core part and the non-core part respectively, and are the losses calculated using the same loss function: represents the action features generated from the 1st to the Tth time step the mean square error loss with the true action feature X0; Integration layer loss function: used to optimize the overall consistency of the whole-body movement and its alignment with the text description; Total loss function: By combining the losses of the refinement layer and the integration layer and assigning corresponding weights, the optimization objective is: where λ 细化层 and λ 集成层 are used to balance the roles of the refinement layer and the integration layer in the entire model during training.

2. The text-driven three-dimensional human motion generation method based on a diffusion model and a large language model according to claim 1, wherein The method also includes functions for implementing parameter editing and movement completion; Parameter editing function: Achieve fine-grained control of the generated movement by adjusting the scaling factor τ of the local action; the scaling factor is used to adjust the action amplitude of each body part, and the user adjusts the generated movement information by setting its parameter value; Movement completion function: Support generating a complete human body movement based on partial input; the user provides partial movement input, and combines the existing movement input and text description to generate the actions of the unspecified body parts; first, extract the features of the existing movement input, and combine with the text description to generate the remaining actions, so that the generated whole-body movement is fused with the input part.

3. The text-driven three-dimensional human motion generation method based on a diffusion model and a large language model according to claim 1, wherein, In step 3, the refinement layer fuses the text features with the local action features through a multi-head cross-attention mechanism to achieve fine-grained adjustment of each body part. The specific operations are as follows: First, the motion features of each body part are mapped to the same latent space as the word-level text features through a linear mapping layer to obtain the motion features in the latent space. Then, a multi-head cross-attention mechanism is applied to combine the word-level text features with the motion features of each body part in the latent space to obtain the fused features of each body part.

4. The text-driven three-dimensional human motion generation method based on a diffusion model and a large language model according to claim 1, characterized in that In step 1, first construct a text feature extraction module. The text feature extraction module uses the pre-trained text model CLIP-ViT-B / 32 to process the input text to obtain word-level semantic representations and sentence-level features. The specific operations are as follows: (111) The input text is tokenized, and each word is mapped to the word embedding space of the pre-trained model. (112) Using the CLIP-ViT-B / 32 model, the word embedding of each word is encoded to obtain word-level text features. (113) Using the CLIP-ViT-B / 32 model to encode the complete sentence to extract sentence-level features. Use a large language model to understand and analyze the actions described in the text, and obtain the core and non-core body parts involved in the actions corresponding to the text generation. During the understanding and analysis process, adopt the input and output introduction of the action generation work in <Problem Description>; introduce the body partition, and indicate the joint chains included in each part. The large language model analyzes and obtains the core participating parts described in the text, and the remaining parts are non-core participating parts. Through the formula Higher weights are assigned to the core body parts during training as follows: (121) Input the text conditions into the large language model. (122) The large language model understands and analyzes the text according to the preset structured prompts and example analyses. (123) Output the list of core participating parts corresponding to the text conditions.

5. The text-driven three-dimensional human motion generation method based on the diffusion model and the large language model according to claim 1, wherein, In step 2, the human body motion is divided into five independent regions: the left arm, the right arm, the left leg, the right leg, and the torso. Step 2 includes the following steps: (211) Define the joints and motion ranges of each region. (212) Separate the joint information of each part from the overall action features. (213) Divide the human body actions into five regions. Based on the core body parts extracted in step 1, divide the five regions into core participating parts and non-core participating parts. The specific operations are as follows: (221) Determine whether each region is a core part according to the classification results of the large language model. (222) Perform the same feature extraction and processing operations on five independent body parts, and adjust the weights through parameters λ 核心 and λ 非核心 in the subsequent step 3, so that the weights of the core participating parts are greater than those of the non-core participating parts, thereby enhancing the model's attention to the core parts.

6. The text-driven three-dimensional human motion generation method based on a diffusion model and a large language model according to claim 1, characterized in that In step 3, the word-level text features extracted in step 1 are fused with the motion features of the five independent body parts in step 2 through a multi-head cross-attention mechanism to achieve fine-grained adjustment of each body part. The specific operations are as follows: (311) Through a linear mapping layer, the motion features of each body part are mapped to the same latent space as the word-level text features to obtain the motion features in the latent space. (312) Apply a multi-head cross-attention mechanism to combine the word-level text features with the motion features of each body part in the latent space obtained in the above step (311) to obtain the fused features of each body part. (313) Process the fused features of each body part using an encoder based on the Transformer architecture to obtain temporally continuous and fine-grained adjusted motion features; In the refinement layer, a loss function is adopted During the training process, by adjusting the parameter λ 核心 and λ 非核心 The weights are used to distinguish and optimize the core and non-core parts. The specific operations are as follows: (321) Determine the distribution of core and non-core parts by analyzing the action descriptions in the input text; (322) Dynamically adjust the loss weights λ of the core part and the non-core part 核心 and λ 非核心 , enabling the model to prioritize the atomic actions corresponding to the core part.

7. The text-driven three-dimensional human motion generation method based on a diffusion model and a large language model according to claim 1, wherein In step 4, use the integration layer to synthesize the refined local action features into a complete human motion, and the specific operations are as follows: (411) Pass the fused features of each body part obtained from the refinement layer in step 3 through their respective independent mapping layers to restore the motion features in the latent space into the joint position information corresponding to different body parts, that is, the motion representations of each body part; Subsequently, according to the composition method corresponding to the body partition, merge the motion representations of the five independent parts to generate a complete full-body motion representation; (412) The complete full-body motion representation is converted into an embedding representation with the same dimension as the sentence-level features through a mapping layer, and fused with the sentence-level features to obtain the spliced full-body motion features; (413) Further optimize and learn the spliced full-body motion features by using an encoder with a deeper Transformer architecture than that in the refinement layer to obtain a diffusion model.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the above computer program, it implements the steps of the text-driven three-dimensional human action generation method based on the diffusion model and the large language model as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the text-driven three-dimensional human action generation method based on the diffusion model and the large language model as described in any one of claims 1-7.

Images