Text-to-action generation methods and systems based on fine-grained representations of body parts

Abstract

This invention discloses a text-to-action generation method and system based on fine-grained representations of body parts, belonging to the field of human action generation. By performing fine-grained discretization modeling of human actions at the body part level, an action encoder and action decoder capable of stably reconstructing continuous action sequences are obtained. Using a fine-grained discrete action space as the generative space, and introducing video latent representations as spatiotemporal dynamic priors, an alignment modeling mechanism between the video latent representations and the discrete action space is established. A conditional generation network is trained to achieve stable prediction and iterative completion of discrete action index sequences during the inference stage. The action decoder then reconstructs these sequences into continuous human action sequences, ultimately generating human action sequences that are semantically consistent with the text, richer in detail, and more temporally coherent. This invention can reduce the technical threshold and data dependence of action content production, and improve the controllability and generalization ability of action generation.

Description

Technical Field

[0001] This invention belongs to the field of human motion generation and cross-modal conditional generation technology, specifically involving a text-to-action generation method and system based on fine-grained representation of body parts. Background Technology

[0002] In recent years, with the development of generative artificial intelligence and 3D human modeling technology, text-driven human motion generation has received widespread attention in scenarios such as digital human animation, virtual interaction, and embodied intelligence. Traditional motion production typically relies on manual keyframe editing, motion capture, and post-processing cleaning and retargeting, resulting in a complex, costly, and time-consuming production chain that struggles to meet the demands for rapid production of large-scale, diverse motion content. Therefore, researchers are increasingly exploring technical approaches that directly generate 3D human motion sequences using natural language descriptions—i.e., text-to-human motion generation—to reduce production costs and improve content production efficiency.

[0003] Existing text-to-human motion generation methods primarily rely on diffusion models or Transformer sequence modeling. These methods encode textual cues as semantic conditions and regress joint trajectories, pose parameters, or high-dimensional motion features in a continuous motion space to generate motion sequences that match the text description. While these methods have made some progress in basic tasks, they still have significant shortcomings in engineering applications: First, human motion exhibits significant differences across body parts. Local parts such as the hands contain more frequent and complex movement patterns. Using a single whole-body representation or uniform granular discretization can easily lead to problems such as missing local details, hand stiffness, or unstable motion rhythms, affecting realism and controllability. Second, relying solely on textual conditions lacks stable spatiotemporal dynamic prior constraints. In long sequences or complex semantic combinations, they are more prone to temporal drift, incoherent motion, and semantic shifts, reducing the usability of the generated results.

[0004] To improve modeling efficiency and stability, some works introduce discrete action representation and conditional generation mechanisms, quantizing continuous actions into discrete index sequences, and then using a sequence generation model to predict the indices and decode to reconstruct the continuous actions. This approach can reduce the generation difficulty, but if the discrete representation does not reflect the structural differences of body parts, it may still lead to over-compression of fine-grained actions such as those involving the hands; at the same time, without cross-body collaborative modeling, the coordination of actions is still difficult to guarantee.

[0005] In addition, some methods introduce video or image features as auxiliary conditions, but most of them only use visual features as additional inputs and have not made full use of the spatiotemporal dynamic distribution priors contained in the latent representations of pre-trained textual video diffusion models. They also lack a systematic alignment modeling strategy for the mapping relationship between video latent representations and discrete action space, which makes it difficult to effectively transform video priors into conditional information that can be directly used by the action generation end. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a text-to-action generation method and system based on fine-grained representation of body parts, which takes into account both the fine-grained action representation capability of body parts and strong spatiotemporal dynamic prior constraints. Specifically, it improves the local detail and cross-part collaborative modeling capabilities by constructing fine-grained discrete action representations for different body parts, and uses the video latent representation generated by a pre-trained text-generated video model as a dynamic prior. Based on the alignment of the video latent and the discrete action space, a conditional generation network is trained to obtain semantically consistent, temporally stable and more realistic human action generation results.

[0007] To achieve the above-mentioned objectives, an embodiment provides a text-to-action generation method based on fine-grained representation of body parts, comprising the following steps:

[0008] Using action description prompts as semantic constraints, and based on these semantic constraints, a pre-trained text-based video diffusion model is used to generate potential video representations corresponding to the action description prompts for action videos.

[0009] A fine-grained representation based on body parts is constructed and trained on the 3D human motion data, so that the motion encoder maps continuous human motion to fine-grained representation. The fine-grained representation is then quantized by multi-scale residual to generate a discrete index sequence of motion. The codebook is searched based on the discrete index sequence of motion to obtain the quantized feature sequence of motion. The motion decoder decodes and reconstructs the quantized feature sequence of motion into a continuous human motion sequence.

[0010] The action encoder uses a mapping network to construct alignment constraints based on video latent representations and action discrete index sequences. The video latent representations after alignment constraints are used as conditional feature sequences. The conditional generator network is trained on the action discrete index sequences for mask modeling tasks, so that the conditional generator network learns to predict the mapping relationship of action discrete index sequences under the video latent representations.

[0011] During inference, a generative model consisting of a pre-trained textual video diffusion model, a mapping network, a conditional generation network, a codebook lookup, and an action decoder is used to generate a continuous sequence of human actions that is semantically consistent with the input action description prompt text.

[0012] Preferably, in the fine-grained representation based on body parts, the human motion representation is divided into multiple body part subspaces, and a motion encoder is set for each body part subspace to map the motion of each body part to a corresponding fine-grained representation. This allows different body parts to obtain discrete representation capabilities of different granularities, thereby improving the fidelity of motion details in local parts such as the hands and avoiding the compression of local high-frequency motion patterns by a uniform codebook for the whole body. The fine-grained representation of each body part generates a corresponding discrete motion index sequence through separate multi-scale residual quantization. The discrete motion indexes of each body part maintain an explicit correspondence in the unified sequence, enabling the conditional generation network to learn global motion trends and local motion details simultaneously within the same sequence modeling framework, improving cross-part coordination.

[0013] Preferably, the multi-scale residual quantization includes: performing multi-scale temporal encoding on the input fine-grained representation to obtain multi-resolution latent features, and performing residual vector quantization on the latent features at each scale to generate multi-scale discrete motion indexes, forming a discrete motion index sequence to improve the stability of long-time-series motions and the ability to represent the details of complex motions. Multi-scale residual quantization models motion sequences through a coarse-to-fine stepwise compensation method, enabling low-frequency overall motion patterns and high-frequency local details to be encoded separately at different scales and fused and reconstructed at the decoding end, thereby reducing the loss of details and jitter problems in the generation of long sequences.

[0014] Preferably, before codebook lookup, the discrete action indexes for different body parts are integrated into a unified discrete action index sequence. Specifically, based on time step alignment, the discrete action indexes corresponding to multiple body parts are interwoven in a preset order or concatenated according to body part order to enhance cross-body action collaborative modeling capabilities and improve the learning efficiency of the conditional generation network. This unified discrete action index sequence provides a definite input-output format for sequence prediction in mask modeling, enabling the network to utilize information from different body parts simultaneously in a bidirectional context, strengthening local and global consistency constraints.

[0015] Preferably, an alignment constraint is constructed using a mapping network based on the video latent representation and discrete action index sequence. This includes: mapping the video latent representation to the conditional feature space of the conditional generation network using the mapping network; the mapped conditional feature sequence and the action quantization feature sequence satisfy the alignment input requirements of the conditional generation network in terms of temporal length and feature dimension, thereby ensuring the stability of alignment modeling training. The mapping network transforms the dynamic evolution trend implicit in the video latent into a conditional feature sequence that the action generation network can directly consume, enabling the spatiotemporal priors learned by the pre-trained text-generated video model to be effectively transferred to the action generation task.

[0016] Preferably, training the conditional generation network for mask modeling based on discrete action index sequences includes:

[0017] A masking label is applied to some positions of the discrete action index sequence as the input sequence. Under the constraint of the conditional feature sequence, the conditional generation network predicts the action index of the masked position of the input sequence and repeats the masking prediction until the complete target action discrete index sequence is obtained, thereby reducing the error accumulation when generating long sequences and improving temporal consistency. A mask loss is constructed to optimize the parameters of the conditional generation network, where the mask loss adopts the cross-entropy loss.

[0018] Because mask modeling can utilize both left and right context information of the sequence during prediction, it can maintain stronger global consistency during inference and naturally form joint constraints on the coupling relationship between parts under the cross-part interwoven index structure.

[0019] Preferably, the pre-trained text-based video diffusion model uses paired text and video data for efficient parameter fine-tuning, making the generated video latent representation under semantic constraints closer to the distribution of training data, thereby improving the semantic consistency and stability of human action generation. This fine-tuning strategy enables the text-based video latent prior to better fit the action semantics and motion distribution of the target data domain, reducing the impact of cross-domain bias on action generation quality and improving the usability of generated actions during the inference stage.

[0020] To achieve the above-mentioned objectives, embodiments of the present invention also provide a text-to-action generation system based on fine-grained representation of body parts, comprising:

[0021] The video representation module is used to generate potential video representations corresponding to the action description prompts by using the action description prompts as semantic constraints and a pre-trained text-based video diffusion model based on these semantic constraints.

[0022] The fine-grained representation modeling module is used to construct and train fine-grained representations based on body parts of 3D human motion data, enabling the motion encoder to map continuous human motions into fine-grained representations. The fine-grained representations are then quantized through multi-scale residuals to generate a discrete index sequence of motions. The codebook is searched based on the discrete index sequence of motions to obtain the quantized feature sequence of motions. The motion decoder decodes and reconstructs the quantized feature sequence of motions into a continuous human motion sequence.

[0023] The Conditional Generative Network (CGN) training module is used to construct alignment constraints based on video latent representations and discrete action index sequences using a mapping network. The video latent representations after alignment constraints are used as conditional feature sequences. The CGN is trained on the action discrete index sequences to perform mask modeling tasks, enabling the CGN to learn the mapping relationship for predicting action discrete index sequences under the video latent representations.

[0024] In the inference application stage, when used for inference, it consists of a pre-trained text-based video diffusion model, a mapping network, a conditional generation network, a codebook lookup, and an action decoder to generate a continuous human action sequence that is consistent with the semantics of the input action description prompt text.

[0025] To achieve the above-mentioned objectives, the embodiments also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned text-to-action generation method based on fine-grained representation of body parts.

[0026] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-mentioned text-to-action generation method based on fine-grained representation of body parts.

[0027] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0028] 1. By combining fine-grained representation of body parts with multi-scale residual quantization, we can improve the ability to express details and the reconstruction quality of human movements, especially local parts such as hands, and alleviate the problems of detail collapse and local distortion.

[0029] 2. The latent video representation generated by the pre-trained textual video model is used as a spatiotemporal dynamic prior. The conditional generation network is trained by aligning the latent representation with the discrete space of the action, so that the generation process is subject to stronger dynamic constraints, thereby improving the temporal coherence and stability of long sequence actions.

[0030] 3. The overall process takes discrete action representation learning, latent alignment condition generation, and action decoding and reconstruction as the core link, which is easy to transfer and extend to different action data and different textual video prior models, and has high engineering usability and application value.

[0031] Furthermore, this method introduces pre-trained video priors into the action generation end in a trainable and alignable manner, enabling the model to not only align with text descriptions at the semantic level, but also obtain stronger motion pattern constraints at the dynamic level, thereby improving the stability and deployability of the generated results in actual digital human-driven and intelligent agent control tasks. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is an overall flowchart of the text-to-human motion generation method provided in the embodiments of the present invention;

[0034] Figure 2 This is a schematic diagram illustrating the structure and process of the pre-trained textual video diffusion model for generating video latent representations, provided in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of the fine-grained representation construction and training process based on body parts provided in an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of multi-scale residual vector quantization provided in an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of training a conditional generation network based on video latent-to-action discrete space alignment modeling provided in an embodiment of the present invention;

[0038] Figure 6 This is an overall structural diagram of text-to-human motion generation during the reasoning stage provided in this embodiment of the invention;

[0039] Figure 7 This is a structural diagram and flowchart of the text-to-human motion generation system provided in an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0041] The inventive concept of this invention is as follows: In text-to-human motion generation tasks, relying solely on textual conditions for motion sequence modeling is insufficient to stably cover complex spatiotemporal dynamics, especially in long-sequence motions, fine-grained hand movements, and cross-part coordination, which are prone to semantic shifts and missing details. To address these issues, this invention provides a text-to-motion generation method and system based on fine-grained representations of body parts. By performing fine-grained discretization modeling of human motion at the body part level, a motion encoder and motion decoder capable of stably reconstructing continuous motion sequences are obtained. Using the fine-grained discrete motion space of body parts as the generative space, a pre-trained text-generated video diffusion model-generated video latent representation is introduced as a spatiotemporal dynamic prior. An alignment modeling mechanism between the video latent representation and the discrete motion space is established, training a conditional generation network to achieve stable prediction and iterative completion of the discrete motion index sequence during the inference stage. This sequence is then reconstructed into a continuous human motion sequence by the motion decoder, ultimately generating a human motion sequence that is semantically consistent with the text, richer in detail, and more temporally coherent. This method can reduce the technical threshold and data dependence of motion content production, improve the controllability and generalization ability of motion generation, and is applicable to application scenarios such as digital human animation production, human-computer interaction, virtual reality and embodied intelligence.

[0042] like Figure 1 As shown, the text-to-action generation method based on fine-grained representation of body parts provided in the embodiment includes the following steps:

[0043] S1 uses the action description prompt text as a semantic constraint, and based on this semantic constraint, a pre-trained text-based video diffusion model is used to generate a video potential representation corresponding to the action description prompt text for the action video.

[0044] In the embodiments, such as Figure 2 As shown, the action description prompt text is denoted as First, Input text encoder The resulting text conditional embedding sequence serves as the semantic constraint for the diffusion sampling process. The formal expression of semantic constraints is as follows:

[0045]

[0046] in, Indicates the number of text tokens. Feature dimensions representing semantic constraints. Represents the feature space.

[0047] During the inference phase, the pre-trained Wensheng video diffusion model uses Gaussian noise as the initial state of the video latent space. Under the scheduling of time step t, the DiT denoising sampling module performs semantic denoising sampling under the given conditions. Guided by [previous method], N iterative denoising steps are performed, where noise and semantic constraints undergo cross-attention calculation, outputting the video latent tensor Z. The video latent tensor Z is represented in four-dimensional tensor form as follows:

[0048]

[0049] in, For potential time length, For the number of potential channels, The latent spatial resolution is determined. To facilitate subsequent conditional mapping and batch training, the video latent representation maintains a consistent tensor organization method in the implementation, and maintains consistent data flow and dimensional configuration between the training and inference phases, enabling the mapping network from video latent to conditional sequence to stably output conditional feature sequences.

[0050] Furthermore, the pre-trained text-to-video diffusion model can be lightly fine-tuned using text-video pairing data. This allows the model to better align the correspondence between text semantic constraints and video latent representations in the distribution of target action videos, reducing semantic inconsistencies caused by domain bias. Through this fine-tuning process, the video latent representation Z generated by the model can more accurately reflect the key motion semantics and temporal variation patterns of the action description prompts, thus providing more reliable conditional input for subsequent mapping learning from video latent to action conditions and improving the overall controllability and stability of action generation.

[0051] S2 constructs and trains fine-grained representations based on body parts for 3D human motion data, enabling the motion encoder to map continuous human motions into fine-grained representations. The fine-grained representations are then quantized through multi-scale residuals to generate a discrete motion index sequence. The motion quantization feature sequence is obtained by searching the codebook based on the discrete motion index sequence. The motion decoder then decodes and reconstructs the motion quantization feature sequence into a continuous human motion sequence.

[0052] In this embodiment, the continuous three-dimensional human motion sequence is denoted as... ,in Indicates the length of the action sequence. This represents the dimension of motion features. Motion features can consist of posture, displacement, and joint-related features output by the human parametric model, and are uniformly represented with fixed dimensions before training to adapt to the network input.

[0053] The goal of fine-grained action representation training is to learn a set of invertible mappings that enable discrete action index sequences to stably carry continuous action information and can be reconstructed by the decoder. Action encoder First, the action sequence is mapped to a latent fine-grained representation using an action encoder, followed by a vector discrete quantization module that includes multi-scale residual quantization and codebook lookup. Obtain motion quantization features Action decoder Quantify the features of the quantized action. Reconstruction into continuous motion The reconstructed learning objectives can be written as:

[0054]

[0055]

[0056] in, Indicates the losses incurred during reconstruction. It represents the square of the distance between the two.

[0057] To enhance the expressive power of discrete representations in fine-grained movements, especially in the coordination of hand and whole-body movements, this embodiment divides human movements into body parts and forms a discrete path for part-based modeling in the network structure. Specifically, as follows... Figure 3 As shown, the human motion representation is divided into multiple body part subspaces, including the main body branch and the hand branch, corresponding to hand motion sequences and body motion sequences. For each body part subspace, a motion encoder (hand encoder and body part encoder) is set up to map the motion of each part to a corresponding fine-grained representation. The fine-grained representation of the hand generates corresponding motion quantization features through a separate vector discretization module. The fine-grained representation of the body and the motion quantization features of the hand are concatenated and input into the corresponding vector discretization module of the body to obtain the body's motion quantization features. Then, the motion quantization features of the body and head are integrated before decoding and reconstruction, enabling different parts to obtain a more suitable discrete representation under their motion statistical distribution. In this way, large-amplitude body movements and high-frequency hand movements can be simultaneously expressed in discrete space, reducing the dilution of local details by a single whole-body representation.

[0058] To enhance the expressive power of long sequence actions and multi-scale motion patterns, this embodiment employs a multi-scale residual quantization mechanism and codebook lookup in the vector discrete quantization module. For example... Figure 4 As shown, let the encoding end time length be... Preset set of time scales from coarse to fine And satisfy the finest scale For the first Layer quantization, first for residual features Interpolate to length along the time dimension Then perform quantization and look up the table to obtain the quantization features. :

[0059]

[0060] in, For time interpolation operators, For vector quantization operators, For codebook, For discrete index sequences of actions, These are the interpolated features. The features will then be quantized. Interpolate back to length and update the residuals. Used for the next layer of quantization:

[0061]

[0062] After quantization at all scales is completed, the quantized features at each scale are interpolated to a uniform length and then summed point by point to obtain the final quantized features. :

[0063]

[0064] Through the above training, the action encoder can map the input action sequence into a fine-grained representation. The fine-grained representation is then processed by the vector discrete quantization module to generate the corresponding action quantization feature sequence. The action decoder can dequantize the predicted action quantization feature sequence into latent features and reconstruct it into a continuous action sequence, providing a definite supervision space and a definite output decoding path for the subsequent conditional generation network.

[0065] S3 employs a mapping network to construct alignment constraints based on video latent representations and discrete action index sequences. The aligned video latent representations are used as conditional feature sequences. The conditional generation network is trained on the action discrete index sequences for mask modeling tasks, enabling the conditional generation network to learn the mapping relationship for predicting action discrete index sequences under the video latent representations.

[0066] In this embodiment, alignment modeling training of video latent representation and action discrete space is performed to obtain a conditional generation network; wherein, the alignment modeling stage uses paired video and action pairing data as supervision to learn the mapping relationship for predicting action discrete index sequences under video latent conditions.

[0067] Specifically, the training data for alignment modeling consists of video sequences. With human motion sequence The two components are aligned in time and maintain semantic consistency. (Action sequence) The action discrete index set is obtained by action encoder and multi-scale residual quantization, and then further constructed into a unified discrete modeling sequence to adapt to the sequence modeling form of the conditional generation network.

[0068] First, construct a unified action discrete index sequence. For the... The discrete indices of the body and hand movements at each time scale are denoted as follows: and The two discrete action indices are interleaved according to their time positions to obtain a unified discrete action index sequence. :

[0069]

[0070] in For a given interleaving operator, make Odd-numbered positions correspond to body indices, and even-numbered positions correspond to hand indices. Further concatenation of the discrete motion index sequences at each scale in scale order yields the final discrete modeling sequence. :

[0071]

[0072] Secondly, action video sequences Input video encoder to obtain video latent features A mapping network is used to map video latent representations into sequences of conditional features. Due to the video latent representation... Since the conditional generator network is a four-dimensional tensor, it receives conditional inputs in sequence form, therefore a mapping network is used. Transform video latent representations into conditional feature sequences :

[0073]

[0074] in The length of the conditional sequence. The conditional feature dimension is defined by the mapping network, which structurally aggregates and projects the latent representations of the video in both space and time. This ensures that the output conditional feature sequence remains consistent with the input interface of the action generation network, and that the same mapping method is maintained during both the training and inference phases, thus forming a stable conditional input channel.

[0075] like Figure 5 As shown, a bidirectional Transformer modeled by a mask is used as the conditional generator network during training. For discrete index sequences of actions Construct a mask modeling task. During training, sample a set of masks from sequence positions. That is, a mask is applied to a portion of the discrete index sequence of actions as a partially visible input sequence, and then a mask is applied to the conditional feature sequence. Predict the index distribution of the masked position under constraints.

[0076] The training objective can be expressed using the cross-entropy loss at the mask position. Suppose the conditional generative network... Output category distribution at each location The actual index is Mask loss at a single location for:

[0077]

[0078] Further introduction of effective position masks This is used to mask the padding locations and ensure stable training of samples of different lengths within the same batch. The overall loss is:

[0079]

[0080] Through the above training, the conditional generative network learns the conditional mapping relationship from the video latent representation to the discrete index sequence of actions, so that the video latent prior can enter the discrete action generation end in the form of conditional feature sequence, and perform deterministic end-to-end optimization in the training objective with discrete index supervision signal.

[0081] S4, during inference, uses a pre-trained text-based video diffusion model, mapping network, conditional generation network, codebook lookup, and action decoder to form a generative model that generates a continuous sequence of human actions consistent with the semantics of the input action description prompt text.

[0082] In this embodiment, during inference, the latent representation of the video is passed through a mapping network and input conditions are used to generate a network-predicted target action index sequence. The prediction process uses a mask modeling method to iteratively complete the action index sequence. The generated target action index sequence is input into the action decoder for decoding and reconstruction, resulting in a continuous human action sequence output that is consistent with the semantics of the action description text prompt.

[0083] Specifically, such as Figure 6 As shown, the input is a text prompt describing the action. First, step S1 is executed to obtain the latent representation of the video using a pre-trained textual video diffusion model. Conditional feature sequences are obtained through a mapping network. The action index generation process then begins: the action index sequence is initialized to a masked state, and the conditional generation network... Under the constraint of the conditional feature sequence, the probability distribution of the index at each position is output; the mask positions are gradually replaced with predicted indices according to the iterative completion strategy, thereby obtaining the complete target action index sequence. This iterative completion method utilizes the left and right context information of the sequence in each update, enabling the generated results to maintain strong global consistency. At the same time, since the unified action index sequence is composed of multi-scale cascades of interwoven body and hand movements, body and hand movements are jointly modeled within the same sequence framework. The temporal coupling relationship across parts can be synchronously constrained during the generation process, thereby improving the coordination of movements and the quality of local details.

[0084] After obtaining the target action index sequence Then, the target action index sequence is obtained based on the codebook lookup operation. The corresponding motion quantization feature sequence is then input into the motion decoder trained in step S2 for decoding and reconstruction, resulting in a continuous human motion sequence consistent with the motion description prompt text. :

[0085] To achieve the above-mentioned objectives, such as Figure 7 As shown, this embodiment of the invention also provides a text-to-action generation system 60 based on fine-grained representations of body parts, including: a video representation module 61, a fine-grained representation modeling module 62, a conditional generation network training module 63, and an inference application stage 64. The video representation module 61 uses action description prompts as semantic constraints and, based on these constraints, employs a pre-trained text-to-video diffusion model to generate potential video representations corresponding to the action description prompts. The fine-grained representation modeling module 62 constructs and trains fine-grained representations based on body parts from 3D human motion data, enabling the action encoder to map continuous human motions to fine-grained representations. These fine-grained representations undergo multi-scale residual quantization to generate a discrete action index sequence. Based on this discrete action index sequence, a codebook is searched to obtain a quantized action feature sequence. The action decoder decodes and reconstructs the quantized action feature sequence into a continuous human motion sequence. The conditional generation network training module 63 is used to construct alignment constraints based on video latent representations and discrete action index sequences using a mapping network. The video latent representations after alignment constraints are used as conditional feature sequences. The conditional generation network is trained on the action discrete index sequences to perform mask modeling tasks, so that the conditional generation network learns the mapping relationship of predicting action discrete index sequences under video latent representations. The inference application stage 64 is used to generate a generation model composed of a pre-trained text-generated video diffusion model, a mapping network, a conditional generation network, a codebook lookup, and an action decoder during inference. Based on the input action description prompt text, it generates a continuous human action sequence that is consistent with the text semantics.

[0086] It should be noted that the text-to-action generation system based on fine-grained body part representation provided in the above embodiments should be illustrated using the above-described functional module divisions when generating text-to-action data. The functions can be assigned to different functional modules as needed, i.e., the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the text-to-action generation system based on fine-grained body part representation provided in the above embodiments and the text-to-action generation construction method embodiment based on fine-grained body part representation belong to the same concept. For details of their implementation, please refer to the text-to-action generation method embodiment based on fine-grained body part representation, which will not be repeated here.

[0087] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-described text-to-action generation method based on fine-grained representations of body parts, specifically including the following steps:

[0088] S1, using the action description prompt text as a semantic constraint, and based on this semantic constraint, using a pre-trained text-based video diffusion model to generate a video potential representation corresponding to the action description prompt text for the action video;

[0089] S2, construct and train a fine-grained representation based on body parts for the three-dimensional human motion data, so that the motion encoder maps continuous human motion to fine-grained representation. The fine-grained representation is quantized by multi-scale residual to generate a discrete index sequence of motion. The codebook is searched based on the discrete index sequence of motion to obtain the motion quantization feature sequence. The motion decoder decodes and reconstructs the motion quantization feature sequence into a continuous human motion sequence.

[0090] S3. A mapping network is used to construct alignment constraints based on video latent representations and discrete action index sequences. The video latent representations after alignment constraints are used as conditional feature sequences. The conditional generation network is trained on the action discrete index sequence for mask modeling tasks, so that the conditional generation network learns to predict the mapping relationship of the action discrete index sequence under the conditional feature sequence.

[0091] S4, during inference, uses a pre-trained text-based video diffusion model, mapping network, conditional generation network, codebook lookup, and action decoder to form a generative model that generates a continuous sequence of human actions consistent with the semantics of the input action description prompt text.

[0092] The computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the text-to-action generation method based on fine-grained representation of body parts described in S1-S4 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0093] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described text-to-action generation method based on fine-grained representations of body parts, specifically including the following steps:

[0094] S1, using the action description prompt text as a semantic constraint, and based on this semantic constraint, using a pre-trained text-based video diffusion model to generate a video potential representation corresponding to the action description prompt text for the action video;

[0095] S2, construct and train a fine-grained representation based on body parts for the three-dimensional human motion data, so that the motion encoder maps continuous human motion to fine-grained representation. The fine-grained representation is quantized by multi-scale residual to generate a discrete index sequence of motion. The codebook is searched based on the discrete index sequence of motion to obtain the motion quantization feature sequence. The motion decoder decodes and reconstructs the motion quantization feature sequence into a continuous human motion sequence.

[0096] S3. A mapping network is used to construct alignment constraints based on video latent representations and discrete action index sequences. The video latent representations after alignment constraints are used as conditional feature sequences. The conditional generation network is trained on the action discrete index sequence for mask modeling tasks, so that the conditional generation network learns to predict the mapping relationship of the action discrete index sequence under the conditional feature sequence.

[0097] S4, during inference, uses a pre-trained text-based video diffusion model, mapping network, conditional generation network, codebook lookup, and action decoder to form a generative model that generates a continuous sequence of human actions consistent with the semantics of the input action description prompt text.

[0098] In this embodiment, computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0099] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A text-to-action generation method based on fine-grained representation of body parts, characterized in that, Includes the following steps: Using action description prompts as semantic constraints, and based on these semantic constraints, a pre-trained text-based video diffusion model is used to generate potential video representations corresponding to the action description prompts for action videos. A fine-grained representation based on body parts is constructed and trained on the 3D human motion data, so that the motion encoder maps continuous human motion to fine-grained representation. The fine-grained representation is then quantized by multi-scale residual to generate a discrete index sequence of motion. The codebook is searched based on the discrete index sequence of motion to obtain the quantized feature sequence of motion. The motion decoder decodes and reconstructs the quantized feature sequence of motion into a continuous human motion sequence. A mapping network is used to construct alignment constraints based on video latent representations and discrete action index sequences. The video latent representations after alignment constraints are used as conditional feature sequences. The conditional generation network is trained on the action discrete index sequences for mask modeling tasks, so that the conditional generation network learns to predict the mapping relationship of action discrete index sequences under the conditional feature sequences. During inference, a generative model consisting of a pre-trained textual video diffusion model, a mapping network, a conditional generation network, a codebook lookup, and an action decoder is used to generate a continuous sequence of human actions that is semantically consistent with the input action description prompt text.

2. The text-to-action generation method based on fine-grained representation of body parts according to claim 1, characterized in that, In the fine-grained representation based on body parts, human motion representation is divided into multiple body part subspaces, and motion encoders are set for each body part subspace to map the motion of each body part to the corresponding fine-grained representation. Then, multi-scale residual quantization is used to generate a discrete index sequence of motion for each body part.

3. The text-to-action generation method based on fine-grained representation of body parts according to claim 1 or 2, characterized in that, The multi-scale residual quantization includes: performing multi-scale temporal encoding on the fine-grained representation of the input to obtain multi-resolution latent features, and performing residual vector quantization on the latent features at each scale to generate multi-scale dynamic discrete indexes, forming a dynamic discrete index sequence.

4. The text-to-action generation method based on fine-grained representation of body parts according to claim 2, characterized in that, Before searching the codebook, the discrete indexes of actions for different body parts are integrated to form a unified sequence of discrete indexes of actions. Specifically, based on time step alignment, the discrete indexes of actions corresponding to multiple body parts are interwoven in a preset order or spliced ​​together in the order of body parts.

5. The text-to-action generation method based on fine-grained representation of body parts according to claim 1, characterized in that, Alignment constraints are constructed using a mapping network based on video latent representations and discrete action index sequences, including: A mapping network is used to map the latent representation of the video onto the conditional feature space of the conditional generation network. The mapped conditional feature sequence and the action quantization feature sequence meet the alignment input requirements of the conditional generation network in terms of temporal length and feature dimension.

6. The text-to-action generation method based on fine-grained representation of body parts according to claim 1, characterized in that, Training a conditional generation network for mask modeling based on discrete action index sequences includes: A mask is applied to a portion of the discrete index sequence of actions as the input sequence. Under the constraint of the conditional feature sequence, the conditional generation network predicts the action index of the masked position in the input sequence and repeats the mask prediction until the complete discrete index sequence of the target action is obtained. A mask loss is constructed to optimize the parameters of the conditional generation network, where the mask loss adopts cross-entropy loss.

7. The text-to-action generation method based on fine-grained representation of body parts according to claim 1 or 6, characterized in that, The condition generation network uses a bidirectional Transformer.

8. A text-to-action generation system based on fine-grained representation of body parts, characterized in that, include: The video representation module is used to generate potential video representations corresponding to the action description prompts by using the action description prompts as semantic constraints and a pre-trained text-based video diffusion model based on these semantic constraints. The fine-grained representation modeling module is used to construct and train fine-grained representations based on body parts of 3D human motion data, enabling the motion encoder to map continuous human motions into fine-grained representations. The fine-grained representations are then quantized through multi-scale residuals to generate a discrete index sequence of motions. The codebook is searched based on the discrete index sequence of motions to obtain the quantized feature sequence of motions. The motion decoder decodes and reconstructs the quantized feature sequence of motions into a continuous human motion sequence. The Conditional Generative Network (CGN) training module is used to construct alignment constraints based on video latent representations and discrete action index sequences using a mapping network. The video latent representations after alignment constraints are used as conditional feature sequences. The CGN is trained on the action discrete index sequences to perform mask modeling tasks, enabling the CGN to learn the mapping relationship for predicting action discrete index sequences under the video latent representations. In the inference application stage, when used for inference, it consists of a pre-trained text-based video diffusion model, a mapping network, a conditional generation network, a codebook lookup, and an action decoder to generate a continuous human action sequence that is consistent with the semantics of the input action description prompt text.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the text-to-action generation method based on fine-grained representation of body parts as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the text-to-action generation method based on fine-grained representation of body parts as described in any one of claims 1-7.

Images