Text-driven action sequence generation method, apparatus and storage medium

By quantifying the density of semantic and motion information and dynamically adjusting the guidance intensity of text and trajectory modalities, the problem of unnatural actions and semantic deviations caused by the imbalance of semantic and motion information in existing technologies is solved, achieving a balance between semantic accuracy and action naturalness, and generating natural and semantically accurate action sequences.

CN122389870BActive Publication Date: 2026-11-17MINZU UNIVERSITY OF CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202610265883.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-03-05
Publication Date
2026-11-17
Estimated Expiration
2046-03-05

AI Technical Summary

Technical Problem

In existing technologies, text-driven action generation methods suffer from an imbalance between semantic and motion information, resulting in unnatural generated actions or semantic deviations.

Method used

By quantifying the density of semantic and motion information in the time dimension, dynamically adjusting the guidance intensity of text and trajectory modalities, and constructing information-balanced embeddings to adaptively adjust influence weights, a balance between semantic accuracy and the naturalness of actions is achieved.

Benefits of technology

It achieves automatic adjustment of focus based on the modal information intensity at different stages during the generation process, achieving a better balance between semantic accuracy and action naturalness, and generating natural and semantically accurate action sequences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122389870B_ABST
    Figure CN122389870B_ABST
Patent Text Reader

Abstract

This application proposes a text-driven action sequence generation method, apparatus, and storage medium. The text-driven action sequence generation method includes: extracting dense semantic information distribution and sparse semantic information distribution; extracting dense motion information distribution and sparse motion information distribution; generating balanced and imbalanced embeddings based on a first joint distribution and a second joint distribution; modulating action trajectory features through balanced and imbalanced embeddings to obtain balanced and imbalanced feature streams; using the balanced feature stream as a query vector and the imbalanced feature stream as a key vector and value vector, performing attention calculation to obtain enhanced action trajectory features; fusing the enhanced action trajectory features with semantic features, predicting noise and updating the action sequence based on the fused features, until the iteration is complete and the target action sequence is output. This application achieves a better balance between semantic accuracy and action naturalness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a method, apparatus, and storage medium for generating text-driven action sequences. Background Technology

[0002] Human motion generation aims to synthesize 3D joint motion sequences based on text descriptions, and is widely used in animation, virtual reality, and robot motion planning. Text-based generation provides the most intuitive human-computer interaction method and has become a current research hotspot. In recent years, diffusion models, with their powerful distribution modeling capabilities, have become the mainstream framework for this task. These methods use text embedding as a condition and generate motion sequences through progressive denoising. This process is essentially an interaction between textual semantic information and trajectory motion information in the temporal dimension: semantic information carries the global action intent, while motion information emphasizes local temporal coherence.

[0003] However, existing methods mostly employ classic fusion strategies such as feature cascading or cross-attention, implicitly assuming that the text and trajectory contribute equally at any given time. In reality, the guiding strength of both changes dynamically over time, and regions of information imbalance can easily lead to generation defects: first, an overemphasis on semantic accuracy at the expense of motion coherence results in unnatural generated actions; second, an overemphasis on temporal smoothness at the expense of semantic fidelity leads to actions that deviate from the description or even become meaningless. Summary of the Invention

[0004] In view of this, this application proposes a text-driven action sequence generation method, apparatus and storage medium to solve the problems of unnatural actions caused by semantic-motion imbalance and semantic deviation caused by excessive smoothness enhancement in related technologies.

[0005] A first aspect of this application provides a method for generating text-driven action sequences, the method comprising: Obtain semantic text describing the target action, and extract the dense and sparse semantic information distributions of the semantic features of the semantic text in the time dimension; An initial noisy action sequence is obtained, and the action trajectory features of the initial noisy action sequence are iteratively denoised. In each iteration of denoising, the dense motion information distribution and sparse motion information distribution of the action trajectory features in the time dimension are extracted. The dense semantic information distribution and the sparse motion information distribution are concatenated to obtain a semantically dominant first joint distribution, and the sparse semantic information distribution and the dense motion information distribution are concatenated to obtain a motion-dominated second joint distribution. Information-balanced embeddings and information-imbalanced embeddings are generated based on the first joint distribution and the second joint distribution; the information-balanced embedding is used to characterize the consistency between the semantic features and the action trajectory features in the time dimension, and the information-imbalanced embedding is used to characterize the difference between the semantic features and the action trajectory features in the time dimension; The motion trajectory features are modulated by the balanced information embedding and the imbalanced information embedding respectively to obtain a balanced feature stream and an imbalanced feature stream; Using the balanced feature stream as the query vector and the unbalanced feature stream as the key vector and value vector, cross-modal attention computation is performed to obtain enhanced motion trajectory features; The enhanced action trajectory features are fused with the semantic features. Based on the fused features, noise is predicted and the action sequence is updated until the iteration is completed and the target action sequence corresponding to the semantic text is output.

[0006] This application embodiment extracts the dense and sparse semantic information distributions of semantic features from the semantic text along the time dimension; and iteratively denoises the action trajectory features of the initial noisy action sequence. In each iteration of denoising, the dense and sparse motion information distributions of the action trajectory features along the time dimension are extracted; enhanced action trajectory features are obtained using the dense, sparse, dense, and sparse semantic information distributions, and then fused with the semantic features. Based on the fused features, noise is predicted and the action sequence is updated until the iteration is complete and the target action sequence corresponding to the semantic text is output. This mechanism can adaptively adjust the influence weights of text and trajectory. Through this mechanism, the model can automatically adjust its focus based on the modal information intensity at different stages during generation, thereby achieving a better balance between semantic accuracy and action naturalness.

[0007] In this embodiment, the semantic features include multiple text lexical units, and multiple action trajectory features exist during the iterative denoising process, each action trajectory feature corresponding to a specific time step; extracting the dense and sparse semantic information distributions of the semantic text's semantic features in the time dimension includes: For any text word among the plurality of text words, a first curve is generated between the text word and the plurality of action trajectory features, resulting in a plurality of first curves corresponding one-to-one with the plurality of text words; each first curve includes a plurality of semantic correlation values, and each semantic correlation value is used to characterize the semantic correlation between the corresponding text word and the action trajectory features at the corresponding time step; For any one of the multiple time steps, based on the multiple first curves, determine multiple target semantic correlation values ​​between the trajectory features at that time step and the multiple text words; From the plurality of target semantic relevance values, the minimum semantic relevance value is selected to obtain a plurality of minimum semantic relevance values ​​that correspond one-to-one with the plurality of time steps; From the plurality of semantically related minimum values, a target semantically related minimum value greater than the first average value is selected, and the time step corresponding to the target semantically related minimum value is determined as the first target time step; the first target average value is the average of the minimum semantically related values ​​under the plurality of time steps, and the semantic information density of the first target time step is densely distributed.

[0008] In this embodiment of the application, the method further includes: From the plurality of target semantic relevance values, the maximum semantic relevance value is selected to obtain a plurality of maximum semantic relevance values ​​that correspond one-to-one with the plurality of time steps; From the plurality of semantically relevant maximum values, a target semantically relevant maximum value that is less than or equal to the second average value is selected, and the time step corresponding to the target semantically relevant maximum value is determined as the second target time step; the second average value is the average of the maximum semantically relevant values ​​under the plurality of time steps, and the semantic information density of the second target time step is sparsely distributed.

[0009] In this embodiment of the application, extracting the dense motion information distribution and sparse motion information distribution of the motion trajectory features in the time dimension includes: For any one of the multiple motion trajectory features, the change amplitude between the motion trajectory feature and the motion trajectory feature of the previous time step is calculated, and a second curve is generated based on the change amplitudes of all adjacent time steps; the second curve includes multiple change amplitudes, and each change amplitude is used to characterize the human motion change information between the corresponding adjacent time steps. The motion information density of the adjacent time steps is determined based on the change amplitude; the motion information density is either a dense motion information distribution or a sparse motion information distribution.

[0010] In this embodiment of the application, determining the motion information density of the adjacent time steps based on the change amplitude includes: For any adjacent time step, determine whether the change amplitude of the adjacent time step is greater than the average change based on the second curve; the average change is the average of the change amplitudes of all adjacent time steps. If the change amplitude of the adjacent time steps is greater than the average change, then the motion information density of the adjacent time steps is determined to be densely distributed. If the change amplitude of the adjacent time steps is less than or equal to the average change, then the motion information density of the adjacent time steps is determined to be sparsely distributed.

[0011] In this embodiment of the application, generating balanced and imbalanced embeddings based on the first joint distribution and the second joint distribution includes: The first joint distribution and the second joint distribution are concatenated to obtain a joint feature vector; The joint feature vector is input into a fully connected layer for linear transformation and mapped to a preset interval through an activation function to obtain the information balanced embedding, wherein the dimension of the information balanced embedding matches the time dimension of the trajectory features; The information-imbalanced embedding is constructed based on the information-balanced embedding; the information-balanced embedding and the information-imbalanced embedding are complementary.

[0012] In this embodiment of the application, predicting noise and updating the action sequence based on the fused features includes: The fused features and the current time step embedding are jointly input into the denoising network, and the prediction of unconditional noise and conditional noise are performed in parallel. The unconditional noise is the predicted noise obtained by inputting empty text features as conditions into the denoising network, and the conditional noise is the predicted noise obtained by inputting the semantic features as conditions into the denoising network. Based on the classifier's free guidance strategy, the unconditional noise and the conditional noise are fused according to the guidance scaling coefficient to obtain the target noise; The mean and variance of the denoised action distribution are calculated based on the target noise. Depending on whether the current time step is greater than a preset threshold, the action distribution is sampled or the mean is taken to obtain the updated action sequence.

[0013] An embodiment of the second aspect of this application provides a text-driven action sequence generation apparatus, comprising: The semantic information distribution extraction module is used to obtain semantic text describing the target action and extract the dense semantic information distribution and sparse semantic information distribution of the semantic features of the semantic text in the time dimension. The motion information distribution extraction module is used to acquire an initial noisy action sequence and iteratively denoise the action trajectory features of the initial noisy action sequence; in each iteration of denoising, the dense motion information distribution and sparse motion information distribution of the action trajectory features in the time dimension are extracted. The joint distribution generation module is used to concatenate the dense semantic information distribution with the sparse motion information distribution to obtain a semantically dominant first joint distribution, and to concatenate the sparse semantic information distribution with the dense motion information distribution to obtain a motion-dominant second joint distribution; An information-balanced embedding generation module is used to generate information-balanced embeddings and information-imbalanced embeddings based on the first joint distribution and the second joint distribution; the information-balanced embedding is used to characterize the consistency between the semantic features and the action trajectory features in the time dimension, and the information-imbalanced embedding is used to characterize the difference between the semantic features and the action trajectory features in the time dimension. The balanced feature stream generation module is used to modulate the motion trajectory features through the information balanced embedding and the information unbalanced embedding to obtain a balanced feature stream and an unbalanced feature stream, respectively. The cross-modal attention calculation module is used to perform cross-modal attention calculation using the balanced feature stream as the query vector and the unbalanced feature stream as the key vector and value vector to obtain enhanced motion trajectory features; The target action sequence generation module is used to fuse the enhanced action trajectory features with the semantic features, predict noise and update the action sequence based on the fused features, until the iteration is completed and the target action sequence corresponding to the semantic text is output.

[0014] An embodiment of the third aspect of this application provides a computer device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the text-driven action sequence generation method described in the first aspect above.

[0015] An embodiment of the fourth aspect of this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the text-driven action sequence generation method described in the first aspect above.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a text-driven action sequence generation method provided in an embodiment of this application is shown. Figure 2 This illustration shows a schematic diagram of the structure of a text-driven action sequence generation device according to an embodiment of this application; Figure 3 This illustration shows a schematic diagram of the structure of a computer device according to an embodiment of this application; Figure 4 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0018] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0019] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0020] The technical scenarios involved in the embodiments of this application are described below.

[0021] In recent years, diffusion models have become the mainstream framework for motion generation due to their powerful ability to model complex distributions. These methods typically use text embeddings as conditional input and generate semantically consistent action sequences by progressively denoising trajectory features. Essentially, this process reflects the interaction between semantic information in the text modality and motion information in the trajectory modality over time: semantic information primarily carries the high-level semantics of global actions, while motion information emphasizes the smoothness and continuity of local temporal sequences. Therefore, generating natural and semantically accurate actions requires dynamically coordinating the expression of semantic information and motion at different time steps.

[0022] Most existing research relies on classic feature fusion strategies, such as feature concatenation or cross-attention, to uniformly mix semantic and motion information over time. This approach implicitly assumes that text and trajectory contribute equally to the generated result at any given time. While this can achieve some success in practice, this assumption is not valid. In reality, the guiding strength of text and trajectory changes dynamically over time, and action sequences with significant deviations between the two are more prone to errors, including two main problems: 1) The model excessively pursues semantic accuracy while neglecting action coherence, resulting in unnatural generated actions; 2) The model overemphasizes temporal smoothness at the expense of semantic fidelity, potentially leading to semantic deviations or even meaningless actions in the generated result.

[0023] To address the aforementioned limitations, this application proposes a simple yet effective Info-Balanced Motion Generator (IBMG). Its core idea is to dynamically balance the guidance strength of the text and trajectory modalities from an information density perspective, mitigating motion distortion caused by their imbalance. Specifically: First, the information density of the two modalities is quantified in the time dimension. For the text modal, the matching degree between the trajectory features at the current time step and all semantic tokens is used as the basis: if the minimum matching degree between the time step and the semantic tokens is still high, it indicates strong semantic constraints and high information density; conversely, it indicates weak text information contribution. For the trajectory modal, the amplitude of motion change between adjacent time steps is calculated: a larger change indicates more posture changes or dynamic details at that moment, i.e., higher motion information density. Subsequently, the semantic and motion information densities are normalized and compared to identify regions with significant information distribution imbalances. For these imbalanced regions, semantic and action information distributions are constructed separately and encoded into an information-balanced embedding. This embedding is explicitly injected into the diffusion generation process to adaptively adjust the influence weights of text and trajectory. Through this mechanism, the model can automatically adjust its focus based on the intensity of modal information at different stages during generation, thereby achieving a better balance between semantic accuracy and naturalness of action.

[0024] According to an embodiment of this application, a text-driven action sequence generation method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] This embodiment provides a text-driven action sequence generation method, which can be used in the aforementioned information-balanced action generator. Figure 1 This is a flowchart of a text-driven action sequence generation method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the semantic text describing the target action, and extract the dense semantic information distribution and sparse semantic information distribution of the semantic features of the semantic text in the time dimension.

[0026] Specifically, the semantic text describing the target action can be input into the CLIP text encoder to obtain semantic features containing multiple text units, such as {v1, v2, ..., vn}.

[0027] In some specific embodiments, step S101 above includes steps S1011-S1014: Step S1011: For any text word among the plurality of text words, generate a first curve of the text word and the plurality of action trajectory features, thereby obtaining a plurality of first curves that correspond one-to-one with the plurality of text words.

[0028] Specifically, each first curve includes multiple semantic correlation values, each semantic correlation value being used to characterize the semantic correlation between the corresponding text word and the action trajectory features at the corresponding time step.

[0029] More specifically, the semantic correlation value between each text word and its corresponding action trajectory feature can be calculated using cosine similarity, as shown below:

[0030] in, Indicates the first Embedded representation of text terms, Indicates time step movement trajectory characteristics, " indicates the dot product operation. Denotes the Euclidean norm. Indicates the first Text lexical units and time steps The semantic correlation values ​​between action trajectory features.

[0031] In this embodiment, at each time step, the semantic correlation between all text words and action trajectory features is calculated, thereby generating a first curve for each text word. The value of each curve represents the semantic alignment strength between the text word and the corresponding action trajectory feature in the time dimension; a larger value indicates a stronger alignment. For multiple text words, multiple curves corresponding one-to-one can be obtained, and these curves are used to represent the distribution of semantic information in the text modality in the time dimension.

[0032] Step S1012: For any time step among the multiple time steps, determine multiple target semantic correlation values ​​between the trajectory features at the time step and the multiple text words based on the multiple first curves.

[0033] Specifically, multiple semantic correlation values ​​of each first curve are used to characterize the semantic correlation between the same text word and the action trajectory features at multiple time steps. For example, the first curve 1 includes N semantic correlation values, namely a1, a2, ..., an; where a1 represents the semantic correlation between text word 1 and the action trajectory features at time step t1, a2 represents the semantic correlation between text word 1 and the action trajectory features at time step t2, and an represents the semantic correlation between text word 1 and the action trajectory features at time step tn.

[0034] More specifically, for any given time step, the semantic relevance value corresponding to the motion trajectory feature at that time step is determined from each first curve. For example, when the time step is time step t1, the target semantic relevance value a1 can be selected from the first curve 1. In this way, multiple target semantic relevance values ​​corresponding to the time step can be determined from multiple first curves.

[0035] Step S1013: Select the minimum semantic correlation value from the plurality of target semantic correlation values ​​to obtain a plurality of minimum semantic correlation values ​​corresponding one-to-one with the plurality of time steps.

[0036] Specifically, for any given time step among multiple time steps, the minimum semantic relevance value can be selected from multiple target semantic relevance values ​​corresponding to that time step, as shown below:

[0037] in, Indicates time step The minimum semantic relevance at that location. Indicates the first Text lexical units and time steps The semantic correlation values ​​between action trajectory features.

[0038] Step S1014: Select the target semantic correlation minimum value that is greater than the first average value from the plurality of semantic correlation minimum values, and determine the time step corresponding to the target semantic correlation minimum value as the first target time step.

[0039] Specifically, the semantic information density of the first target time step is densely distributed, and the first target average value is the average of the minimum semantic relevance values ​​under the multiple time steps.

[0040] More specifically, the target semantic relevance minimum value that is greater than the first average value can be selected from the plurality of semantic relevance minimum values ​​using the following formula:

[0041] in, Indicates time step The semantic information density at that location is densely distributed. This represents the average of the minimum semantic relevance values ​​across all time steps.

[0042] In some specific embodiments, after step S1014, the method further includes: Step S1015: Select the maximum semantic correlation value from the plurality of target semantic correlation values ​​to obtain a plurality of maximum semantic correlation values ​​that correspond one-to-one with the plurality of time steps.

[0043] Specifically, for any given time step among multiple time steps, the maximum semantic relevance value can be selected from multiple target semantic relevance values ​​corresponding to that time step, as shown below:

[0044] in, Indicates time step The maximum semantic relevance at that location, Indicates the first Text lexical units and time steps The semantic correlation values ​​between action trajectory features.

[0045] Step S1016: Select a target semantic correlation maximum value that is less than or equal to the second average value from the plurality of semantic correlation maximum values, and determine the time step corresponding to the target semantic correlation maximum value as the second target time step.

[0046] Specifically, the second average value is the average of the maximum semantic relevance values ​​at the plurality of time steps, and the semantic information density of the second target time step is sparsely distributed.

[0047] More specifically, the target semantic relevance maximum value, which is less than or equal to the second average value, can be selected from the plurality of semantic relevance maximum values ​​using the following formula:

[0048] in, Indicates time step The semantic information density at this location is sparsely distributed. This represents the average of the maximum semantic relevance values ​​across all time steps.

[0049] In the embodiments of this application, by converting semantic information into dense and sparse semantic information distributions, the model can effectively map text modalities into comparable distribution states.

[0050] Step S102: Obtain the initial noisy action sequence, and iteratively denoise the action trajectory features of the initial noisy action sequence; in each iteration of denoising, extract the dense motion information distribution and sparse motion information distribution of the action trajectory features in the time dimension.

[0051] Specifically, there are multiple action trajectory features in the iterative denoising process, and each action trajectory feature corresponds to a specific time step.

[0052] In some specific embodiments, step S102 above includes steps S1021-S1022: Step S1021: For any one of the multiple motion trajectory features, calculate the change amplitude between the motion trajectory feature and the motion trajectory feature of the previous time step, and generate a second curve based on the change amplitude of all adjacent time steps.

[0053] Specifically, the second curve includes multiple amplitude values, each of which is used to characterize the human motion change information between corresponding adjacent time steps. Each amplitude value can be characterized by the difference between corresponding adjacent time steps, as shown below:

[0054] in, This indicates the element-wise absolute value operation. The difference is used to characterize the time step. The characteristics of the motion trajectory at the location and the previous time step The amplitude of change between the motion trajectory features at -1.

[0055] This application embodiment calculates the difference between adjacent time steps. This yields a second curve that varies with time, where each point reflects the magnitude of change between adjacent time steps, used to illustrate the density distribution of motion trajectory features in the time dimension.

[0056] Step S1022: Determine the motion information density of the adjacent time steps based on the change amplitude.

[0057] Specifically, the motion information density is either a dense motion information distribution or a sparse motion information distribution.

[0058] In some specific embodiments, step S1022 above further includes steps a1-a3: Step a1: For any adjacent time step, determine whether the change amplitude of the adjacent time step is greater than the average change based on the second curve.

[0059] Specifically, the average change is the average of the change magnitudes across all adjacent time steps, and can be expressed by the following formula:

[0060] in, Indicates time step The characteristics of the motion trajectory at the location and the previous time step The amplitude of the change between the motion trajectory features at -1 This represents the number of all time steps. This represents the average value of the changes over all adjacent time steps, i.e., the average change.

[0061] Step a2: If the change amplitude of the adjacent time step is greater than the average change, then the motion information density of the adjacent time step is determined to be densely distributed.

[0062] Step a3: If the change amplitude of the adjacent time steps is less than or equal to the average change, then the motion information density of the adjacent time steps is determined to be sparsely distributed.

[0063] In the embodiments of this application, the following are used To quantify time steps The range of change in the characteristics of the motion trajectory, if If the change is greater than the average change over all time steps, it indicates a significant change in motion, thus possessing high information density, i.e., a dense distribution; otherwise, A change less than or equal to the average indicates a smoother motion and lower information density, i.e., a sparse distribution, as shown below:

[0064]

[0065] in, Indicates time step With time step The motion information density between -1 and 1 is densely distributed. Indicates time step With time step The motion information density between -1 and -1 is sparsely distributed.

[0066] Step S103: The dense semantic information distribution and the sparse motion information distribution are concatenated to obtain a semantically dominant first joint distribution, and the sparse semantic information distribution and the dense motion information distribution are concatenated to obtain a motion-dominated second joint distribution.

[0067] Specifically, dense semantic information can be distributed Distribution of sparse motion information The data is concatenated and processed through convolutional layers to obtain the semantically dominant first joint distribution. Conversely, sparse semantic information can be distributed... Distribution of dense motion information The data is then concatenated and processed through convolutional layers to obtain the motion-dominated second joint distribution. As shown below:

[0068]

[0069] Step S104: Generate information-balanced embeddings and information-imbalanced embeddings based on the first joint distribution and the second joint distribution.

[0070] Specifically, the information-balanced embedding is used to characterize the consistency between the semantic features and the action trajectory features in the time dimension, and the information-imbalanced embedding is used to characterize the difference between the semantic features and the action trajectory features in the time dimension.

[0071] In some specific embodiments, step S104 above includes steps S1041-S1043: Step S1041: Concatenate the first joint distribution and the second joint distribution to obtain a joint feature vector.

[0072] Step S1042: Input the joint feature vector into the fully connected layer for linear transformation, and map it to a preset interval through an activation function to obtain the information balanced embedding, wherein the dimension of the information balanced embedding matches the time dimension of the trajectory features; Step S1043: Construct the information imbalanced embedding based on the information balanced embedding; the information balanced embedding is complementary to the information imbalanced embedding.

[0073] In this embodiment of the application, the first joint distribution can be With the second joint distribution Concatenate them into a joint feature vector It is then input into a linear transformation and a sigmoid activation function to generate an information-balanced embedding. :

[0074] in, This represents the Sigmoid activation function. and These represent the weight matrix and bias of the linear transformation, respectively. The resulting balanced embedding information... It can represent the balance between semantic text and action trajectory features in the time dimension.

[0075] More specifically, to enhance the model's sensitivity to modal imbalance, we use information-balanced embedding. An information-balanced modulation module was designed, namely: constructing complementary information-imbalanced embeddings. This is used to highlight areas where semantic text and action trajectory features are more likely to differ.

[0076] Step S105: Modulate the motion trajectory features using the balanced information embedding and the unbalanced information embedding respectively to obtain a balanced feature stream and an unbalanced feature stream.

[0077] Specifically, information can be balanced and embedded using the Hadamard product. Embedded with information imbalance Integrating into motion trajectory features In this process, two modulated characteristic streams are obtained: a balanced characteristic stream and an unbalanced characteristic stream.

[0078] Step S106: Using the balanced feature stream as the query vector and the unbalanced feature stream as the key vector and value vector, perform cross-modal attention calculation to obtain enhanced motion trajectory features.

[0079] Specifically, the balanced feature stream can be mapped to a query matrix through linear projection. And by linear projection, the unbalanced feature streams are mapped to key matrices respectively. Sum matrix Then calculate the scaled dot product attention:

[0080] in, is the dimension of the key vector; the attention output is concatenated with the semantic text embedding as an enhanced action trajectory feature.

[0081] Step S107: The enhanced action trajectory features are fused with the semantic features. Based on the fused features, noise is predicted and the action sequence is updated until the iteration is completed and the target action sequence corresponding to the semantic text is output.

[0082] In some specific embodiments, step S107 above includes steps S1071-S1073: Step S1071: The fused features are embedded into the current time step and input into the denoising network, and predictions of unconditional noise and conditional noise are performed in parallel.

[0083] Specifically, the unconditional noise is the predicted noise obtained by inputting the empty text features as conditions into the denoising network, and the conditional noise is the predicted noise obtained by inputting the semantic features as conditions into the denoising network.

[0084] Step S1072: Based on the classifier's free guidance strategy, the unconditional noise and the conditional noise are fused according to the guidance scaling coefficient to obtain the target noise.

[0085] Step S1073: Calculate the mean and variance of the denoised action distribution based on the target noise. Depending on whether the current time step is greater than a preset threshold, sample or take the mean from the action distribution to obtain the updated action sequence.

[0086] Corresponding to the above implementation of the text-driven action sequence generation method, this application also provides a text-driven action sequence generation apparatus for executing the text-driven action sequence generation method described in any of the above embodiments. For example... Figure 2 As shown, the text-driven action sequence generation device includes: The semantic information distribution extraction module is used to obtain semantic text describing the target action and extract the dense semantic information distribution and sparse semantic information distribution of the semantic features of the semantic text in the time dimension. The motion information distribution extraction module is used to acquire an initial noisy action sequence and iteratively denoise the action trajectory features of the initial noisy action sequence; in each iteration of denoising, the dense motion information distribution and sparse motion information distribution of the action trajectory features in the time dimension are extracted. The joint distribution generation module is used to concatenate the dense semantic information distribution with the sparse motion information distribution to obtain a semantically dominant first joint distribution, and to concatenate the sparse semantic information distribution with the dense motion information distribution to obtain a motion-dominant second joint distribution; An information-balanced embedding generation module is used to generate information-balanced embeddings and information-imbalanced embeddings based on the first joint distribution and the second joint distribution; the information-balanced embedding is used to characterize the consistency between the semantic features and the action trajectory features in the time dimension, and the information-imbalanced embedding is used to characterize the difference between the semantic features and the action trajectory features in the time dimension. The balanced feature stream generation module is used to modulate the motion trajectory features through the information balanced embedding and the information unbalanced embedding to obtain a balanced feature stream and an unbalanced feature stream, respectively. The cross-modal attention calculation module is used to perform cross-modal attention calculation using the balanced feature stream as the query vector and the unbalanced feature stream as the key vector and value vector to obtain enhanced motion trajectory features; The target action sequence generation module is used to fuse the enhanced action trajectory features with the semantic features, predict noise and update the action sequence based on the fused features, until the iteration is completed and the target action sequence corresponding to the semantic text is output.

[0087] Optionally, the semantic information distribution extraction module is further configured to generate a first curve between any text word in the plurality of text words and the plurality of action trajectory features, thereby obtaining a plurality of first curves corresponding one-to-one with the plurality of text words; each first curve includes a plurality of semantic correlation values, each semantic correlation value being used to characterize the semantic correlation between the corresponding text word and the action trajectory features at the corresponding time step; for any time step in the plurality of time steps, based on the plurality of first curves, determine a plurality of target semantic correlation values ​​between the trajectory features at the time step and the plurality of text words; filter out the minimum semantic correlation value from the plurality of target semantic correlation values, thereby obtaining a plurality of minimum semantic correlation values ​​corresponding one-to-one with the plurality of time steps; filter out values ​​greater than from the plurality of minimum semantic correlation values. The target semantic relevance minimum is determined by the first average value, and the time step corresponding to the target semantic relevance minimum value is determined as the first target time step. The first target average value is the average of the minimum semantic relevance values ​​under the plurality of time steps, and the semantic information density of the first target time step is densely distributed. The maximum semantic relevance value is selected from the plurality of target semantic relevance values ​​to obtain a plurality of maximum semantic relevance values ​​that correspond one-to-one with the plurality of time steps. The target semantic relevance maximum value that is less than or equal to the second average value is selected from the plurality of maximum semantic relevance values, and the time step corresponding to the target semantic relevance maximum value is determined as the second target time step. The second average value is the average of the maximum semantic relevance values ​​under the plurality of time steps, and the semantic information density of the second target time step is sparsely distributed.

[0088] Optionally, the motion information distribution extraction module is further configured to calculate the change amplitude between any of the multiple motion trajectory features and the motion trajectory feature of the previous time step, and generate a second curve based on the change amplitudes of all adjacent time steps; the second curve includes multiple change amplitudes, each change amplitude being used to characterize the human motion change information between corresponding adjacent time steps; the motion information density of the adjacent time steps is determined according to the change amplitudes; the motion information density is either a dense motion information distribution or a sparse motion information distribution.

[0089] Optionally, the motion information distribution extraction module is further configured to, for any adjacent time step, determine whether the change amplitude of the adjacent time step is greater than the average change based on the second curve; the average change is the average of the change amplitudes of all adjacent time steps; if the change amplitude of the adjacent time step is greater than the average change, then the motion information density of the adjacent time step is determined to be densely distributed; if the change amplitude of the adjacent time step is less than or equal to the average change, then the motion information density of the adjacent time step is determined to be sparsely distributed.

[0090] Optionally, the information-balanced embedding generation module is further configured to concatenate the first joint distribution and the second joint distribution to obtain a joint feature vector; input the joint feature vector into a fully connected layer for linear transformation, and map it to a preset interval through an activation function to obtain the information-balanced embedding, wherein the dimension of the information-balanced embedding matches the time dimension of the trajectory features; construct the information-imbalanced embedding based on the information-balanced embedding; the information-balanced embedding and the information-imbalanced embedding are complementary.

[0091] Optionally, the target action sequence generation module is further configured to embed the fused features and the current time step into a common input denoising network, and perform parallel prediction of unconditional noise and conditional noise; the unconditional noise is the predicted noise obtained by inputting empty text features as conditions into the denoising network, and the conditional noise is the predicted noise obtained by inputting the semantic features as conditions into the denoising network; based on the classifier's free guidance strategy, the unconditional noise and the conditional noise are fused according to the guidance scale coefficient to obtain target noise; based on the target noise, the mean and variance of the denoised action distribution are calculated, and depending on whether the current time step is greater than a preset threshold, the action distribution is sampled or the mean is taken to obtain the updated action sequence.

[0092] The text-driven action sequence generation device and the text-driven action sequence generation method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0093] This application also provides a computer device for executing the above-described text-driven action sequence generation method. Please refer to... Figure 3 This illustrates a schematic diagram of a computer device provided by some embodiments of this application. For example... Figure 3 As shown, the computer device 3 includes: a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected via the bus 302. The memory 301 stores a computer program that can run on the processor 300. When the processor 300 runs the computer program, it executes the text-driven action sequence generation method provided in any of the foregoing embodiments of this application.

[0094] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0095] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 301 is used to store programs. After receiving an execution instruction, the processor 300 executes the program. The text-driven action sequence generation method disclosed in any of the foregoing embodiments can be applied to the processor 300, or implemented by the processor 300.

[0096] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.

[0097] The computer device provided in this application embodiment and the text-driven action sequence generation method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, run or implement.

[0098] This application also provides a computer-readable storage medium corresponding to the text-driven action sequence generation method provided in the foregoing embodiments. Please refer to... Figure 4 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the text-driven action sequence generation method provided in any of the foregoing embodiments.

[0099] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0100] The computer-readable storage medium provided in the above embodiments of this application and the text-driven action sequence generation method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0101] It should be noted that: Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0102] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0103] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0104] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating text-driven action sequences, characterized in that, The method includes: Obtain semantic text describing the target action, and extract the dense and sparse semantic information distributions of the semantic features of the semantic text in the time dimension; An initial noisy action sequence is obtained, and the action trajectory features of the initial noisy action sequence are iteratively denoised. In each iteration of denoising, the dense motion information distribution and sparse motion information distribution of the action trajectory features in the time dimension are extracted. The dense semantic information distribution and the sparse motion information distribution are concatenated to obtain a semantically dominant first joint distribution, and the sparse semantic information distribution and the dense motion information distribution are concatenated to obtain a motion-dominated second joint distribution. Information-balanced embeddings and information-imbalanced embeddings are generated based on the first joint distribution and the second joint distribution; the information-balanced embedding is used to characterize the consistency between the semantic features and the action trajectory features in the time dimension, and the information-imbalanced embedding is used to characterize the difference between the semantic features and the action trajectory features in the time dimension; The motion trajectory features are modulated by the balanced information embedding and the imbalanced information embedding respectively to obtain a balanced feature stream and an imbalanced feature stream; Using the balanced feature stream as the query vector and the unbalanced feature stream as the key vector and value vector, cross-modal attention computation is performed to obtain enhanced motion trajectory features. The enhanced action trajectory features are fused with the semantic features. Based on the fused features, noise is predicted and the action sequence is updated until the iteration is completed and the target action sequence corresponding to the semantic text is output. In the iterative denoising process, there are multiple motion trajectory features, each corresponding to a specific time step; the dense motion information distribution and sparse motion information distribution of the motion trajectory features in the time dimension are extracted, including: For any one of the multiple motion trajectory features, the change amplitude between the motion trajectory feature and the motion trajectory feature of the previous time step is calculated, and a second curve is generated based on the change amplitudes of all adjacent time steps; the second curve includes multiple change amplitudes, and each change amplitude is used to characterize the human motion change information between the corresponding adjacent time steps. The motion information density of the adjacent time steps is determined based on the change amplitude; the motion information density is either a dense motion information distribution or a sparse motion information distribution.

2. The method according to claim 1, characterized in that, The semantic features include multiple text terms; the extraction of the dense and sparse semantic information distributions of the semantic text over time includes: For any text word among the plurality of text words, a first curve is generated between the text word and the plurality of action trajectory features, resulting in a plurality of first curves corresponding one-to-one with the plurality of text words; each first curve includes a plurality of semantic correlation values, and each semantic correlation value is used to characterize the semantic correlation between the corresponding text word and the action trajectory features at the corresponding time step; For any one of the multiple time steps, based on the multiple first curves, determine multiple target semantic correlation values ​​between the trajectory features at that time step and the multiple text words; From the plurality of target semantic relevance values, the minimum semantic relevance value is selected to obtain a plurality of minimum semantic relevance values ​​that correspond one-to-one with the plurality of time steps; From the plurality of semantically related minimum values, a target semantically related minimum value greater than the first average value is selected, and the time step corresponding to the target semantically related minimum value is determined as the first target time step; the first average value is the average of the minimum semantically related values ​​under the plurality of time steps, and the semantic information density of the first target time step is densely distributed.

3. The method according to claim 2, characterized in that, The method further includes: From the plurality of target semantic relevance values, the maximum semantic relevance value is selected to obtain a plurality of maximum semantic relevance values ​​that correspond one-to-one with the plurality of time steps; From the plurality of semantically relevant maximum values, a target semantically relevant maximum value that is less than or equal to the second average value is selected, and the time step corresponding to the target semantically relevant maximum value is determined as the second target time step; the second average value is the average of the maximum semantically relevant values ​​under the plurality of time steps, and the semantic information density of the second target time step is sparsely distributed.

4. The method according to claim 1, characterized in that, Determining the motion information density of adjacent time steps based on the change amplitude includes: For any adjacent time step, determine whether the change amplitude of the adjacent time step is greater than the average change based on the second curve; the average change is the average of the change amplitudes of all adjacent time steps. If the change amplitude of the adjacent time steps is greater than the average change, then the motion information density of the adjacent time steps is determined to be densely distributed. If the change amplitude of the adjacent time steps is less than or equal to the average change, then the motion information density of the adjacent time steps is determined to be sparsely distributed.

5. The method according to claim 1 or 2, characterized in that, Information-balanced embeddings and information-imbalanced embeddings are generated based on the first joint distribution and the second joint distribution, including: The first joint distribution and the second joint distribution are concatenated to obtain a joint feature vector; The joint feature vector is input into a fully connected layer for linear transformation and mapped to a preset interval through an activation function to obtain the information balanced embedding, wherein the dimension of the information balanced embedding matches the time dimension of the trajectory features; The information-imbalanced embedding is constructed based on the information-balanced embedding; the information-balanced embedding and the information-imbalanced embedding are complementary.

6. The method according to claim 1 or 2, characterized in that, Based on the fused features, noise is predicted and the action sequence is updated, including: The fused features and the current time step embedding are jointly input into the denoising network, and the prediction of unconditional noise and conditional noise are performed in parallel. The unconditional noise is the predicted noise obtained by inputting empty text features as conditions into the denoising network, and the conditional noise is the predicted noise obtained by inputting the semantic features as conditions into the denoising network. Based on the classifier's free guidance strategy, the unconditional noise and the conditional noise are fused according to the guidance scaling coefficient to obtain the target noise; The mean and variance of the denoised action distribution are calculated based on the target noise. Depending on whether the current time step is greater than a preset threshold, the action distribution is sampled or the mean is taken to obtain the updated action sequence.

7. A text-driven action sequence generation device, characterized in that, The device includes: The semantic information distribution extraction module is used to obtain semantic text describing the target action and extract the dense semantic information distribution and sparse semantic information distribution of the semantic features of the semantic text in the time dimension. The motion information distribution extraction module is used to acquire an initial noisy action sequence and iteratively denoise the action trajectory features of the initial noisy action sequence; in each iteration of denoising, the dense motion information distribution and sparse motion information distribution of the action trajectory features in the time dimension are extracted. The joint distribution generation module is used to concatenate the dense semantic information distribution with the sparse motion information distribution to obtain a semantically dominant first joint distribution, and to concatenate the sparse semantic information distribution with the dense motion information distribution to obtain a motion-dominant second joint distribution; An information-balanced embedding generation module is used to generate information-balanced embeddings and information-imbalanced embeddings based on the first joint distribution and the second joint distribution; the information-balanced embedding is used to characterize the consistency between the semantic features and the action trajectory features in the time dimension, and the information-imbalanced embedding is used to characterize the difference between the semantic features and the action trajectory features in the time dimension. A balanced feature stream generation module is used to modulate the motion trajectory features through the information balanced embedding and the information unbalanced embedding to obtain a balanced feature stream and an unbalanced feature stream, respectively. The cross-modal attention calculation module is used to perform cross-modal attention calculation using the balanced feature stream as the query vector and the unbalanced feature stream as the key vector and value vector to obtain enhanced motion trajectory features; The target action sequence generation module is used to fuse the enhanced action trajectory features with the semantic features, predict noise and update the action sequence based on the fused features, until the iteration is completed and the target action sequence corresponding to the semantic text is output. In the iterative denoising process, there are multiple motion trajectory features, each corresponding to a specific time step; the dense motion information distribution and sparse motion information distribution of the motion trajectory features in the time dimension are extracted, including: For any one of the multiple motion trajectory features, the change amplitude between the motion trajectory feature and the motion trajectory feature of the previous time step is calculated, and a second curve is generated based on the change amplitudes of all adjacent time steps; the second curve includes multiple change amplitudes, and each change amplitude is used to characterize the human motion change information between the corresponding adjacent time steps. The motion information density of the adjacent time steps is determined based on the change amplitude; the motion information density is either a dense motion information distribution or a sparse motion information distribution.

8. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the text-driven action sequence generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the text-driven action sequence generation method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Text-driven three-dimensional human body action generation method

    CN119810382A

  • Motion generation method and device based on multi-modal signal, equipment and storage medium

    CN120931775A