Action generation method and device based on joint optimization of frequency domain and text semantic information

Abstract

This application provides a method and apparatus for action generation based on joint optimization of frequency domain and text semantic information, relating to the field of action generation technology. The method includes: decomposing the initial latent spatial motion features into low-frequency and high-frequency components; obtaining the temporal action features based on the low-frequency and high-frequency components; concatenating the temporal action features with the text features to be processed to obtain fused features; and using the fused features as input to output an action sequence using a pre-trained action generation model. By designing a frequency state space model, effectively combining low-frequency and high-frequency information, the model's ability to model static poses and dynamic detailed actions is improved. Compared with traditional methods, this application can generate more natural and coherent action sequences, especially performing well in handling complex multi-stage actions, significantly reducing the phenomenon of overly abrupt or lacking detail between actions.

Description

Technical Field

[0001] This application relates to the field of action generation technology, and in particular to an action generation method and apparatus based on joint optimization of frequency domain and text semantic information. Background Technology

[0002] In the field of human motion generation, generating natural and physically consistent action sequences based on conditional inputs (such as text descriptions) is a widely studied technique. This technology has broad applications in animation, virtual reality (VR) / augmented reality (AR), game development, and human-computer interaction. Current mainstream methods mainly include autoencoder models, generative adversarial networks (GANs), and autoregressive models. These methods provide varying degrees of support for motion generation, but still have many limitations in generating complex, coherent, and detailed action sequences. In recent years, diffusion models have been gradually introduced into the field of motion generation due to their excellent performance in image and video generation. Diffusion models gradually restore the distribution of target data through denoising, enabling the generation of more natural actions. However, this technique still faces significant challenges in practical applications.

[0003] Existing motion generation methods based on diffusion models fall short in generating fine-grained motions, particularly in capturing the transitional details between static poses and dynamic movements. For example, for complex commands such as "get up-walk-lie down," existing models struggle to smoothly capture the subtle changes between each stage of the action, resulting in stiff or unnatural generated images that fail to accurately simulate the natural transitions and nuanced performance of human actions. Summary of the Invention

[0004] The purpose of this application is to provide an action generation method and apparatus based on joint optimization of frequency domain and text semantic information, which can capture the detailed changes between actions at each stage and produce more natural action generation results.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides an action generation method based on joint optimization of frequency domain and textual semantic information, including:

[0007] The text features to be processed and the corresponding initial latent spatial motion features are obtained, wherein the text features to be processed are variables obtained by feature extraction of the text description to be processed, and the initial latent spatial motion features are variables randomly sampled from a Gaussian distribution;

[0008] The initial potential spatial motion features are used as input, and the time-domain action features are output using a pre-trained frequency-space state model.

[0009] The temporal action features and the text features to be processed are concatenated to obtain the fused features;

[0010] The fused features are used as input, and a pre-trained action generation model is used to output an action sequence.

[0011] Using the initial potential spatial motion features as input, and utilizing a pre-trained frequency-space state model, the temporal action features are output, specifically including:

[0012] The initial potential spatial motion features are decomposed into low-frequency components and high-frequency components, wherein the low-frequency components are used to capture the global static motion features of the motion, and the high-frequency components are used to capture the dynamic motion detail features of the motion.

[0013] The time-domain action features are obtained based on the low-frequency components and the high-frequency components.

[0014] In a second aspect, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the action generation method based on joint optimization of frequency domain and text semantic information as described in the first aspect.

[0015] Thirdly, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the action generation method based on joint optimization of frequency domain and text semantic information as described in the first aspect.

[0016] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the action generation method based on joint optimization of frequency domain and textual semantic information as described in the first aspect.

[0017] According to the specific embodiments provided in this application, this application has the following technical effects:

[0018] This application provides a method and apparatus for action generation based on joint optimization of frequency domain and text semantic information. The method includes: decomposing the initial latent spatial motion features into low-frequency components and high-frequency components; obtaining the temporal action features based on the low-frequency components and the high-frequency components; concatenating the temporal action features and the text features to be processed to obtain fused features; and using the fused features as input, outputting an action sequence using a pre-trained action generation model. By designing a Frequency State Space Model (FreqSSM), the method effectively combines low-frequency and high-frequency information, improving the model's ability to model static poses and dynamic detailed actions. Compared with traditional methods, this application can generate more natural and coherent action sequences, especially performing well in handling complex multi-stage actions (such as "getting up-walking-lying down"), significantly reducing the phenomenon of overly abrupt or lacking details between actions. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 An architecture diagram of an action generation method based on joint optimization of frequency domain and text semantic information provided in Embodiment 1 of this application;

[0021] Figure 2 This is a diagram illustrating the architecture of the frequency state-space model in Embodiment 1 of this application;

[0022] Figure 3 This is an architecture diagram of the text state space model in Embodiment 1 of this application;

[0023] Figure 4 This is a schematic diagram of the action generation result in Embodiment 1 of this application;

[0024] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 2 of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1

[0028] Research has revealed that existing diffusion models struggle to smoothly capture the subtle changes between different stages of the action, resulting in stiff or unnatural output. This is primarily because current methods fail to fully utilize the frequency characteristics of the action sequence. Specifically, low-frequency information is crucial for static pose modeling, while high-frequency information determines the subtle changes in the action. However, existing diffusion models lack the ability to organically combine these two types of information.

[0029] To address this, this embodiment provides an action generation method based on joint optimization of frequency domain and textual semantic information, including:

[0030] S1: Obtain the text features to be processed and the corresponding initial latent spatial motion features, wherein the text features to be processed are variables obtained by feature extraction of the text description to be processed, and the initial latent spatial motion features are variables randomly sampled from a Gaussian distribution;

[0031] S2: Using the initial potential spatial motion features as input, output temporal action features using a pre-trained frequency-space state model;

[0032] S3: Concatenate the temporal action features and the text features to be processed to obtain the fused features;

[0033] S4: Using the fused features as input, the pre-trained action generation model is used to output an action sequence, wherein the action generation model is a diffusion model;

[0034] S21: Decompose the initial potential spatial motion features into low-frequency components and high-frequency components, wherein the low-frequency components are used to capture the global static motion features of the motion, and the high-frequency components are used to capture the dynamic motion detail features of the motion.

[0035] S22: Construct a frequency domain state transition matrix based on the low-frequency component, the high-frequency component, and the scaling parameter, wherein the scaling parameter is a variable determined by the time dynamic relationship between the low-frequency component and the high-frequency component;

[0036] S23: Update the hidden state according to the frequency domain state transition matrix, wherein the initial hidden state is a zero vector;

[0037] S24: Reconstruct the time-domain action features based on the updated hidden state, the low-frequency component, and the high-frequency component.

[0038] This embodiment improves the model's ability to model static postures and dynamic details by designing a frequency state-space model that effectively combines low-frequency and high-frequency information. Compared with traditional methods, this invention can generate more natural and coherent action sequences, especially performing well in handling complex multi-stage actions (such as "getting up-walking-lying down"), significantly reducing the phenomenon of overly stiff or lacking details between actions.

[0039] Furthermore, this embodiment also found that in text-driven action generation tasks, the relevant methods perform poorly in terms of semantic consistency between text and action. Most methods achieve conditional constraints by simply concatenating text features and action features. This approach struggles to accurately capture text semantics when faced with complex instructions. For example, for the instruction "step over an object and continue walking," the generated action often omits the crucial action of "stepping," resulting in a mismatch between the generated result and the text description.

[0040] Furthermore, the related technologies lack stability when generating long sequence actions. As the sequence length increases, the generated actions may become vague, repetitive, or lack logical coherence. This is because existing models suffer from error accumulation over long steps, leading to a decrease in the matching degree between the generated results and the input conditions.

[0041] These problems mainly stem from several key limitations in the design of existing diffusion models: the simple concatenation of text features and action features ignores the multimodal alignment requirements between them; in addition, the denoising process of existing diffusion models is not optimized for action generation tasks and fails to make full use of conditional information to finely control the generation process.

[0042] Therefore, to solve these problems and further improve the performance of motion generation, S3 and S4 of this embodiment specifically include:

[0043] S31: Based on the temporal action features and the text features, a first fusion feature is obtained;

[0044] S32: Based on the text features and the action state matrix, a text-guided state matrix is ​​obtained, wherein the action state matrix is ​​used to describe the relationship between actions and multiple states, and the value of the action state matrix is ​​dynamically updated.

[0045] S33: Based on the first fusion feature, the text-guided state matrix, and the updated hidden state, obtain the semantically aligned action features;

[0046] S34: The temporal action features and the semantically aligned action features are fused to obtain the fused features.

[0047] S41: Input the fused features into the core denoiser to obtain denoised features, wherein the core denoiser is a denoising network based on UNet;

[0048] S42: The denoised features are decoded using a decoder to obtain the action sequence, wherein the pre-trained action generation model includes the core denoiser and the decoder.

[0049] This embodiment effectively addresses the shortcomings of existing technologies in fine-grained action generation, text-action semantic consistency, and long sequence generation stability, significantly improving the quality and applicability of generated actions.

[0050] To enable those skilled in the art to better understand the specific training and application processes of the motion generation technology in this embodiment, the following description is provided in conjunction with... Figures 1-4 To provide a detailed explanation.

[0051] The related technologies discovered through research in this embodiment have the following defects, including:

[0052] First, existing technologies lack the ability to accurately capture fine-grained movements when generating complex actions, particularly in the generation of static poses and dynamic details. Current diffusion-based methods fail to effectively utilize the frequency domain information of action sequences, resulting in an inability to simultaneously address the modeling needs of low-frequency information for static poses and the ability of high-frequency information to depict dynamic details during the generation process. Consequently, the generated actions often appear stiff and disjointed, failing to meet the requirements of real-world scenarios.

[0053] Secondly, text-driven action generation techniques suffer from insufficient semantic consistency between the text and the generated actions when dealing with complex semantic descriptions. Related methods typically achieve conditional constraints by simply concatenating text and action features. However, this approach fails to fully capture the key semantic information in the text description, resulting in generated actions that do not accurately reflect the textual instructions. For example, for complex instructions containing multiple stages or detailed actions, the generated results often omit key actions or exhibit semantic deviations.

[0054] Furthermore, the methods exhibit poor stability when generating long sequences of actions. Due to the accumulation of errors during the generation process, long sequences of actions are prone to repetition, ambiguity, or logical incoherence. This problem significantly reduces the usability and user experience of the generated actions in practical applications.

[0055] To address this, this embodiment proposes an action generation technique based on joint optimization of frequency domain and textual semantic information. By introducing frequency domain information, it accurately captures low-frequency and high-frequency features of action sequences to improve the generation effect of static poses and dynamic details; by optimizing the alignment of text and action features, it enhances the semantic consistency between text and generated actions; and by improving the denoising process of the diffusion model, it improves the stability and quality of generating long sequence actions. This embodiment aims to comprehensively solve the above-mentioned shortcomings of existing technologies in action generation, providing high-quality action generation solutions for fields such as animation production, virtual reality, game development, and human-computer interaction.

[0056] This embodiment proposes a human action generation method based on a diffusion model. The aim is to address the issues of fine-grained action generation and semantic consistency between text and action through joint optimization of frequency domain and text semantics, while simultaneously improving the stability and quality of long sequence generation. The framework of this diffusion model-based action generation method is as follows: Figure 1 As shown, the action generation method includes:

[0057] Step 1: Construct an action language dataset, which contains action sequences and corresponding text descriptions.

[0058] Step 2: Data preprocessing and feature extraction.

[0059] (21) The action sequence is standardized to ensure uniformity in action amplitude and time span. The standardized action sequence is then projected into the latent space using a pre-trained variational autoencoder (VAE) to obtain the action features z in the latent space, specifically including:

[0060] (211) Input action sequence x 1:L It is a representation of the motion of human joints in three-dimensional space. These data are represented in the form of a time series, where L represents the frame number, and each frame specifically includes the three-dimensional coordinates of each joint.

[0061] (212) Normalize the action sequence to map the joint position data to a uniform scale so that the model can better handle the differences between the data.

[0062] (213) To reduce data redundancy, the encoder ε of a variational autoencoder (VAE) is used to project the action sequence into the latent space, obtaining the action features z = ε(x) in the latent space. 1:L This process extracts high-level features of the actions, reducing the computational complexity of the model.

[0063] (22) Feature extraction of text description: Extract text features f using the frozen CLIP model. tAs input for subsequent generation conditions, guiding the action generation process, the frozen CLIP model refers to a model whose parameters are no longer updated during training. Specifically:

[0064] (221) For each action sequence, the text description ω is N words in length. 1:N Using a pre-trained CLIP model Extract the embedded text.

[0065] (222) CLIP Model It is a jointly trained text and image model whose text encoder, based on Transformer results, is capable of generating text features. The semantic information of the text was captured. (223) Text features f t It is used as a conditional input to guide the semantic consistency of subsequent actions.

[0066] Step 3: To address the issues of static pose and dynamic detail capture, a frequency domain modeling (FreqMamba) module is designed. This module utilizes a frequency state space model to process motion features. The framework of the frequency state space model is as follows: Figure 2 As shown, it specifically includes:

[0067] (311) The action feature z in the latent space is decomposed into low-frequency component f using Discrete Wavelet Transform (DWT). low and high-frequency components f high Low-frequency components are used for modeling static actions, primarily capturing global static movements such as sitting and standing postures, while high-frequency components are used for generating details of dynamic actions, mainly capturing dynamic details such as rapid steps and gesture changes. This frequency domain decomposition can effectively improve the hierarchy and adaptability of feature representation.

[0068] (312) To enhance the temporal correlation of the components, the convolution operation Conv(·) is used to extract the temporal dynamic relationship between low-frequency features (i.e., low-frequency components) and high-frequency features (i.e., high-frequency components): (f low f high = Conv(DWT(z)), where the kernel size and stride are dynamically adjusted based on the frame rate of the motion data. The convolution operation, by processing the temporal dynamic relationship between low-frequency and high-frequency features, provides a basis for adjusting the values ​​of α and β. For example, if the convolution operation reveals drastic dynamic changes in motion within a certain time period, the high-frequency feature f obtained from the convolution... high The corresponding weight β might be increased, thus making the generated action sequence focus more on the details of the action during this period. Conversely, if the action is relatively stable during a certain period, the low-frequency feature f... low It may occupy a larger proportion, ensuring the stability of global static action features.

[0069] (321) Based on the temporal dynamic relationship of the low-frequency and high-frequency features extracted through convolution operation in step (312) above, adjust the learnable scaling parameters α and β to dynamically adjust the proportion of low-frequency and high-frequency features in the state transition matrix, and construct the frequency domain state transition matrix: A n =A+αf low +βf high , where A is the initial value of the state transition matrix, specifically set to a random value following a standard Gaussian distribution (this initialization method can avoid the symmetry violation problem during training and provide a certain initial diversity for the model). This dynamic adjustment process ensures that the generated action sequence can balance the influence of static and dynamic features.

[0070] The parameters α and β are weight coefficients learned during model training. Specifically, in each training iteration, the model updates the values ​​of α and β through backpropagation based on the error between the current prediction and the actual result. For example, suppose in a generation task, the static part of the action (such as a person's posture) is more important, while dynamic details (such as changes in the person's movement speed) do not require much attention. The model may adjust α to a larger value while keeping β small, thus allowing low-frequency components to occupy a larger proportion in the state transition matrix. Conversely, in another task, if dynamic details (such as rapid changes in movement) are more important, β will be adjusted to a larger value to enhance the role of high-frequency components.

[0071] (322) Generation and reconstruction of hidden states: frequency domain state transition matrix A n Used to update the hidden state h t :h′ t =A n h t +Bz, where B represents the latent feature projection matrix, responsible for projecting the action features z in the latent space onto the hidden state h. t A compatible space is provided to ensure they can be effectively integrated. The hidden state h... t The hidden state represents the action state at the current moment. As the action sequence is generated, the hidden state will be continuously updated. Therefore, at the beginning stage of sequence generation (i.e., the first time step), the initial hidden state h0 is set as a zero vector.

[0072] (323) Combine the low-frequency component f obtained in step (311) using Discrete Wavelet Transform (DWT) low and high-frequency components f high The frequency domain enhancement information is transformed into time domain features through inverse discrete wavelet transform (IDWT), that is, the low-frequency and high-frequency components are reconstructed into time domain action features. The action-state matrix C here is usually initialized with random values ​​or a zero matrix and dynamically optimized during training.

[0073] Step 4: To ensure semantic consistency between the generated action and the input text description, a Text Semantic Alignment (TextMamba) module is designed. This module utilizes the Text State Space Model (TextSSM) to process text features and temporal action features. The framework of the Text State Space Model is as follows: Figure 2 As shown, it specifically includes:

[0074] (411) In order to achieve consistency between action and text semantics, temporal action features are analyzed. and text features f t Generate fused features through residual connections

[0075] (412) Construct the text-guided state matrix C in the text state space. s By using text features f t By fusing with the action state matrix C, semantic alignment of text and actions is achieved: C s =f t +C. Text-guided state matrices can capture the relationship between text semantics and action features, thereby achieving more accurate semantic alignment.

[0076] (413) Text-guided state matrix C s Action features used to generate semantic alignment Specifically, the hidden state h is updated using the state matrix. t and combined with fusion features Perform the calculation: Here, D represents a learnable linear transformation matrix used to adjust the weight ratio of text and action features.

[0077] Step 5: Optimize the denoising process of the diffusion model.

[0078] (511) In the process of action generation, in order to further improve the quality and stability of the generated details, the temporal action features provided by the frequency domain modeling (FreqMamba) module are used. Semantic alignment features provided by the TextMamba module (i.e., semantically aligned action features) are fused to form the final denoised feature z′. t : This fusion mechanism ensures the organic combination of frequency domain information and textual semantic information, enabling the generated actions to achieve high-quality performance in terms of dynamic details and semantic consistency.

[0079] (512) via decoder The denoised latent features z′ t Decoded into action sequence The input to this process is the denoising feature z′. t The output is a three-dimensional joint motion sequence.

[0080] (513) In the optimization of the denoising process, a diffusion model denoising strategy is adopted, and a core denoiser based on Mamba is designed. The denoiser uses a UNet-based denoising network to capture fine-grained action features and text consistency information.

[0081] The key idea of ​​the diffusion model is to progressively add noise during the generation process, thereby generating intermediate noise distributions. In this process, the input motion features undergo multiple iterations of noise addition and removal, with the noise intensity gradually decreasing at each step, ultimately restoring high-quality motion features. Specifically, in the forward process of diffusion, noise is progressively added to the data, forming multiple intermediate noise distributions. The reverse process in the diffusion model refers to the model progressively removing noise and restoring clear features by predicting the mean and variance of the features at each generation step. The specific formula is: p θ (z t-1 |z t )=N(z t-1 μ θ (z t ,t),Σ θ (t)), where μ θ and Σ θ These represent the mean and variance predicted by the neural network, respectively. They determine the specific details of the denoising process and guide the gradual recovery of features from noise to clarity.

[0082] During the training phase, the mean squared error (MSE) between the real noise and the predicted noise is minimized. After training, the trained model can be used to generate high-quality action sequences consistent with the input text description.

[0083] During the reasoning phase, high-quality action sequences are generated through a diffusion process in the latent space, specifically including:

[0084] The model takes a text description as input, which guides the generation of the target action sequence.

[0085] Text descriptions are processed by the CLIP model to extract text features f t As input conditions for generation;

[0086] At the same time, the model also needs to input the motion features z of the latent space, which are randomly sampled from a Gaussian distribution.t .

[0087] The denoising feature z′ is processed by the decoder. t The system decodes the input text to generate a high-quality sequence of actions that matches the description in the input text, and outputs it to the target application scenario, such as animation production or virtual reality system.

[0088] The action generation method described in this embodiment addresses the shortcomings of existing action generation technologies by using frequency domain modeling and text semantic alignment optimization. This enables the generation of high-quality, fine-grained, and semantically consistent human action sequences, significantly improving the quality, accuracy, and diversity of action generation, and resolving the semantic inconsistency between text and actions. Compared to existing technologies, this embodiment has the following significant advantages:

[0089] 1. Significantly improved fine-grained motion generation capability

[0090] This embodiment improves the model's ability to model static postures and dynamic details by designing a frequency state-space model that effectively combines low-frequency and high-frequency information. Compared with traditional methods, this embodiment can generate more natural and coherent action sequences, especially performing well in handling complex multi-stage actions (such as "getting up-walking-lying down"), significantly reducing the phenomenon of overly stiff or lacking details between actions.

[0091] 2. Enhanced text-action semantic consistency

[0092] This embodiment utilizes a text state space model to achieve precise alignment of text features and action features in the latent space. Compared to the simple concatenation of text and action features in existing technologies, this embodiment can accurately capture the semantic information in the text description and fully reflect it in the generated action. For example, for the complex instruction "step over an object and continue walking," this embodiment can generate a complete sequence containing the key action "step," ensuring semantic consistency between the text description and the generated result.

[0093] 3. Improved stability of long sequence action generation

[0094] This embodiment significantly reduces the error accumulation problem in long sequence generation by optimizing the denoising process of the diffusion model. Experiments show that this embodiment is less prone to ambiguity, repetition, or lack of logical coherence when generating action sequences with long time spans, ensuring the overall coherence and accuracy of the generated actions.

[0095] 4. Overall improvement in production quality

[0096] Compared to existing methods, this embodiment demonstrates superior generation quality. Experimental results based on Fréchet InceptionDistance (FID) show that the actions generated by this embodiment more closely resemble the high-level feature distribution of real actions, with a significantly reduced FID value. Furthermore, this embodiment also excels in diversity and multimodality metrics, generating richer and more varied action sequences.

[0097] 5. Optimization of training efficiency and inference performance

[0098] By performing a diffusion process in the latent space and employing the efficient Mamba module as the core component of the denoiser, this embodiment significantly reduces computational complexity while maintaining long-range modeling capabilities. Experiments show that this embodiment improves training and inference speeds compared to existing methods under the same hardware configuration, making it suitable for large-scale data processing and real-time applications.

[0099] 6. Wide applicability to various application scenarios

[0100] This embodiment performs exceptionally well in scenarios such as animation production, virtual reality (VR) / augmented reality (AR), game development, and human-computer interaction, generating high-quality motion sequences that meet practical needs. For example, in virtual reality, the motion generated by this embodiment enhances immersion and interactive experience; in animation production, the generated motion exhibits higher naturalness and detail.

[0101] This embodiment, through innovative frequency domain modeling and text semantic alignment technology, not only solves the key problems of existing action generation technology, but also significantly improves the quality, accuracy and efficiency of action generation, providing technical support for a variety of practical application scenarios.

[0102] like Figure 4 As shown, taking the text description "someone is playing the violin" as an example, the method proposed in this embodiment can generate a semantically consistent three-dimensional human motion sequence based on the input text prompt, through the following steps:

[0103] (1): The text description "someone is playing the violin" is extracted as text feature f by the CLIP model. t .

[0104] (2): The action features z in the latent space of the action sequence are extracted by variational autoencoder and preliminarily represent the basic action information.

[0105] (3): In the frequency domain modeling module, the low-frequency feature f low It describes a wide range of dynamic information during violin playing, such as hand movements and high-frequency characteristics f. high This describes subtle differences in the torso.

[0106] (4): In the text semantic alignment module, text feature f t Embedded into action features, the state matrix C is dynamically adjusted. s This ensures that the generated actions are consistent with the semantics of the text.

[0107] (5): In the decoding stage, through the denoising process of the diffusion model, a three-dimensional human motion sequence with rich details and natural movements is finally generated.

[0108] Through the detailed steps described above, this invention provides an efficient and accurate motion generation scheme, which significantly improves the detail quality and application feasibility of motion generation.

[0109] In summary, the human action generation method based on a diffusion model provided in this embodiment is an efficient and accurate action generation scheme that significantly improves the detail quality and application feasibility of action generation. By combining frequency domain modeling and text semantic alignment techniques, it significantly enhances the fine-grained representation and semantic consistency of generated actions, overcoming the shortcomings of existing technologies in complex action generation and long sequence stability. This technology can generate more natural, coherent action sequences that conform to semantic descriptions, while improving generation efficiency and quality. It is widely applicable to fields such as animation production, virtual reality, game development, and human-computer interaction, and has significant technical value and commercial prospects.

[0110] Example 2

[0111] This embodiment provides a computer device, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 5 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores data from the action generation method based on joint optimization of frequency domain and textual semantic information in Embodiment 1. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the action generation method based on joint optimization of frequency domain and textual semantic information as described in Embodiment 1.

[0112] Those skilled in the art will understand that Figure 5The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0113] Example 3

[0114] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the action generation method based on joint optimization of frequency domain and text semantic information as described in Embodiment 1 above.

[0115] Example 4

[0116] This embodiment provides a computer program product, including a computer program that, when executed by a processor, implements the action generation method based on joint optimization of frequency domain and text semantic information as described in Embodiment 1 above.

[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0119] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0120] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An action generation method based on joint optimization of frequency domain and textual semantic information, characterized in that, The action generation method based on joint optimization of frequency domain and text semantic information includes: The text features to be processed and the corresponding initial latent spatial motion features are obtained, wherein the text features to be processed are variables obtained by feature extraction of the text description to be processed, and the initial latent spatial motion features are variables randomly sampled from a Gaussian distribution; Using the initial potential spatial motion features as input, and utilizing a pre-trained frequency-space state model, the temporal action features are output, specifically including: The initial potential spatial motion features are decomposed into low-frequency components and high-frequency components, wherein the low-frequency components are used to capture the global static motion features of the motion, and the high-frequency components are used to capture the dynamic motion detail features of the motion. A frequency domain state transition matrix is ​​constructed based on the low-frequency component, the high-frequency component, and the scaling parameter, wherein the scaling parameter is a variable determined by the time dynamic relationship between the low-frequency component and the high-frequency component; The hidden state is updated according to the frequency domain state transition matrix, wherein the initial hidden state is a zero vector; The temporal action features are reconstructed based on the updated hidden state, the low-frequency component, and the high-frequency component. The temporal action features and the text features to be processed are concatenated to obtain fused features, specifically including: Based on the temporal action features and the text features, a residual connection is specifically performed on the temporal action features and the text features to obtain the first fused feature; Based on the text features and the action state matrix, the text-guided state matrix is ​​obtained; Based on the first fusion feature, the text-guided state matrix, and the updated hidden state, semantically aligned action features are obtained; The temporal action features and the semantically aligned action features are fused together to obtain the fused features; Using the fused features as input, a pre-trained action generation model is used to output an action sequence, specifically including: The fused features are input into the core denoiser to obtain denoised features, wherein the core denoiser is a denoising network based on UNet; The denoised features are decoded using a decoder to obtain the action sequence, wherein the pre-trained action generation model includes the core denoiser and the decoder; The expression for the frequency domain state transition matrix is: ； in, This is the frequency domain state transition matrix; The initial value of the state transition matrix is ​​set to a random value that follows a standard Gaussian distribution; and This is a proportional parameter; Low-frequency components; These are high-frequency components; Convolution operations are used to extract the temporal dynamic relationship between low-frequency and high-frequency components, providing... and The values ​​provide the basis for adjustment, where the size of the convolution kernel and the stride are dynamically adjusted according to the frame rate of the action data.

2. The action generation method based on joint optimization of frequency domain and text semantic information according to claim 1, characterized in that, The expression for the time-domain action feature is: ； in, For time-domain action features; This is the action state matrix; For the updated hidden state; It is the inverse discrete wavelet transform; Low-frequency components; These are high-frequency components.

3. The action generation method based on joint optimization of frequency domain and text semantic information according to claim 1, characterized in that, The expression for the semantic alignment action feature is: ； in, Action features for semantic alignment; A state matrix guided by text; For the updated hidden state; It is a learnable linear transformation matrix; This is the first fusion feature.

4. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the action generation method based on joint optimization of frequency domain and textual semantic information as described in any one of claims 1-3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the action generation method based on joint optimization of frequency domain and textual semantic information as described in any one of claims 1-3.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the action generation method based on joint optimization of frequency domain and textual semantic information as described in any one of claims 1-3.

Images