Action generation method and device based on frequency domain and text semantic information joint optimization

Abstract

The invention provides an action generation method and device based on frequency domain and text semantic information joint optimization, and relates to the technical field of action generation, and the method comprises the steps: decomposing an initial potential space motion feature into a low-frequency component and a high-frequency component; obtaining the time domain action characteristics according to the low-frequency component and the high-frequency component; splicing the time domain action features and the to-be-processed text features to obtain fusion features; and taking the fusion features as input, and outputting an action sequence by using a pre-trained action generation model. By designing the frequency state space model, low-frequency information and high-frequency information are effectively combined, and the modeling capacity of the model for static postures and dynamic detailed actions is improved. Compared with a traditional method, the method has the advantages that a more natural and coherent action sequence can be generated, particularly, the method is excellent in performance when complex multi-stage actions are processed, and the phenomenon that the actions are excessively stiff or lack of details is remarkably reduced.

Description

Technical Field

[0001] The present application relates to the technical field of action generation, and in particular to an action generation method and device based on joint optimization of frequency domain and text semantic information. Background Art

[0002] In the field of human motion generation, generating natural and physically logical motion sequences based on conditional inputs (such as text descriptions) is a widely-regarded technology. This technology has wide applications in animation production, virtual reality (VR) / augmented reality (AR), game development, and human-computer interaction. Existing mainstream methods mainly include autoencoder models, generative adversarial networks, and autoregressive models. These methods provide varying degrees of support for motion generation, but still have many limitations in generating complex, coherent, and detailed motion sequences. In recent years, diffusion models have been gradually introduced into the field of motion generation due to their excellent performance in image and video generation. Diffusion models gradually restore the target data distribution by denoising, and can generate more natural motions. However, this technology still has significant problems in practical applications.

[0003] Existing action generation methods based on diffusion models are not good enough in generating fine-grained actions, especially in capturing the transition details between static postures and dynamic actions. For example, for complex instructions such as "stand up-walk-lie down", existing models have difficulty smoothly capturing the detailed changes between each stage of the action, and the generated results appear stiff or unnatural, which cannot well simulate the natural transition and detailed performance of humans when performing these actions. Summary of the invention

[0004] The purpose of this application is to provide a method and device for action generation based on joint optimization of frequency domain and text semantic information, which can capture the detailed changes between actions in each stage and make the action generation results more natural.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides an action generation method based on joint optimization of frequency domain and text semantic information, comprising:

[0007] Obtaining a text feature to be processed and a corresponding initial latent space motion feature, wherein the text feature to be processed is a variable obtained by extracting features from the text description to be processed, and the initial latent space motion feature is a variable obtained by random sampling from a Gaussian distribution;

[0008] Taking the initial latent space motion features as input, outputting time domain motion features using a pre-trained frequency-space state model;

[0009] Concatenating the time-domain action feature and the text feature to be processed to obtain a fusion feature;

[0010] Taking the fused features as input, and using a pre-trained action generation model to output an action sequence;

[0011] The initial latent space motion features are used as input, and the pre-trained frequency-space state model is used to output the time domain motion features, specifically including:

[0012] Decomposing the initial latent space motion features into low-frequency components and high-frequency components, wherein the low-frequency components are used to capture global static motion features of the action, and the high-frequency components are used to capture dynamic motion detail features of the action;

[0013] The time domain motion feature is obtained according to the low frequency component and the high frequency component.

[0014] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the action generation method based on joint optimization of frequency domain and text semantic information as described in the first aspect above.

[0015] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the action generation method based on joint optimization of frequency domain and text semantic information as described in the first aspect above.

[0016] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the action generation method based on joint optimization of frequency domain and text semantic information as described in the first aspect above.

[0017] According to the specific embodiments provided in this application, this application has the following technical effects:

[0018] The present application provides an action generation method and device based on the joint optimization of frequency domain and text semantic information, the method comprising: decomposing the initial latent space motion features into low-frequency components and high-frequency components; obtaining the time-domain action features according to the low-frequency components and the high-frequency components; splicing the time-domain action features and the text features to be processed to obtain fusion features; using the fusion features as input, and outputting an action sequence using a pre-trained action generation model. By designing a frequency state space model (FreqSSM), the low-frequency information and high-frequency information are effectively combined, which improves the modeling ability of the model for static postures and dynamic detailed actions. Compared with traditional methods, the present application can generate more natural and coherent action sequences, especially when processing complex multi-stage actions (such as "getting up-walking-lying down"), and significantly reduces the phenomenon of excessive stiffness or lack of details between actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 An architecture diagram of an action generation method based on joint optimization of frequency domain and text semantic information provided in Example 1 of the present application;

[0021] Figure 2 This is a schematic diagram of the frequency state space model in Example 1 of the present application;

[0022] Figure 3 This is a structural diagram of the text state space model in Example 1 of the present application;

[0023] Figure 4 This is a schematic diagram of the action generation result in Example 1 of the present application;

[0024] Figure 5 A schematic diagram of the structure of a computer device provided in Example 2 of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] Example 1

[0028] Research has found that existing diffusion models have difficulty smoothly capturing the detailed changes between actions at different stages, and the generated results appear stiff or unnatural. This is mainly because existing methods fail to fully utilize the frequency characteristics of action sequences. Specifically, low-frequency information is crucial for static posture modeling, while high-frequency information determines the detailed changes in actions. However, existing diffusion models lack the ability to organically combine these two types of information.

[0029] In this regard, this embodiment provides an action generation method based on joint optimization of frequency domain and text semantic information, including:

[0030] S1: Obtaining a text feature to be processed and a corresponding initial latent space motion feature, wherein the text feature to be processed is a variable obtained by extracting features from the text description to be processed, and the initial latent space motion feature is a variable obtained by randomly sampling from a Gaussian distribution;

[0031] S2: taking the initial latent space motion features as input and outputting time domain motion features using a pre-trained frequency-space state model;

[0032] S3: splicing the time domain action feature and the text feature to be processed to obtain a fusion feature;

[0033] S4: using the fusion feature as input and using a pre-trained action generation model to output an action sequence, wherein the action generation model is a diffusion model;

[0034] S21: decomposing the initial latent space motion features into low-frequency components and high-frequency components, wherein the low-frequency components are used to capture global static motion features of the action, and the high-frequency components are used to capture dynamic motion detail features of the action;

[0035] S22: constructing a frequency domain state transfer matrix according to the low frequency component, the high frequency component and a proportional parameter, wherein the proportional parameter is a variable determined by a time dynamic relationship between the low frequency component and the high frequency component;

[0036] S23: updating the hidden state according to the frequency domain state transfer matrix, wherein the initial hidden state is a zero vector;

[0037] S24: Reconstruct the time domain action feature according to the updated hidden state, the low frequency component and the high frequency component.

[0038] This embodiment improves the modeling ability of static postures and dynamic detailed movements by designing a frequency state space model and effectively combining low-frequency information and high-frequency information. Compared with traditional methods, the present invention can generate more natural and coherent action sequences, especially when processing complex multi-stage actions (such as "standing up-walking-lying down"), and significantly reduces the phenomenon of excessive stiffness or lack of details between actions.

[0039] In addition, this embodiment also found that in text-driven action generation tasks, related methods performed poorly in terms of semantic consistency between text and action. Most methods implement conditional constraints by simply splicing text features and action features. This approach is difficult to accurately capture text semantics when faced with complex instructions. For example, for the instruction "step over an object and continue walking", the generated action often omits the key action of "stepping", resulting in the generation result not being consistent with the text description.

[0040] In addition, the relevant technologies are not stable enough when generating long sequences of actions. As the length of the sequence increases, the generated actions may appear ambiguous, repetitive, or lack logic. This is because the existing models have error accumulation problems in long time steps, resulting in a decrease in the matching degree between the generated results and the input conditions.

[0041] These problems mainly arise from several key limitations in the design of existing diffusion models: the simple concatenation of text features and action features ignores the need for multimodal alignment between the two; in addition, the denoising process of the existing diffusion model is not optimized for the action generation task, and fails to fully utilize conditional information to finely control the generation process.

[0042] Therefore, to solve these problems and further improve the performance of action generation, S3 and S4 of this embodiment specifically include:

[0043] S31: Obtaining a first fusion feature according to the time domain action feature and the text feature;

[0044] S32: Obtain a text-guided state matrix according to the text features and the action state matrix, wherein the action state matrix is ​​used to describe the relationship between an action and a plurality of states, and the value of the action state matrix is ​​dynamically updated.

[0045] S33: Obtaining semantically aligned action features according to the first fusion feature, the text-guided state matrix and the updated hidden state;

[0046] S34: Fusing the time-domain motion feature and the semantically aligned motion feature to obtain the fused feature.

[0047] S41: Inputting the fused features into a core denoiser to obtain denoised features, wherein the core denoiser is a denoising network based on UNet;

[0048] S42: Decoding the denoising features using a decoder to obtain the action sequence, wherein the pre-trained action generation model includes the core denoiser and the decoder.

[0049] This embodiment effectively solves the shortcomings of the prior art in terms of fine-grained action generation, text-action semantic consistency, and long sequence generation stability, and significantly improves the quality of generated actions and application applicability.

[0050] In order to make those skilled in the art more clear about the specific training process and application process of the action generation technology of this embodiment, Figure 1-Figure 4 Provide specific explanation.

[0051] The related technologies found in this embodiment have the following defects, including:

[0052] First, related technologies lack the ability to accurately capture fine-grained actions when generating complex actions, especially in generating static postures and dynamic detailed actions. Current diffusion model-based methods fail to effectively utilize the frequency domain information of action sequences, resulting in the inability to balance the modeling requirements of low-frequency information for static postures and the ability of high-frequency information to depict dynamic details during the generation process. As a result, the generated actions often appear stiff and incoherent, making it difficult to meet the needs of real-world scenarios.

[0053] Secondly, when dealing with complex semantic descriptions, text-driven action generation technology has the problem of insufficient semantic consistency between text and generated actions. Related methods usually implement conditional constraints by simply concatenating text features and action features. However, this method cannot fully capture the key semantic information in the text description, resulting in the generated actions failing to accurately reflect the text instructions. For example, for complex instructions containing multiple stages or detailed actions, the generated results often miss key actions or have semantic deviations.

[0054] In addition, related methods have poor stability when generating long sequences of actions. Due to the accumulation of errors in the generation process, long sequences of actions are prone to repetition, ambiguity, or logical incoherence. This problem will significantly reduce the usability and user experience of the generated actions in practical applications.

[0055] In this regard, this embodiment proposes an action generation technology based on the joint optimization of frequency domain and text semantic information. By introducing frequency domain information, the low-frequency and high-frequency features of the action sequence are accurately captured to improve the generation effect of static postures and dynamic details; by optimizing the alignment of text and action features, the semantic consistency between text and generated actions is enhanced; at the same time, by improving the denoising process of the diffusion model, the stability and quality of generating long sequences of actions are improved. This embodiment aims to comprehensively solve the above-mentioned deficiencies of the existing technology in action generation, and provide high-quality action generation solutions for animation production, virtual reality, game development, human-computer interaction and other fields.

[0056] This embodiment proposes a method for human action generation based on a diffusion model, the purpose of which is to solve the problem of fine-grained action generation and consistency between text and action semantics through joint optimization of frequency domain and text semantics, while improving the stability and quality of long sequence generation. The framework of the action generation method based on the diffusion model is as follows Figure 1 As shown, the action generation method includes:

[0057] Step 1: Construct an action language dataset, which contains action sequences and corresponding text descriptions.

[0058] Step 2: Data preprocessing and feature extraction.

[0059] (21) The action sequence is standardized to ensure the uniformity of the action amplitude and time span, and the standardized action sequence is projected into the latent space through a pre-trained variational autoencoder (VAE) to obtain the action feature z of the latent space, which specifically includes:

[0060] (211) Input action sequence x 1:L It is the movement representation of human joints in three-dimensional space. These data are expressed in the form of time series. L represents the number of frames, and each frame specifically includes the three-dimensional coordinates of each joint.

[0061] (212) The action sequence is normalized to map the joint position data to a uniform scale so that the model can better handle the differences between the data.

[0062] (213) In order to reduce the redundant characteristics of the data, the encoder ε of the variational autoencoder (VAE) is used to project the action sequence into the latent space, and the action feature z = ε(x 1:L ). This process extracts high-level features of the action and reduces the computational complexity of the model.

[0063] (22) Feature extraction of text description: Use the frozen CLIP model to extract text features f tAs the subsequent generation condition input to guide the action generation process, the frozen CLIP model refers to a model whose parameters are no longer updated during the training process, specifically:

[0064] (221) For each action sequence, the length of the text description ω is N words 1:N , using the pre-trained CLIP model Extract text embeddings.

[0065] (222) CLIP model It is a jointly trained text and image model whose text encoder is based on Transformer results and can generate text features Captures the semantic information of the text. (223) Text feature f t is used as conditional input to guide the semantic consistency of subsequent action generation.

[0066] Step 3: To solve the problem of capturing static postures and dynamic details, a frequency domain modeling (FreqMamba) module is designed. This module uses a frequency state space model to process motion features. The framework of the frequency state space model is as follows: Figure 2 As shown, specifically including:

[0067] (311) Discrete wavelet transform (DWT) is used to decompose the action feature z in the latent space into low-frequency components f low and high frequency component f high The low-frequency components are used for modeling static actions, mainly capturing the global static actions of the action, such as sitting and standing, etc., while the high-frequency components are used for detail generation of dynamic actions, mainly capturing the dynamic action details in the action, such as fast steps, gesture changes, etc. This frequency domain decomposition can effectively improve the hierarchy and adaptability of feature expression.

[0068] (312) In order to enhance the temporal correlation of components, the convolution operation Conv(·) is used to extract the temporal dynamic relationship between low-frequency features (i.e., low-frequency components) and high-frequency features (i.e., high-frequency components): (f low , f high )=Conv(DWT(z)), where the size and step size of the convolution kernel are dynamically adjusted according to the frame rate of the action data. The convolution operation provides a basis for adjusting the values ​​of α and β by processing the temporal dynamic relationship between low-frequency and high-frequency features. For example, if the convolution operation reveals that the dynamic changes of the action in a certain period of time are more drastic, the high-frequency feature f obtained by convolution high The corresponding weight β may be increased, so that the generated action sequence pays more attention to the details of the action during this period of time. On the contrary, if the action is relatively stable in a certain period of time, the low-frequency feature f low It may occupy a larger proportion to ensure the stability of global static motion features.

[0069] (321) According to the temporal dynamic relationship between the low-frequency and high-frequency features extracted by the convolution operation in the above step (312), the learnable ratio parameters α and β are adjusted to dynamically adjust the ratio of the low-frequency features and the high-frequency features in the state transfer matrix, and construct the frequency domain state transfer matrix: n =A+αf low +βf high , where A is the initial value of the state transfer matrix, which is specifically set to a random value following a standard Gaussian distribution (this initialization method can avoid the problem of symmetry destruction during training and provide a certain initial diversity for the model). This dynamic adjustment process ensures that the generated action sequence can balance the influence of static and dynamic features.

[0070] The parameters α and β are weight coefficients learned during the model training process. Specifically, in each training iteration, the model will update the values ​​of α and β through the back-propagation mechanism according to the error between the current predicted result and the actual result. For example, assuming that in a generation task, the static part of the action (such as the posture of the character) is more important, while the dynamic details (such as the change in the speed of the character's movement) do not require too much attention, the model may adjust α to a larger value, while β remains small, so that the low-frequency component occupies a larger proportion in the state transfer matrix. Conversely, in another task, if the dynamic details (such as rapid changes in movement) are more important, β will be adjusted to a larger value to enhance the role of the high-frequency component.

[0071] (322) Generation and reconstruction of hidden states: frequency domain state transfer matrix A n Used to update the hidden state h t :h′ t =A n h t +Bz, where B represents the potential feature projection matrix, which is responsible for projecting the action feature z in the latent space to the hidden state h t compatible space, ensuring that they can be effectively integrated. The hidden state h t Represents the action state at the current moment. As the action sequence is generated, the hidden state will be continuously updated. Therefore, at the beginning of the sequence generation (i.e., the first time step), the initial hidden state h0 is set to a zero vector.

[0072] (323) Combined with the low-frequency component f obtained by discrete wavelet transform (DWT) in step (311) low and high frequency component f high , the frequency domain enhancement information is transformed into time domain features through inverse discrete wavelet transform (IDWT), that is, the low-frequency components and high-frequency components are reconstructed into time domain motion features The action-state matrix C here is usually initialized to random values ​​or a zero matrix and dynamically optimized during training.

[0073] Step 4: To ensure the semantic consistency between the generated action and the input text description, a text semantic alignment (TextMamba) module is designed. This module uses the text state space model (TextSSM) to process text features and temporal action features. The framework of the text state space model is as follows: Figure 2 As shown, specifically including:

[0074] (411) In order to achieve consistency between action and text semantics, the temporal action features and text features f t , generate fusion features through residual connection

[0075] (412) In the text state space, construct the text-guided state matrix C s , by taking the text feature f t Fuse with the action state matrix C to achieve semantic alignment between text and action: C s =f t +C. The text-guided state matrix can capture the relationship between text semantics and action features, thus achieving more accurate semantic alignment.

[0076] (413) Text-guided state matrix C s For generating semantically aligned action features Specifically, the hidden state h is updated using the state matrix t , and combined with fusion features Perform the calculation: Where D represents a learnable linear transformation matrix, which is used to adjust the weight ratio of text and action features.

[0077] Step 5: Optimization of diffusion model denoising process.

[0078] (511) In the process of motion generation, in order to further improve the quality and stability of generated details, the time domain motion features provided by the frequency domain modeling (FreqMamba) module are used and semantic alignment features provided by the TextMamba module (i.e., semantically aligned action features) are fused to form the final denoising feature z′ t : This fusion mechanism ensures the organic combination of frequency domain information and text semantic information, so that the generated actions can achieve high-quality performance in both dynamic details and semantic consistency.

[0079] (512) Through the decoder The denoised latent feature z′ t Decoding into action sequences The input of this process is the denoised feature z′ t , the output is a three-dimensional joint motion sequence.

[0080] (513) In the optimization of the denoising process, a denoising strategy of a diffusion model is adopted, and a core denoiser based on Mamba is designed. The denoiser uses a denoising network based on UNet to capture fine-grained action features and text consistency information.

[0081] The key idea of ​​the diffusion model is to gradually add noise during the generation process to generate an intermediate noise distribution. During this process, the input motion features will undergo multiple iterations of noise addition and denoising, and the noise intensity will gradually decrease at each step, eventually restoring high-quality motion features. Specifically, in the forward process of the diffusion process, noise is gradually added to the data to form multiple intermediate noise distributions. The reverse process in the diffusion model means that in each step of the generation, the model gradually removes noise and restores clear features by predicting the mean and variance of the features. The specific formula is: p θ (z t-1 |z t )=N(z t-1 ;μ θ (z t ,t),Σ θ (t)), where μ θ and Σ θ They represent the mean and variance predicted by the neural network, respectively, which determine the specific details of the denoising process and guide the gradual recovery of features from noise to clarity.

[0082] During the training phase, the mean square error (MSE) between the real noise and the predicted noise is minimized. After training, the trained model can be used to generate high-quality action sequences that are consistent with the input text description.

[0083] In the inference phase, high-quality action sequences are generated through a diffusion process in the latent space, including:

[0084] The input of the model is a text description, which is used to guide the generation of target action sequences;

[0085] The text description is extracted through the CLIP model. t As a conditional input for generation;

[0086] At the same time, the model also needs to input the motion features z of the latent space randomly sampled from the Gaussian distributiont .

[0087] The denoising feature z′ is processed by the decoder t Decode and generate high-quality action sequences that match the input text description, and output them to target application scenarios, such as animation production or virtual reality systems.

[0088] The above-mentioned action generation method of this embodiment, in view of the shortcomings of the existing action generation technology, can generate high-quality, fine-grained and semantically consistent human action sequences through frequency domain modeling and text semantic alignment optimization, significantly improving the quality, accuracy and diversity of action generation, and solving the problem of semantic inconsistency between text and action. Compared with the existing technology, this embodiment has the following significant advantages:

[0089] 1. Significantly improved fine-grained action generation capabilities

[0090] This embodiment improves the modeling ability of static postures and dynamic detailed movements by designing a frequency state space model and effectively combining low-frequency information and high-frequency information. Compared with traditional methods, this embodiment can generate more natural and coherent action sequences, especially when processing complex multi-stage actions (such as "standing up-walking-lying down"), and significantly reduces the phenomenon of excessive stiffness or lack of details between actions.

[0091] 2. Enhanced text-action semantic consistency

[0092] This embodiment uses a text state space model to achieve precise alignment of text features and action features in the latent space. Compared with the method of simply splicing text and action features in the prior art, this embodiment can accurately capture the semantic information in the text description and fully reflect it in the generated action. For example, for the complex instruction "step over an object and continue walking", this embodiment can generate a complete sequence containing the key action "step", ensuring the semantic consistency of the text description and the generated result.

[0093] 3. Improved stability of long sequence action generation

[0094] This embodiment significantly reduces the error accumulation problem in long sequence generation by optimizing the denoising process of the diffusion model. Experiments show that this embodiment is less likely to have ambiguity, repetition or lack of logic when generating action sequences with a long span, ensuring the overall coherence and accuracy of the generated actions.

[0095] 4. Comprehensive improvement of generation quality

[0096] Compared with existing methods, this embodiment performs well in terms of generation quality. Experimental results based on Fréchet Inception Distance (FID) show that the actions generated by this embodiment are closer to the high-level feature distribution of real actions, and the FID value is significantly reduced. At the same time, this embodiment also performs well in diversity and multimodality indicators, and can generate richer and more diverse action sequences.

[0097] 5. Training efficiency and inference performance optimization

[0098] By performing a diffusion process in the latent space and using the efficient Mamba module as the core component of the denoiser, this embodiment significantly reduces the computational complexity while maintaining the long-distance modelling capability. Experiments show that the training and inference speeds of this embodiment are both improved compared to existing methods under the same hardware configuration, making it suitable for large-scale data processing and real-time applications.

[0099] 6. Wide applicability of application scenarios

[0100] This embodiment performs well in scenes such as animation production, virtual reality (VR) / augmented reality (AR), game development, and human-computer interaction, and can generate high-quality action sequences that meet actual needs. For example, in virtual reality, the actions generated by this embodiment can enhance the sense of immersion and interactive experience; in animation production, the generated actions have higher naturalness and detailed expression.

[0101] This embodiment uses innovative frequency domain modeling and text semantic alignment technology to not only solve the key problems of existing action generation technology, but also significantly improves the quality, accuracy and efficiency of action generation, providing technical support for a variety of practical application scenarios.

[0102] like Figure 4 As shown, taking the text description of "someone is playing the violin" as an example, the method proposed in this embodiment can generate a semantically consistent three-dimensional human action sequence according to the input text prompt, through the following steps:

[0103] (1): The text description “someone is playing the violin” is extracted by the CLIP model as text feature f t .

[0104] (2): The action feature z in the latent space of the action sequence is extracted through a variational autoencoder, which preliminarily represents the basic action information.

[0105] (3): In the frequency domain modeling module, the low-frequency feature f low Describes the large-scale dynamic information during the violin playing process, such as hand swings, high-frequency features f high describes slight differences in the torso.

[0106] (4): In the text semantic alignment module, the text feature f t is embedded into the action features and dynamically adjusts the state matrix C s , thus ensuring that the generated actions are consistent with the text semantics.

[0107] (5): In the decoding stage, through the denoising process of the diffusion model, a 3D human motion sequence with rich details and natural movements is finally generated.

[0108] Through the above detailed steps, the present invention provides an efficient and accurate action generation solution, which significantly improves the detail quality and application feasibility of action generation.

[0109] In summary, the method for implementing human motion generation based on a diffusion model provided in this embodiment is an efficient and accurate motion generation solution that can significantly improve the detail quality and application feasibility of motion generation: by combining frequency domain modeling and text semantic alignment technology, the fine-grained performance and semantic consistency of the generated motion are significantly improved, solving the shortcomings of the existing technology in complex motion generation and long sequence stability. This technology can generate more natural, coherent and semantically compliant motion sequences, while improving generation efficiency and quality. It is widely applicable to animation production, virtual reality, game development, human-computer interaction and other fields, and has important technical value and commercial prospects.

[0110] Example 2

[0111] This embodiment provides a computer device, which may be a server or a terminal. Its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the action generation method based on the joint optimization of frequency domain and text semantic information in Example 1. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the action generation method based on the joint optimization of frequency domain and text semantic information described in Example 1 is implemented.

[0112] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0113] Example 3

[0114] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the action generation method based on joint optimization of frequency domain and text semantic information described in the above embodiment 1 is implemented.

[0115] Example 4

[0116] This embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the action generation method based on joint optimization of frequency domain and text semantic information described in the above embodiment 1 is implemented.

[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0118] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0119] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An action generation method based on joint optimization of frequency domain and text semantic information, characterized in that: The action generation method based on joint optimization of frequency domain and text semantic information includes: Obtaining a text feature to be processed and a corresponding initial latent space motion feature, wherein the text feature to be processed is a variable obtained by extracting features from the text description to be processed, and the initial latent space motion feature is a variable obtained by random sampling from a Gaussian distribution; Taking the initial latent space motion features as input, outputting time domain motion features using a pre-trained frequency-space state model; Concatenating the time-domain action feature and the text feature to be processed to obtain a fusion feature; Taking the fused features as input, and using a pre-trained action generation model to output an action sequence; The initial latent space motion features are used as input, and the pre-trained frequency-space state model is used to output the time domain motion features, specifically including: Decomposing the initial latent space motion features into low-frequency components and high-frequency components, wherein the low-frequency components are used to capture global static motion features of the action, and the high-frequency components are used to capture dynamic motion detail features of the action; The time domain motion feature is obtained according to the low frequency component and the high frequency component.

2. The action generation method based on joint optimization of frequency domain and text semantic information according to claim 1 is characterized in that: Obtaining the time domain motion feature according to the low frequency component and the high frequency component specifically includes: constructing a frequency domain state transfer matrix according to the low frequency component, the high frequency component and a proportional parameter, wherein the proportional parameter is a variable determined by a time dynamic relationship between the low frequency component and the high frequency component; Updating a hidden state according to the frequency domain state transfer matrix, wherein an initial hidden state is a zero vector; The time domain action feature is reconstructed according to the updated hidden state, the low frequency component and the high frequency component.

3. The action generation method based on joint optimization of frequency domain and text semantic information according to claim 2 is characterized in that: The time domain action feature and the text feature to be processed are concatenated to obtain a fusion feature, which specifically includes: Obtaining a first fusion feature according to the time-domain action feature and the text feature; According to the text features and the action state matrix, a text-guided state matrix is ​​obtained; Obtaining semantically aligned action features according to the first fused features, the text-guided state matrix, and the updated hidden state; The time domain action feature and the semantically aligned action feature are fused to obtain the fused feature.

4. The action generation method based on joint optimization of frequency domain and text semantic information according to claim 1 is characterized in that: The fusion features are used as input, and the pre-trained action generation model is used to output an action sequence, specifically including: Inputting the fused features into a core denoiser to obtain denoised features, wherein the core denoiser is a denoising network based on UNet; The denoising feature is decoded by using a decoder to obtain the action sequence, wherein the pre-trained action generation model includes the core denoiser and the decoder.

5. The action generation method based on joint optimization of frequency domain and text semantic information according to claim 2, characterized in that: The expression of the frequency domain state transfer matrix is: A n =A+αf low +βf high ; Among them, A n is the frequency domain state transfer matrix; A is the initial value of the state transfer matrix; α and β are scale parameters; f low is the low frequency component; f high is the high frequency component.

6. The action generation method based on joint optimization of frequency domain and text semantic information according to claim 2 is characterized in that: The expression of the time domain motion feature is: in, is the time domain action feature; C is the action state matrix; h t is the updated hidden state; IDWT() is the inverse discrete wavelet transform; f low is the low frequency component; f high is the high frequency component.

7. The action generation method based on joint optimization of frequency domain and text semantic information according to claim 3 is characterized in that: The expression of the semantically aligned action feature is: in, is the semantically aligned action feature; C s is the state matrix guided by the text; h t is the updated hidden state; D is a learnable linear transformation matrix; is the first fusion feature.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the action generation method based on joint optimization of frequency domain and text semantic information as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the action generation method based on joint optimization of frequency domain and text semantic information described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the action generation method based on joint optimization of frequency domain and text semantic information described in any one of claims 1 to 7 is implemented.

Images