Motion generation model-based motion generation method and apparatus, and device

Abstract

A motion generation model-based motion generation method and apparatus, and a device, relating to the technical field of artificial intelligence. The method comprises: acquiring a text containing motion information (210); generating text features of the text by means of a text encoder (220); by means of a first diffusion model and in a feature space of a first dimension, generating an intermediate motion sequence on the basis of the text features (230); and by means of a second diffusion model and in a feature space of a second dimension, performing detail enhancement processing on the intermediate motion sequence so as to obtain an output motion sequence matched with the text, the second dimension being greater than the first dimension (240). The present application preliminarily generates the intermediate motion sequence by means of the first diffusion model, and performs detail enhancement processing on the intermediate motion sequence by means of the second diffusion model, thus improving the detail richness of the output motion sequence.

Description

Action generation method, device and equipment based on action generation model

[0001] This application claims priority to the Chinese patent application filed on August 3, 2023, with application number 202310969504.3 and invention name “Action generation method, device and apparatus based on action generation model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to an action generation method, device and equipment based on an action generation model. Background Art

[0003] Text-driven motion synthesis (TSS) is a method that generates human motion sequences based on textual content. The textual content can cover a variety of motion scenarios in daily life (e.g., walking, talking, exercising, etc.).

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, and device for generating an action based on an action generation model. The technical solution includes the following aspects.

[0006] According to one aspect of an embodiment of the present application, an action generation method based on an action generation model is provided, wherein the action generation model includes a text encoder, a first diffusion model, and a second diffusion model. The method includes: obtaining text containing action information; generating text features of the text through the text encoder; generating an intermediate action sequence according to the text features in a feature space of a first dimension through the first diffusion model; and performing detail enhancement processing on the intermediate action sequence in a feature space of a second dimension through the second diffusion model to obtain an output action sequence matching the text, wherein the second dimension is larger than the first dimension.

[0007] According to one aspect of an embodiment of the present application, a training method for an action generation model is provided, wherein the action generation model includes a text encoder, a first diffusion model, and a second diffusion model, and the method includes: obtaining a training sample set of the action generation model, the training sample set including at least one action-text pair, each action-text pair including a sample text and an original action sequence with a matching relationship; generating text features of the sample text through the text encoder; generating a first action sequence matching the sample text according to the text features in a feature space of a first dimension through the first diffusion model; generating a second action sequence matching the sample text according to the text features in a feature space of a second dimension through the second diffusion model, wherein the second dimension is larger than the first dimension; adjusting parameters of the first diffusion model according to the first action sequence and the original action sequence, and adjusting parameters of the second diffusion model according to the second action sequence and the original action sequence, to obtain a trained action generation model.

[0008] According to one aspect of an embodiment of the present application, an action generation device based on an action generation model is provided, wherein the action generation model includes a text encoder, a first diffusion model and a second diffusion model, and the device includes: a text acquisition module for acquiring text containing action information; a text feature generation module for generating text features of the text through the text encoder; an intermediate sequence generation module for generating an intermediate action sequence according to the text features in a feature space of a first dimension through the first diffusion model; and an output sequence generation module for performing detail enhancement processing on the intermediate action sequence in a feature space of a second dimension through the second diffusion model to obtain an output action sequence matching the text, wherein the second dimension is larger than the first dimension.

[0009] According to one aspect of an embodiment of the present application, a training device for an action generation model is provided, wherein the action generation model includes a text encoder, a first diffusion model, and a second diffusion model, and the method includes: a sample set acquisition module for acquiring a training sample set for the action generation model, wherein the training sample set includes at least one action-text pair, each of the action-text pairs including a sample text and an original action sequence having a matching relationship; a text feature generation module for generating text features of the sample text through the text encoder; a first sequence generation module for generating a first action sequence matching the sample text according to the text features in a feature space of a first dimension through the first diffusion model; a second sequence generation module for generating a second action sequence matching the sample text according to the text features in a feature space of a second dimension through the second diffusion model, wherein the second dimension is larger than the first dimension; and a parameter adjustment module for adjusting the parameters of the first diffusion model according to the first action sequence and the original action sequence, and adjusting the parameters of the second diffusion model according to the second action sequence and the original action sequence, to obtain a trained action generation model.

[0010] According to one aspect of an embodiment of the present application, a computer device is provided, which includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned action generation method based on the action generation model, or the training method of the action generation model.

[0011] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned action generation method based on the action generation model, or the training method of the action generation model.

[0012] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes a computer program, and the computer program is loaded and executed by a processor to implement the above-mentioned action generation method based on the action generation model, or the training method of the action generation model.

[0013] The technical solution provided by the embodiment of the present application can bring the following beneficial effects: the first diffusion model can process the features under the feature space of the first dimension, and the second diffusion model can process the features under the feature space of the second dimension. Since the second dimension is larger than the first dimension, the second diffusion model can pay more attention to fine-grained features. Therefore, the intermediate action sequence is generated by the first diffusion model in the feature space of the first dimension, and the modeling diffusion of the text in the feature space of the low dimension is preliminarily realized. Since the details of the action sequence obtained by diffusion in the feature space of the low dimension are not rich enough, the intermediate action sequence is further enhanced by the second diffusion model in the feature space of the second dimension to obtain the output action sequence. Since the action sequence is refined in the feature space of the high dimension, the details of the output action sequence are richer and more matched with the text. Compared with the problem of the lack of details in the generated action sequence caused by the diffusion process of the text modeling in the feature space of the low dimension in the related art, the technical solution provided by the present application enhances the detail features in the output action sequence in the feature space of the high dimension, thereby improving the richness of details in the output action sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a schematic diagram of an implementation environment provided by an embodiment of the present application.

[0015] FIG2 is a flowchart of an action generation method based on an action generation model provided in an embodiment of the present application.

[0016] FIG3 is a schematic diagram showing a comparison between a reconstruction effect of an action sequence provided in an embodiment of the present application and a real action sequence.

[0017] FIG4 is a schematic structural diagram of an action generation model provided in an embodiment of the present application.

[0018] FIG5 is a flowchart of another action generation method based on an action generation model provided in an embodiment of the present application.

[0019] FIG6 is a schematic diagram of an action sequence provided in an embodiment of the present application.

[0020] FIG7 is a comparative schematic diagram of the generation effects of an action sequence generation method provided in an embodiment of the present application.

[0021] FIG8 is a flowchart of another method for training an action generation model provided in an embodiment of the present application.

[0022] FIG9 is an algorithm flow chart of a process of using an action generation model provided in an embodiment of the present application.

[0023] FIG10 is a block diagram of an action generation device based on an action generation model provided in an embodiment of the present application.

[0024] FIG11 is a block diagram of another training device for an action generation model provided in an embodiment of the present application.

[0025] FIG12 is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0027] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0028] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0029] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0030] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0031] The technical solution of this application mainly involves machine learning technology in artificial intelligence technology, mainly involving the training and use process of action generation models.

[0032] In the context of text-driven action generation, the related art first uses an auto-encoder (AE) to learn the text features of the text, and then uses variational autoencoders (VAE) to convert the feature distribution of the text into a normal distribution in a low-dimensional feature space. This modeling of the diffusion process in this feature space generates an action sequence corresponding to the text features. However, due to the insufficient representation capability of the low-dimensional feature space, the human action sequences generated by this method often lack rich details.

[0033] Based on this, the present application provides an action generation method based on an action generation model. For detailed process, please refer to the description of each embodiment below.

[0034] Please refer to Figure 1, which shows a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment can be a training and use system for an action generation model. The implementation environment can include: a model training device 10 and a model use device 20.

[0035] The model training device 10 can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, multimedia player, vehicle terminal, server, intelligent robot, or other electronic devices with strong computing capabilities. The model training device 10 is used to train the action generation model.

[0036] In the embodiment of the present application, the action generation model is a machine learning model trained based on the action generation model training method, and is used to generate an output action sequence that matches the text based on the text containing action information. The model training device 10 can use machine learning to train the action generation model so that it has the ability to generate an output action sequence that matches the text based on the text. The specific model training method can be referred to in the following embodiment.

[0037] The action generation model includes a text encoder, a first diffusion model, and a second diffusion model. The text encoder is used to encode the text and generate text features of the text; the first diffusion model is used to generate an intermediate action sequence in a low-dimensional feature space; the second diffusion model is used to perform detail enhancement processing on the intermediate action sequence in a high-dimensional feature space to obtain an output action sequence that matches the text. It should be noted that the low dimension and high dimension here are relative, and the dimension of the feature space processed by the second diffusion model is higher than the dimension of the feature space processed by the first diffusion model. For example, the first diffusion model is used to process in a feature space of the first dimension, and the second diffusion model is used to process in a feature space of the second dimension, where the second dimension is larger than the first dimension.

[0038] The dimension of a feature space refers to the number of elements contained in the features in that feature space. For example, if the features in the feature space are feature vectors, the dimension of the feature space refers to the number of eigenvalues ​​contained in the feature vectors in that feature space, that is, the length of the feature vectors. Eigenvalues ​​in different dimensions are used to represent characteristics of different attributes. For example, if the first dimension is 2*256, then the dimension of the features in the feature space of the first dimension is 2*256. If the second dimension is 8*256, then the dimension of the features in the feature space of the second dimension is 8*256. The second dimension is larger than the first dimension.

[0039] The first and second diffusion models are generative models in the field of artificial intelligence. These models use neural networks to gradually recover the true data distribution from a Gaussian noise distribution through multiple rounds of iterative learning. The diffusion model primarily involves two diffusion processes: forward diffusion and backward diffusion. The forward diffusion process incrementally adds Gaussian noise to the text, generating a series of noisy texts with varying noise intensities. These noisy texts are then used to learn the diffusion network. The backward diffusion process utilizes the trained diffusion network to incrementally remove noise from the noisy texts, thereby recovering the action sequence from the Gaussian noise.

[0040] The first diffusion model and the second diffusion model both generate action sequences through forward diffusion and backward diffusion. The difference is that the feature space processed by the first diffusion model has a lower dimensionality than the feature space processed by the second diffusion model. The first diffusion model can process coarse-grained features during the diffusion process, while the second diffusion model can mine fine-grained features. Therefore, after the coarse-grained action sequence is generated by the first diffusion model, the second diffusion model can be used to refine the coarse-grained action sequence to generate an action sequence with richer details. The first diffusion model can be referred to as the "basic diffusion model," and the second diffusion model can be referred to as the "advanced diffusion model."

[0041] Therefore, in an embodiment of the present application, the text is input into the action generation model, and the text features of the text are first generated through a text encoder. Then, an intermediate action sequence is generated according to the text features in a low-dimensional feature space through a first diffusion model. Finally, the intermediate action sequence is enhanced in detail through a second diffusion model in a high-dimensional feature space to obtain an output action sequence that matches the text.

[0042] The trained action generation model can be deployed in a model-using device 20 for use. This device can be a terminal device such as a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, multimedia player, vehicle-mounted terminal, intelligent robot, or a server. When it is necessary to generate an output action sequence that matches a text, the model-using device 20 can use the trained action generation model to achieve this function.

[0043] The model training device 10 and the model using device 20 can be two independent devices or the same device. If the model training device 10 and the model using device 20 are the same device, the model training device 10 can be deployed in the model using device 20.

[0044] In the embodiments of the present application, the execution subject of each step can be a computer device, which refers to an electronic device with data calculation, processing and storage functions. The computer device can be a terminal device such as a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, multimedia playback device, vehicle terminal, intelligent robot, etc., or it can be a server. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The computer device can be a model training device 10 as shown in Figure 1, or it can be a model use device 20.

[0045] Please refer to Figure 2, which shows a flowchart of an action generation method based on an action generation model provided in an embodiment of the present application. The action generation model includes a text encoder, a first diffusion model, and a second diffusion model. The execution entity of each step of the method can be a computer device. The method can include at least one of the following steps 210 to 240.

[0046] Step 210: Obtain text containing action information.

[0047] Action information refers to a text segment containing an action description, wherein the action description is a text description of the motion characteristics of a person's body part, and can cover a variety of motion scenes in daily life. Exemplarily, it can be a text description of the motion characteristics of a single body part of a person. For example, it can be a text description of the motion characteristics of a person's legs, such as descriptions of walking, jogging, kicking, and stepping back; it can also be a text description of the motion characteristics of a person's hands, such as descriptions of raising hands, waving, clapping, and clenching fists; it can also be a text description of the motion characteristics of a person's head, such as descriptions of looking up, lowering the head, talking, smiling, and so on. Exemplarily, it can also be a text description of the overall motion characteristics of multiple body parts of a person, such as descriptions of running with arms swinging, bending over to pick up something, climbing stairs, dancing, and so on.

[0048] The characters mentioned here can be real people or virtual characters. If the characters are virtual characters, they are not limited to human form and can include animal form and any virtual form created independently.

[0049] The text includes action information, which is used to describe an action. The action information refers to a text segment in the text that describes an action. For example, the text may be "someone waves his right hand," where "waves his right hand" is the action information.

[0050] Step 220: Generate text features of the text through a text encoder.

[0051] The text encoder is used to encode the text and generate text features of the text. Exemplarily, the text encoder can be a clip text encoder.

[0052] The computer device inputs the text into the text encoder, which encodes the text and outputs the text features. The text features are used to represent the semantics of the text. If the text contains action information describing an action, the text features include features representing the action.

[0053] In an embodiment of the present application, the text features generated by the text encoder may be text features of a first dimension, where the first dimension is the dimension of the feature space processed by the first diffusion model.

[0054] Step 230 : Generate an intermediate action sequence based on text features in a feature space of a first dimension using a first diffusion model.

[0055] Among them, the intermediate action sequence includes at least one action. Since the intermediate action sequence is obtained by diffusing the text feature, the action in the intermediate action sequence matches the action represented by the text feature, and the action represented by the text feature is the action described by the text, so the action in the intermediate action sequence matches the action described by the text. However, since the first diffusion model only processes in a relatively low-dimensional feature space, it is difficult to mine fine-grained features, so the intermediate action sequence generated by the first diffusion model has relatively low detail richness, that is, the intermediate action sequence has a relatively low degree of matching with the text. The relative here refers to compared with the output action sequence generated by the second diffusion model.

[0056] In some embodiments, the first diffusion model includes a first variational autoencoder and a first denoiser, the first variational autoencoder including a first encoder and a first decoder.

[0057] The first variational autoencoder is any variational autoencoder including a first encoder and a first decoder. The first variational autoencoder is used to add Gaussian noise to the encoded features and decode the noisy features.

[0058] In an embodiment of the present application, the first diffusion model converts the text features of the text into a latent space distribution in a feature space of the first dimension, then models diffusion in this feature space, and reconstructs the latent space distribution in the feature space into an intermediate action sequence. The Gaussian noise added during the encoding and decoding process of the first variational autoencoder is noise of the first dimension. The first encoder is used to convert the text features into a latent space distribution in the feature space of the first dimension, and the first decoder is used to reconstruct the latent space distribution of the first dimension into an intermediate action sequence.

[0059] Different variational autoencoders can correspond to feature spaces of different dimensions. In feature spaces of different dimensions, the action sequences reconstructed by the variational autoencoder are different. Generally speaking, the variational autoencoder corresponding to the high-dimensional feature space has a better reconstruction effect on the action sequence than the variational autoencoder corresponding to the low-dimensional feature space.

[0060] Figure 3 shows a comparison of the reconstruction results of different variational autoencoders (VAEs) on action sequences and the real action sequences in feature spaces of different dimensions. As can be seen from Figure 3, when the dimensionality of the feature space is low, for example, when the feature dimensions are 1*256, 2*256, and 4*256, the reconstruction results of the action sequence by the variational autoencoder lack the details of the hand movements. For details, please refer to the hand movements in the marked boxes in VAE-1, VAE-2, and VAE-4. When the dimensionality of the feature space is high, for example, when the feature dimensions are 8*256 and 12*256, the reconstruction results of the action sequence by the variational autoencoder are richer in hand movement details. For details, please refer to the hand movements in the marked boxes in VAE-8 and VAE-12. Compared with the hand movement details in the real action sequence, it is obvious that the hand movement sequences in VAE-8 and VAE-12 are closer to the hand movement details in the real action sequence. It can be seen that the reconstruction effect of VAE-8 and VAE-12 on the action sequence is better than that of VAE-1, VAE-2 and VAE-4.

[0061] Step 230 includes at least one sub-step of steps 231 to 232 (not shown in FIG. 2 ).

[0062] Step 231 : Generate a random motion feature of random noise through a first encoder, where the dimension of the random motion feature is a first dimension.

[0063] The first encoder is an encoder included in the first variational autoencoder.

[0064] A random motion feature refers to a randomly generated noise motion feature, which is used to represent the characteristic information of a random motion sequence. The random motion feature can be understood as a feature of a random noise, and the random noise can represent a random motion sequence. The random motion feature can be generated by a random number (or can be called a random seed), for example, the random motion feature is composed of multiple random numbers. Different random numbers correspond to different random motion features, and a random number refers to an arbitrary number. The random motion features corresponding to different random numbers have different motion characteristics, which can be motion characteristics with different motion styles, for example, an animation style with more exaggerated motions, a traditional style with more subtle motions, a casual style with more casual motions, and so on.

[0065] In some embodiments, the random motion feature is directly randomly generated by the first encoder, rather than first randomly generating random noise and then encoding the random noise to obtain the random motion feature. Alternatively, a random noise may be randomly generated first and then encoded by the first encoder to obtain the random motion feature of the first dimension.

[0066] Step 232 : Generate an intermediate action sequence based on the random action features and the text features through a first noise reducer and a first decoder.

[0067] In some embodiments, a first noise of a first dimension is added to the random action feature to obtain a noisy random action feature; the noisy random action feature is denoised according to the text feature through a first denoiser to obtain a denoised random action feature of the first dimension; the denoised random action feature is decoded through a first decoder to obtain an intermediate action sequence.

[0068] The first diffusion model includes a forward diffusion process and a backward diffusion process. The forward diffusion process of the first diffusion model incrementally adds noise to the random action features, causing them to gradually lose their characteristic information. After T noise additions, the random action features become a latent space distribution devoid of any action features, where T is a positive integer. Furthermore, the backward diffusion process of the first diffusion model denoises and decodes this latent space distribution to reconstruct the intermediate action sequence. The backward diffusion process of the first diffusion model is the processing of the first denoiser and the first decoder. The latent space distribution refers to the probability distribution in the latent space, which is a continuous vector space. The features in this latent space are potential and abstract representations of the data.

[0069] The first denoiser is any denoiser that is used to remove noise from the random action features after adding noise one by one according to the constraints (i.e., text features) in the feature space of the first dimension, so that the action features constrained by the text features are gradually revealed. After T denoising operations, the denoised random action features will become action features that match the text.

[0070] The first decoder is included in the first variational autoencoder and is configured to reconstruct the output data of the first denoiser into an intermediate action sequence. The intermediate action sequence is the output data of the first diffusion model and is configured to represent the intermediate denoising results obtained by the first diffusion model through stepwise denoising.

[0071] Because the denoised random action features are obtained under the constraints of the textual features of the text, the denoised random action features match the semantics of the text. Since the intermediate action sequence is obtained by decoding the denoised random action features, the intermediate action sequence matches the semantics of the text. The semantics of the text refers to the semantic information corresponding to the text, which is used to represent the semantic features to be expressed by the text, including the semantics of the action information. The semantic matching of the intermediate action sequence with the text means that the intermediate action sequence matches the semantics of the action information in the text. The intermediate action sequence can provide a preliminary and approximate representation of the action information contained in the text.

[0072] In the embodiment of the present application, a first diffusion model is used to initially reconstruct an intermediate action sequence in a low-dimensional feature space based on the random action features of the first dimension and the text features of the first dimension. This provides a basis for the subsequent diffusion process of detail enhancement of the intermediate action sequence in the high-dimensional feature space, thereby improving the convenience and speed of the action sequence reconstruction process. Furthermore, through the forward and backward diffusion processes of the first diffusion model, the random action features are denoised under the constraints of the text features, so that the reconstructed intermediate action sequence matches the text as closely as possible, thereby improving the accuracy of the action sequence reconstruction process.

[0073] Step 240 : Perform detail enhancement processing on the intermediate action sequence in a second-dimensional feature space using a second diffusion model to obtain an output action sequence that matches the text, where the second dimension is greater than the first dimension.

[0074] The intermediate action sequence is the input data of the second diffusion model. The second diffusion model is used to further enhance the intermediate action sequence based on the intermediate action sequence obtained by the first diffusion model to obtain an output action sequence that matches the text and has richer details.

[0075] Detail enhancement processing refers to adjusting the actions in the intermediate action sequence to improve the detail richness of the actions in the intermediate action sequence. In the embodiment of the present application, using the second diffusion model for detail enhancement processing refers to adding noise to the intermediate action sequence through the second diffusion model, and then gradually removing the noise to restore an output action sequence with higher detail richness.

[0076] The output action sequence matches the text, which means that the actions in the output action sequence are consistent with the actions described in the text. For example, if the action described in the text is running, then at least one action in the output action sequence constitutes a running action.

[0077] In some embodiments, the second diffusion model includes a second variational autoencoder and at least two second denoisers, the second variational autoencoder including a second encoder and a second decoder.

[0078] The second variational autoencoder is any variational autoencoder including a second encoder and a second decoder. The second variational autoencoder is used to add Gaussian noise to the encoded features and decode the noisy features.

[0079] In an embodiment of the present application, the second diffusion model converts the intermediate action sequence into a latent space distribution in a second-dimensional feature space, then models the diffusion process in this feature space, and further reconstructs the latent space distribution in the feature space into an output action sequence that matches the text. The Gaussian noise added to the encoding and decoding process of the second variational autoencoder is noise of the second dimension. The second encoder is used to convert the intermediate action sequence into a latent space distribution in the second-dimensional feature space, and the second decoder is used to reconstruct the latent space distribution of the second dimension into an output action sequence.

[0080] Since the second dimension is larger than the first dimension, compared with the first diffusion model, the feature space corresponding to the second diffusion model is a high-dimensional feature space, and the feature space corresponding to the first diffusion model is a low-dimensional feature space. The second variational autoencoder in the second diffusion model reconstructs the action sequence better than the variational autoencoder in the first diffusion model.

[0081] By using the output data of the first diffusion model as the input data of the second diffusion model, the second diffusion model is used to perform detail enhancement processing on the intermediate action sequence output by the first diffusion model. This allows the output action sequence to more closely match the text in terms of details, thereby improving the detail richness of the action sequence. As a result, the output action sequence has a higher degree of match with the text than the intermediate action sequence, and the output action sequence has a higher detail richness than the intermediate action sequence.

[0082] Step 240 includes at least one sub-step of steps 241 to 242 (not shown in FIG. 2 ).

[0083] In step 241 , an intermediate action feature of the intermediate action sequence is generated by a second encoder, where the dimension of the intermediate action feature is a second dimension.

[0084] The second encoder is an encoder included in the second variational autoencoder, which is used to convert the intermediate action sequence into a latent space distribution in the feature space of the second dimension.

[0085] The intermediate action feature is the latent space distribution of the intermediate action sequence generated by the second encoder, and is used to represent the feature distribution of the intermediate action sequence in the feature space of the second dimension.

[0086] In some embodiments, the second encoder encodes the intermediate action sequence to obtain the intermediate action feature.

[0087] Step 242 : Generate an output action sequence matching the text based on the intermediate action features through at least two second noise reducers and a second decoder.

[0088] In some embodiments, a second noise of a second dimension is added to the intermediate action feature to obtain a noisy intermediate action feature; the noisy intermediate action feature is denoised in turn by at least two second denoisers to obtain a denoised intermediate action feature of the second dimension; the denoised intermediate action feature is decoded by a second decoder to obtain an output action sequence that matches the text.

[0089] The intermediate action sequence is used as the input data of the second diffusion model. The second diffusion model includes a forward diffusion process and a backward diffusion process. Through the forward diffusion process of the first diffusion model, noise is added to the intermediate action features one by one, and the intermediate action features lose their characteristic information one by one. h After adding noise, the intermediate action features will become a latent space distribution without any action features, T h is an integer greater than or equal to 2. Furthermore, the latent space distribution is denoised and decoded through the backward diffusion process of the second diffusion model to reconstruct the output action sequence. The backward diffusion process of the second diffusion model is the processing process of at least two second denoisers and a second decoder.

[0090] In some embodiments, the noisy intermediate action features are denoised in sequence by at least two second denoisers to obtain denoised intermediate action features of a second dimension, including: denoising the noisy intermediate action features in sequence according to text features of the text by at least two second denoisers to obtain denoised intermediate action features of a second dimension.

[0091] At least two second denoisers are any two denoisers, which are used to remove noise from the intermediate action features after adding noise in the feature space of the second dimension according to the constraint conditions (i.e. text features), so that the action features after the text feature control constraint are gradually revealed, and T is performed. h After the first denoising, the denoised intermediate action features will be further transformed into action features that match the text.

[0092] The second decoder, included in the second variational autoencoder, is used to reconstruct the output data of the at least two second denoisers into an output action sequence that matches the text. The output action sequence is the output data of the second diffusion model, representing the action sequence that matches the text and capable of restoring the action details described in the text.

[0093] In an embodiment of the present application, through the forward diffusion process and backward diffusion process of the second diffusion model, the intermediate action sequence output by the first diffusion model is denoised under the constraint conditions with text features, thereby enhancing the detail features in the reconstructed output action sequence, making the output action sequence more matched with the text in details, and improving the richness of details in the action sequence.

[0094] Figure 4 shows a schematic diagram of the structure of an action generation model. The implementation process of the first diffusion model and the second diffusion model is roughly the same. Both use an encoder to convert the input data into a noise distribution in the feature space of the variational autoencoder, add noise to the noise distribution, and then convert the noisy noise distribution into a de-noised feature distribution in the feature space. Finally, the decoder reconstructs the de-noised feature distribution into an action sequence. The difference between the network frameworks of the two diffusion models is that the dimension of the feature space of the first diffusion model is lower than that of the feature space of the second diffusion model, and in the first diffusion model, a first denoiser is used to denoise the noisy random action features, while in the second diffusion model, at least two second denoisers are used to denoise the noisy intermediate action features (two second denoisers are used as an example in Figure 4).

[0095] In Figure 4, the gray blocks displayed in the first and second denoisers represent text features. The denoisers denoise the noisy motion features based on these text features, thereby generating denoised motion features. The number of denoising operations during the denoising process is related to the dimensionality of the added noise. For example, if the dimensionality of the first noise is smaller than that of the second noise, the number of denoising operations performed by the first denoiser in the first diffusion model is smaller than the sum of the denoising operations performed by at least two second denoisers in the second diffusion model.

[0096] Figure 4 also shows the connection relationship between the first diffusion model and the second diffusion model. The output data of the first diffusion model will serve as the input data of the second diffusion model. The second diffusion model is based on the intermediate action sequence output by the first diffusion model, and further performs detail enhancement processing on the intermediate action sequence to obtain an output action sequence that is more matched with the action information in the text. The output action sequence can better restore the action details described in the text.

[0097] In the technical solution provided by the embodiment of the present application, the first diffusion model can process the features under the feature space of the first dimension, and the second diffusion model can process the features under the feature space of the second dimension. Since the second dimension is larger than the first dimension, the second diffusion model can pay more attention to fine-grained features. Therefore, the intermediate action sequence is generated by the first diffusion model in the feature space of the first dimension, and the modeling diffusion of the text in the feature space of the low dimension is preliminarily realized. Since the details of the action sequence obtained by diffusion in the feature space of the low dimension are not rich enough, the intermediate action sequence is further enhanced by the second diffusion model in the feature space of the second dimension to obtain the output action sequence. Since the action sequence is refined in the feature space of the high dimension, the details of the output action sequence are richer and more matched with the text. Compared with the problem of insufficient details in the generated action sequence caused by only modeling the text diffusion process in the feature space of the low dimension in the related art, the technical solution provided by the present application enhances the detail features in the output action sequence in the feature space of the high dimension, thereby improving the richness of details in the output action sequence.

[0098] Please refer to Figure 5, which shows a flowchart of another action generation method based on an action generation model provided by an embodiment of the present application. The execution subject of each step of the method can be a computer device. The method can include at least one step from the following steps 510 to 590.

[0099] Step 510: Obtain text containing action information and generate text features of the text through a text encoder.

[0100] Step 520 : Generate a random motion feature of random noise through a first encoder, where the dimension of the random motion feature is a first dimension.

[0101] The relevant contents in the above steps 510 and 520 can be referred to the above embodiments and will not be described again here.

[0102] Step 530: Add first noise of a first dimension to the random motion feature to obtain a noisy random motion feature.

[0103] The first noise refers to Gaussian noise in the first dimension, and the random motion feature is the characteristic distribution in the first dimension. Optionally, the first noise can be added to the random motion feature T times, or the first noise can be added all at once to obtain a noisy random motion feature. The noisy random motion feature is a noise distribution without motion characteristics, and the dimension of the noisy random motion feature is also the first dimension.

[0104] Step 540 : De-noise the random action features after adding noise according to the text features by using a first denoiser to obtain the de-noised random action features of the first dimension.

[0105] The first denoiser removes the noise from the noisy random action features based on the text features. After T denoising cycles, the denoised random action features of the first dimension are obtained. The denoised random action features match the semantic information of the text features and include action features corresponding to the action information.

[0106] Optionally, the number of the first noise reducers may be one or more. In the embodiment of the present application, in order to improve the operating efficiency of the first diffusion model, the number of the first noise reducers is set to 1, that is, the single noise reducer network framework shown in FIG4 .

[0107] Step 550: Decode the denoised random action features through a first decoder to obtain an intermediate action sequence.

[0108] The denoised random action features are decoded by the first decoder, and the denoised random action features are reconstructed into an intermediate action sequence.

[0109] In an embodiment of the present application, by modeling the diffusion process of random action features in the feature space of the first dimension and denoising the random action features after noise addition, an intermediate action sequence is preliminarily obtained. Since the distribution of the feature space of the low dimension is relatively simple, it is easier to realize the conversion from noise distribution to feature distribution, and the processing process is relatively convenient.

[0110] Step 560 : Generate intermediate action features of the intermediate action sequence through a second encoder, where the dimension of the intermediate action features is the second dimension.

[0111] The relevant contents in step 560 can be referred to the above embodiment and will not be repeated here.

[0112] Step 570: Add the second noise of the second dimension to the intermediate motion feature to obtain the noisy intermediate motion feature.

[0113] The second noise refers to the Gaussian noise of the second dimension, and the intermediate action feature is the feature distribution of the second dimension. Optionally, the second noise can be divided into T h The noise can be added to the intermediate motion features one by one, or the second noise can be added to the random motion features all at once to obtain the noisy intermediate motion features. The noisy intermediate motion features are noise distributions without motion features, and the dimension of the noisy intermediate motion features is also the second dimension.

[0114] Step 580: De-noise the noisy intermediate motion features in sequence through at least two second denoisers to obtain de-noised intermediate motion features of a second dimension.

[0115] Optionally, at least two second denoisers are used to remove noise from the random action features after adding noise according to the text features, and after T h After the second denoising, the denoised intermediate action features of the second dimension are obtained. The denoised intermediate action features are action features that are further matched with the detailed features of the action information on the basis of matching the semantic information of the text features.

[0116] In some embodiments, the number of the second noise reducers is N, where N is an integer greater than or equal to 2.

[0117] Then the i-th second denoiser performs T on the noised intermediate action features. i Second denoising, N second denoisers perform T denoising on the intermediate action features after denoising. h Second denoising, T i is a positive integer. For example, the first second denoiser performs T1 denoising on the intermediate action features after adding noise, and the second second denoiser performs T2 denoising on the intermediate action features after adding noise, then T h =T1+T2+…T N .

[0118] In the embodiment of the present application, in order to achieve the coordination of both operating efficiency and generation effect and avoid the reduction of operating efficiency due to the excessive number of second denoisers, the number of second denoisers is set to 2, i.e. the multi-denoiser network framework shown in FIG4 . Among them, the first second denoiser performs T1 denoising on the intermediate action features after adding noise, and the second second denoiser performs T2 denoising on the intermediate action features after adding noise, then T h =T1+T2.

[0119] Step 580 includes at least one sub-step from step 581 to step 583 (not shown in the figure).

[0120] Step 581: The denoised intermediate action features are subjected to T by the i-th second denoiser. i The denoising is performed once, and the intermediate action features after denoising are obtained. i is a positive integer less than or equal to N, and the initial value of i is 1. i Is a positive integer.

[0121] Step 582: If i is less than N, set i=i+1 and start again from step 581.

[0122] If i is less than N, it means that the denoising of the intermediate action features after adding noise has not been completed, and it is necessary to continue denoising the intermediate action features after the i-th denoising. Let i = i + 1, and perform T denoising on the intermediate action features after the i-th denoising through the i + 1 second denoiser. i+1 The denoising is performed once to obtain the (i+1)th denoised intermediate action feature.

[0123] In step 583 , if i is equal to N, the i-th intermediate motion feature after noise reduction is determined as the intermediate motion feature after noise reduction.

[0124] If i is equal to N, it means that the denoising of the noisy intermediate motion feature has been completed, and the Nth denoised intermediate motion feature is the denoised intermediate motion feature obtained by denoising the noisy intermediate motion feature in sequence by at least two second denoisers.

[0125] That is, if i is equal to 1, the noise-added intermediate action features are subjected to T by the first second denoiser. i The first denoising is performed to obtain the intermediate action feature after the first denoising. If i is greater than 1 and less than N, the intermediate action feature after the first denoising is T-processed by the second denoiser i-1. i The denoising is performed once to obtain the intermediate action feature after denoising of the i-th time. If i is equal to N, the intermediate action feature after denoising of the N-1th time is subjected to T by the N-th second denoiser. i The Nth denoised intermediate motion feature is obtained, and the Nth denoised intermediate motion feature is the denoised intermediate motion feature.

[0126] For example, taking N equal to 5 as an example, the first second denoiser performs T1 denoising on the intermediate motion feature after noise addition to obtain the first denoised intermediate motion feature. Then, the second second denoiser performs T2 denoising on the intermediate motion feature after noise reduction to obtain the second denoised intermediate motion feature. Then, the third second denoiser performs T3 denoising on the intermediate motion feature after noise reduction to obtain the third denoised intermediate motion feature. And so on, until the fifth second denoiser performs T5 denoising on the intermediate motion feature after noise reduction to obtain the fifth denoised intermediate motion feature, which is the denoised intermediate motion feature.

[0127] In an embodiment of the present application, multiple second denoisers are set in the second diffusion model to denoise the intermediate action features after noisy, so that each second denoiser is responsible for the denoising process within a specific time period, and the denoising process is more detailed, thereby enhancing the detail features in the output action sequence and improving the detail richness in the output action sequence.

[0128] Step 590: Decode the denoised intermediate action features through a second decoder to obtain an output action sequence that matches the text.

[0129] The denoised random action features are decoded by the second decoder, and the denoised intermediate action features are reconstructed into an output action sequence that matches the text.

[0130] The technical solution provided by the embodiment of the present application preliminarily generates an intermediate action sequence through a first diffusion model in a low-dimensional feature space based on the denoising process of a first denoiser, and then generates an output action sequence that further matches the action information of the text on the basis of matching the semantics of the text through a second diffusion model in a high-dimensional feature space based on the denoising process of at least two second denoisers, thereby enhancing the detail features in the output action sequence and improving the detail richness in the output action sequence.

[0131] The action generation method based on the action generation model provided in this application combines the respective advantages of the first diffusion model in the low-dimensional feature space and the second diffusion model in the high-dimensional feature space, allowing the first diffusion model and the second diffusion model to be responsible for different stages of the reverse diffusion process respectively, and finally generating a motion sequence that matches the text and is rich in details of the action information.

[0132] Figure 6 shows a schematic diagram of multiple action sequences that are consistent with the semantics of the text but are different. As can be seen from Figure 6, the action generation method based on the action generation model proposed in the embodiment of the application can generate multiple action sequences for each text that are consistent with the semantics of the text but have different detailed features of the action information.

[0133] As shown in Figure 6 (1), action sequence 1 and action sequence 2 are both generated based on the text "a person dancing with someone". Although action sequence 1 is different from the real action sequence, it matches the semantics of the text. Therefore, both action sequence 1 and action sequence 2 can be used as action sequences corresponding to the text "a person dancing with someone". The same reason applies to Figures 6 (2) and (3). Action sequence 2 shown in Figure 6 (2) is different from the real action sequence, and action sequence 1 shown in Figure 6 (3) is different from the real action sequence. However, action sequence 1 and action sequence 2 in Figure 6 (2) can both be used as action sequences corresponding to the text "a person is waving his right hand", and action sequence 1 and action sequence 2 in Figure 6 (3) can both be used as action sequences corresponding to the text "a person steps back and sits on a chair with his arms at his sides, and then stands up from the chair".

[0134] As shown in FIG6 , it can be seen that the technical solution proposed in this application not only ensures the generation effect, but also has the ability to generate diverse results.

[0135] FIG7 is a schematic diagram showing a comparison of the generation effects of the technical solution of the present application and other action sequence generation methods.

[0136] By comparison, it can be seen that the method proposed in this application, namely the refined text-driven action generation method based on the hierarchical diffusion model (Basic-to-Advanced Hierarchical Diffusion Model, B2A-HDM), can better generate action sequences that match the semantics of the text and are richer in details. For example, according to Figures (1), (2), (3), and (4) in Figure 7, it can be seen that Motion Diffuse and MDM cannot generate motion sequences that match the semantics of the text, while T2M-GPT and MLD do not generate enough action details. The B2A-HDM proposed in this application can simultaneously take into account the matching degree of the generated results with the text and the restoration degree of the action details.

[0137] Table 1 below shows the quantitative comparison of the results of the B2A-HDM proposed in this application and other action sequence generation methods on the HumanML3D dataset, where the best results are presented in bold.

[0138] Table 1

[0139] The evaluation metrics used for comparison include R-Precision, FID, MM Dist, Diversity, and MModality. P-Precision and MM Dist measure the degree of match between the generated results and the text; FID measures the degree of restoration of the generated results, that is, whether the generated results are close to the real samples; and Diversity and MModality measure the diversity of the generated results.

[0140] Table 2 below shows the quantitative results comparison of the B2A-HDM proposed in this application and other action sequence generation methods on the KIT-ML dataset.

[0141] Table 2

[0142] The comparison results in Tables 1 and 2 above demonstrate that the proposed B2A-HDM method significantly outperforms other action sequence generation methods in terms of both the degree of match between generated results and text and the degree of restoration of generated results. Furthermore, B2A-HDM also achieves impressive results in terms of diversity, demonstrating its ability to generate diverse results.

[0143] Please refer to Figure 8, which shows a flowchart of a method for training an action generation model provided by one embodiment of the present application. The action generation model includes a text encoder, a first diffusion model, and a second diffusion model. Each step of the method may be performed by a computer device. The method may include at least one of the following steps 810 to 850.

[0144] Step 810: Obtain a training sample set for the action generation model. The training sample set includes at least one action text pair. Each action text pair includes a sample text and an original action sequence with a matching relationship.

[0145] A matching relationship between the sample text and the original action sequence means that the action described in the sample text matches the action in the original action sequence. In other words, the action described in the sample text is consistent with the action in the original action sequence. For example, if the action described in the sample text is a shooting action, at least one action in the original action sequence constitutes a shooting action.

[0146] Optionally, an action text pair includes a sample text and an original action sequence, but a sample text may correspond to multiple original action sequences, so an action text pair may be constructed based on each sample text and one of the original action sequences, and each sample text may construct at least one action text pair.

[0147] Step 820: Generate text features of the sample text through a text encoder.

[0148] Step 830 : Generate a first action sequence matching the sample text according to the text features in a feature space of the first dimension using a first diffusion model.

[0149] In some embodiments, the first diffusion model includes a pre-trained first variational autoencoder and a first denoiser, the first variational autoencoder including a first encoder and a first decoder.

[0150] In some embodiments, a first action feature of a first random noise is generated through a first encoder, and the dimension of the first action feature is a first dimension; a first action sequence matching the sample text is generated based on the first action feature and the text feature through a first noise reducer and a first decoder.

[0151] In some embodiments, a first noise of a first dimension is added to the first action feature to obtain a noisy first action feature; the noisy first action feature is denoised according to the text feature by a first denoiser to obtain a denoised first action feature of the first dimension; the denoised first action feature is decoded by a first decoder to obtain a first action sequence that matches the sample text.

[0152] The relevant contents in the above steps 820 and 830 can be referred to the above embodiments and will not be repeated here.

[0153] Step 840 : Generate a second action sequence matching the sample text according to the text features in a feature space of a second dimension using a second diffusion model, where the second dimension is greater than the first dimension.

[0154] In some embodiments, the second diffusion model includes a pre-trained second variational autoencoder and at least two second denoisers, the second variational autoencoder including a second encoder and a second decoder.

[0155] In some embodiments, a second action feature of a second random noise is generated through a second encoder, and the dimension of the second action feature is a second dimension; a second action sequence matching the sample text is generated based on the second action feature and the text feature through at least two second noise reducers and a second decoder.

[0156] In some embodiments, a second noise of a second dimension is added to the second action feature to obtain a noisy second action feature; the noisy second action feature is denoised in turn according to the text features through at least two second denoisers to obtain a denoised second action feature of the second dimension; the denoised second action feature is decoded through a second decoder to obtain a second action sequence that matches the sample text.

[0157] In some embodiments, the number of the second noise reducers is N, where N is an integer greater than or equal to 2. The second action feature after adding noise is subjected to T according to the text feature by the i-th second noise reducer. i The denoising is performed once to obtain the second action feature after denoising. i is a positive integer less than or equal to N, and the initial value of i is 1. i To be straight.

[0158] In some embodiments, if i is less than N, then let i=i+1, and perform T again on the second motion feature after adding noise through the i-th second noise reducer. i The step of performing denoising once and obtaining the i-th second motion feature after denoising is started; if i is equal to N, the i-th second motion feature after denoising is determined as the second motion feature after denoising.

[0159] For the relevant contents in the above step 840, please refer to the above embodiment corresponding to step 830, which will not be repeated here.

[0160] In the embodiment of the present application, the training method for the second diffusion model is the same as the training method for the first diffusion model. Both are based on the action text pairs in the training sample set and train the action features of random noise. The difference is that the first diffusion model models the diffusion process in the feature space of the first dimension, while the second diffusion model models the diffusion process in the feature space of the second dimension.

[0161] Step 850 , adjusting the parameters of the first diffusion model according to the first action sequence and the original action sequence, and adjusting the parameters of the second diffusion model according to the second action sequence and the original action sequence, to obtain a trained action generation model.

[0162] The motion generation model training method uses a first diffusion model and a second diffusion model trained separately, that is, the first diffusion model and the second diffusion model are trained as independent diffusion models. When generating motion sequences based on the motion generation model, the output data of the first diffusion model is used as the input data of the second diffusion model, linking the two models.

[0163] Therefore, step 850 needs to adjust the parameters of the first diffusion model and the second diffusion model respectively. After the adjustment is completed, the trained motion generation model can be obtained.

[0164] Since both the first and second variational autoencoders are pre-trained, adjusting the parameters of the first and second diffusion models can be translated into adjusting the parameters of the first denoiser and the at least two second denoisers. Furthermore, this can be translated into adjusting the parameters of the first and at least two second denoisers based on the noise added to the first and second variational autoencoders, respectively, and the noise in the motion sample pairs.

[0165] Exemplarily, the training process of the first diffusion model is taken as an example for description, and the training process of the second diffusion model can refer to the training process of the first diffusion model.

[0166] The first denoiser is responsible for modeling the backward diffusion process in the first-dimensional feature space. This process, through multiple iterations, gradually recovers the motion features in the first-dimensional feature space from the Gaussian noise. To train the first denoiser, the first motion features are gradually denoised using the forward diffusion process to obtain input data. This gradual denoising process conforms to the Markov chain properties and can be expressed using the following mathematical formula.

[0167] Where T represents the total number of noise additions in the forward diffusion process of the first diffusion model, z trepresents the first action feature after the t-th noise addition, β t is the noise weight hyperparameter related to the number of noise additions t, β t The value range is (0,1), i represents a vector of all 1s or a unit matrix, and the presentation form of I is related to the presentation form of the added noise. Indicates that the added noise is Gaussian distributed noise, q(z t |z t-1 ) represents the first action feature z after the t-1th noise addition t-1 Based on the t-th noise addition, the first action feature z is obtained t probability.

[0168] By using the re-parameterization technique, the noise z added in any number of noise additions t t , which can be sampled using a more concise representation.

[0169] in, α s =1-β s ,∈ t Represents the noise that conforms to the Gaussian distribution added during the t-th noise addition process.

[0170] Step 850 includes at least one sub-step from step 851 to step 853 (not shown in FIG. 8 ).

[0171] Step 851: Determine the predicted noise of the original action sequence according to the original action sequence.

[0172] Prediction noise refers to the noise to be added in the diffusion process of the original action sequence obtained based on the sample text according to the original action sequence.

[0173] Step 852: Calculate a first loss function value based on the first noise and the predicted noise, and calculate a second loss function value based on the second noise and the predicted noise.

[0174] The first loss function value represents the difference between the first noise and the predicted noise, and the second loss function value represents the difference between the second noise and the predicted noise.

[0175] Exemplarily, an MSE (Mean Squared Error) loss algorithm may be used to calculate a first loss function value according to the first noise and the predicted noise. The first loss function value may be expressed as the following formula.

[0176] Among them, ∈ represents the first noise, represents the prediction noise, τ θ(w) represents the text features obtained by the text encoder encoding the sample text. Represents the first loss function value.

[0177] In addition, to enable the first diffusion model to model the diffusion process in a higher-dimensional feature space, the calculation of the first loss function has been improved so that it imposes a greater penalty on model prediction errors in the early stages of denoising (when t is larger) and a smaller penalty on prediction errors in the later stages of denoising (when t is smaller). This can be specifically expressed as:

[0178] Among them, w1 and w2 are weight hyperparameters, w1 and w2 are positive numbers, which are used to convert the first loss function value The value range of is constrained within the threshold range. For example, the value range can be constrained between 0.5 and 5, and then w1 and w2 can be set to 4.5 and 0.5 respectively.

[0179] The calculation method of the second loss function value is the same as the calculation method of the first loss function value, which will not be repeated here.

[0180] Step 853: Adjust the parameters of the first denoiser according to the first loss function value to obtain a trained first diffusion model, and adjust the parameters of at least two second denoisers according to the second loss function value to obtain a trained second diffusion model.

[0181] For example, taking the training process of the first diffusion model as an example, according to the first loss function value The parameters of the first denoiser are adjusted to obtain a trained first diffusion model, and the parameters of at least two second denoisers are adjusted according to the second loss function value to obtain a trained second diffusion model, thereby obtaining a trained motion generation model.

[0182] FIG9 shows an algorithm flow chart of the process of using the motion generation model, first using the first noise reducer of the first diffusion model In the feature space of the first dimension, according to the text w, a complete T-step back diffusion process is performed to obtain the action features in the feature space of the first dimension Then, using the first decoder in the first diffusion model Action Features Restore to action sequence l , the action sequence s l This is the intermediate action sequence in the above embodiment.

[0183] The action sequence s output by the first diffusion model l As input data of the second diffusion model, the second encoder ε of the second diffusion model is usedh The action sequence s generated by the first diffusion model l Mapped to the second dimension feature space, and T h The forward diffusion noise adding process of the step is used to obtain the motion characteristics after noise addition. Then, use the denoiser with the second diffusion model and Complete the remaining reverse diffusion denoising process in sequence to obtain the action features in the second dimensional feature space Finally, the second decoder of the second diffusion model is used Action Features Restored to action sequence s, the action sequence s is the output action sequence in the above embodiment.

[0184] The technical solution provided by the embodiments of this application achieves good training results for both the first and second diffusion models by training them separately. This allows for combining diffusion models of different dimensions to create a multi-layered diffusion model, thereby generating action sequences that match the sample text. Furthermore, this avoids the problem of being unable to significantly adjust the parameters of the second diffusion model during joint training, thereby limiting the use of the second diffusion model to its sole use as the second diffusion model in the training model.

[0185] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0186] Please refer to Figure 10, which shows a block diagram of an action generation device based on an action generation model provided by an embodiment of the present application. The action generation model includes a text encoder, a first diffusion model and a second diffusion model. The device has the function of implementing the above-mentioned action generation method based on the action generation model, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be the computer device introduced above, or it can be set in a computer device. As shown in Figure 10, the device 1000 may include: a text acquisition module 1010, a text feature generation module 1020, an intermediate sequence generation module 1030 and an output sequence generation module 1040.

[0187] The text acquisition module 1010 is used to acquire text containing action information.

[0188] The text feature generation module 1020 is configured to generate text features of a text through a text encoder.

[0189] The intermediate sequence generation module 1030 is configured to generate an intermediate action sequence based on text features in a feature space of a first dimension using a first diffusion model.

[0190] The output sequence generation module 1040 is configured to perform detail enhancement processing on the intermediate action sequence in a second dimensional feature space using a second diffusion model to obtain an output action sequence matching the text, where the second dimension is greater than the first dimension.

[0191] In some embodiments, the first diffusion model includes a first variational autoencoder and a first denoiser, the first variational autoencoder includes a first encoder and a first decoder; the intermediate sequence generation module 1030 includes a random feature generation unit and an intermediate sequence generation unit.

[0192] The random feature generating unit is used to generate a random action feature of random noise through a first encoder, where the dimension of the random action feature is a first dimension.

[0193] The intermediate sequence generating unit is used to generate an intermediate action sequence according to the random action features and the text features through the first denoiser and the first decoder.

[0194] In some embodiments, the intermediate sequence generation unit is used to add a first noise of a first dimension to the random action feature to obtain a noisy random action feature; denoise the noisy random action feature according to the text feature through a first denoiser to obtain a denoised random action feature of the first dimension; and decode the denoised random action feature through a first decoder to obtain an intermediate action sequence.

[0195] In some embodiments, the second diffusion model includes a second variational autoencoder and at least two second denoisers, the second variational autoencoder includes a second encoder and a second decoder; the output sequence generation module 1040 includes an intermediate feature generation unit and an output sequence generation unit.

[0196] The intermediate feature generating unit is used to generate intermediate action features of the intermediate action sequence through the second encoder, where the dimension of the intermediate action features is the second dimension.

[0197] An output sequence generating unit is configured to generate an output action sequence matching the text according to the intermediate action features through at least two second noise reducers and a second decoder.

[0198] In some embodiments, the output sequence generation unit is used to add a second noise of a second dimension to the intermediate action feature to obtain a noisy intermediate action feature; denoise the noisy intermediate action feature in turn through at least two second denoisers to obtain a denoised intermediate action feature of the second dimension; and decode the denoised intermediate action feature through a second decoder to obtain an output action sequence that matches the text.

[0199] In some embodiments, the number of the second denoisers is N, where N is an integer greater than or equal to 2; the output sequence generating unit is configured to perform T on the noised intermediate action features through the i-th second denoiser. i The denoising is performed once, and the intermediate action features after denoising are obtained. i is a positive integer less than or equal to N, and the initial value of i is 1. i is a positive integer; if i is less than N, let i=i+1, and perform T again on the intermediate action features after adding noise through the i-th second denoiser. i The step of performing denoising once and obtaining the ith denoised intermediate motion feature is started; if i is equal to N, the ith denoised intermediate motion feature is determined as the denoised intermediate motion feature.

[0200] In the technical solution provided by the embodiment of the present application, the first diffusion model can process the features under the feature space of the first dimension, and the second diffusion model can process the features under the feature space of the second dimension. Since the second dimension is larger than the first dimension, the second diffusion model can pay more attention to fine-grained features. Therefore, the intermediate action sequence is generated by the first diffusion model in the feature space of the first dimension, and the modeling diffusion of the text in the feature space of the low dimension is preliminarily realized. Since the details of the action sequence obtained by diffusion in the feature space of the low dimension are not rich enough, the intermediate action sequence is further enhanced by the second diffusion model in the feature space of the second dimension to obtain the output action sequence. Since the action sequence is refined in the feature space of the high dimension, the details of the output action sequence are richer and more matched with the text. Compared with the problem of insufficient details in the generated action sequence caused by only modeling the text diffusion process in the feature space of the low dimension in the related art, the technical solution provided by the present application enhances the detail features in the output action sequence in the feature space of the high dimension, thereby improving the richness of details in the output action sequence.

[0201] Please refer to Figure 11, which shows a block diagram of another training device for an action generation model provided in an embodiment of the present application. The action generation model includes a text encoder, a first diffusion model and a second diffusion model. The device has the function of implementing the above-mentioned action generation method based on the action generation model, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be the computer device introduced above, or it can be set in a computer device. As shown in Figure 11, the device 1100 may include: a sample set acquisition module 1110, a text feature generation module 1120, a first sequence generation module 1130, a second sequence generation module 1140 and a parameter adjustment module 1150.

[0202] The sample set acquisition module 1110 is used to acquire a training sample set for the action generation model. The training sample set includes at least one action text pair. Each action text pair includes a sample text and an original action sequence with a matching relationship.

[0203] The text feature generation module 1120 is configured to generate text features of a sample text through a text encoder.

[0204] The first sequence generating module 1130 is configured to generate a first action sequence matching the sample text according to text features in a feature space of a first dimension using a first diffusion model.

[0205] The second sequence generating module 1140 is configured to generate a second action sequence matching the sample text according to text features in a feature space of a second dimension using a second diffusion model, where the second dimension is greater than the first dimension.

[0206] The parameter adjustment module 1150 is used to adjust the parameters of the first diffusion model according to the first action sequence and the original action sequence, and to adjust the parameters of the second diffusion model according to the second action sequence and the original action sequence to obtain a trained action generation model.

[0207] In some embodiments, the first diffusion model includes a pre-trained first variational autoencoder and a first denoiser, the first variational autoencoder including a first encoder and a first decoder; the first sequence generation module 1130 includes a first feature generation unit and a first sequence generation unit.

[0208] The first feature generating unit is configured to generate a first motion feature of the first random noise through a first encoder, where the dimension of the first motion feature is a first dimension.

[0209] The first sequence generating unit is configured to generate a first action sequence matching the sample text according to the first action feature and the text feature through a first noise reducer and a first decoder.

[0210] In some embodiments, the first sequence generation unit is used to add first noise of a first dimension to the first action feature to obtain the noisy first action feature; denoise the noisy first action feature according to the text feature through the first denoiser to obtain the denoised first action feature of the first dimension; and decode the denoised first action feature through the first decoder to obtain a first action sequence that matches the sample text.

[0211] In some embodiments, the second diffusion model includes a pre-trained second variational autoencoder and at least two second denoisers, the second variational autoencoder includes a second encoder and a second decoder; the second sequence generation module 1140 includes a second feature generation unit and a second sequence generation unit.

[0212] The second feature generating unit is used to generate a second motion feature of the second random noise through a second encoder, where the dimension of the second motion feature is a second dimension.

[0213] The second sequence generating unit is configured to generate a second action sequence matching the sample text according to the second action feature and the text feature through at least two second noise reducers and a second decoder.

[0214] In some embodiments, the second sequence generation unit is used to add second noise of a second dimension to the second action feature to obtain a noisy second action feature; denoise the noisy second action feature according to the text features in turn through at least two second denoisers to obtain a denoised second action feature of the second dimension; and decode the denoised second action feature through a second decoder to obtain a second action sequence that matches the sample text.

[0215] In some embodiments, the number of the second noise reducers is N, where N is an integer greater than or equal to 2; the second sequence generating unit is configured to perform T on the noised second motion feature through the i-th second noise reducer. i The denoising is performed once to obtain the second action feature after denoising. i is a positive integer less than or equal to N, and the initial value of i is 1. i is a positive integer; if i is less than N, let i=i+1, and perform T again on the second action feature after adding noise through the i-th second denoiser. i The step of performing denoising once and obtaining the i-th second motion feature after denoising is started; if i is equal to N, the i-th second motion feature after denoising is determined as the second motion feature after denoising.

[0216] In some embodiments, the parameter adjustment module 1150 is used to determine the predicted noise of the original action sequence based on the original action sequence; calculate a first loss function value based on the first noise and the predicted noise, and calculate a second loss function value based on the second noise and the predicted noise; adjust the parameters of the first denoiser according to the first loss function value to obtain a trained first diffusion model, and adjust the parameters of at least two second denoisers according to the second loss function value to obtain a trained second diffusion model.

[0217] The technical solution provided by the embodiments of this application achieves good training results for both the first and second diffusion models by training them separately. This allows for combining diffusion models of different dimensions to create a multi-layered diffusion model, thereby generating action sequences that match the sample text. Furthermore, this avoids the problem of being unable to significantly adjust the parameters of the second diffusion model during joint training, thereby limiting the use of the second diffusion model to its sole use as the second diffusion model in the training model.

[0218] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0219] Please refer to Figure 12, which shows a block diagram of a computer device 1200 provided in one embodiment of the present application. The computer device 1200 can be any electronic device with data computing, processing, and storage capabilities. The computer device 1200 can be used to implement the action generation method based on the action generation model or the action generation model training method provided in the above embodiments.

[0220] Typically, the computer device 1200 includes a processor 1201 and a memory 1202 .

[0221] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI processor for processing computing operations related to machine learning.

[0222] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned action generation method based on the action generation model, or the training method of the action generation model.

[0223] Those skilled in the art will appreciate that the structure shown in FIG12 does not limit the computer device 1200 , and may include more or fewer components than shown, or combine certain components, or adopt a different component arrangement.

[0224] In an illustrative embodiment, a computer-readable storage medium is also provided, storing a computer program that, when executed by a processor of a computer device, implements the aforementioned method for generating an action generation model based on an action generation model, or the method for training an action generation model. Optionally, the computer-readable storage medium may be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, or an optical data storage device.

[0225] In an exemplary embodiment, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the above-described method for generating an action based on an action generation model or the method for training an action generation model.

[0226] It should be noted that this application can display a prompt interface, pop-up window or output voice prompt information before collecting the user's relevant data and during the process of collecting the user's relevant data. The prompt interface, pop-up window or voice prompt information is used to remind the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining the user's relevant data after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining the user's relevant data are terminated, that is, the user's relevant data is not obtained. In other words, all user data collected by this application are processed strictly in accordance with the requirements of relevant national laws and regulations. The informed consent or separate consent of the personal information subject is obtained only when the user agrees and authorizes it to collect the data. Subsequent data use and processing are carried out within the scope of authorization of laws and regulations and the personal information subject, and the collection, use and processing of relevant user data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0227] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0228] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An action generation method based on an action generation model, executed by a computer device, wherein the action generation model includes a text encoder, a first diffusion model and a second diffusion model, and the method includes: Get the text containing the action information; Generating text features of the text by the text encoder; Generate an intermediate action sequence according to the text features in a feature space of a first dimension using the first diffusion model; The intermediate action sequence is subjected to detail enhancement processing in a feature space of a second dimension through the second diffusion model to obtain an output action sequence matching the text, wherein the second dimension is greater than the first dimension.

2. The method according to claim 1, wherein: The first diffusion model includes a first variational autoencoder and a first denoiser, and the first variational autoencoder includes a first encoder and a first decoder; The step of generating an intermediate action sequence according to the text features in a feature space of a first dimension by using the first diffusion model includes: Generate, by means of the first encoder, a random action feature of random noise, wherein the dimension of the random action feature is the first dimension; The intermediate action sequence is generated according to the random action features and the text features through the first noise reducer and the first decoder.

3. The method according to claim 1 or 2, wherein: The step of generating the intermediate action sequence according to the random action feature and the text feature by the first noise reducer and the first decoder comprises: Adding first noise of the first dimension to the random action feature to obtain a noisy random action feature; De-noising the random action features after adding noise according to the text features by the first denoiser to obtain the random action features after adding noise in the first dimension; The denoised random action features are decoded by the first decoder to obtain the intermediate action sequence.

4. The method according to any one of claims 1 to 3, wherein: The second diffusion model includes a second variational autoencoder and at least two second denoisers, the second variational autoencoder including a second encoder and a second decoder; The step of performing detail enhancement processing on the intermediate action sequence in the feature space of the second dimension by the second diffusion model to obtain an output action sequence matching the text includes: Generate an intermediate action feature of the intermediate action sequence through the second encoder, where the dimension of the intermediate action feature is the second dimension; An output action sequence matching the text is generated according to the intermediate action features through the at least two second noise reducers and the second decoder.

5. The method according to any one of claims 1 to 4, wherein: The step of generating an output action sequence matching the text according to the intermediate action features by the at least two second noise reducers and the second decoder comprises: adding the second noise of the second dimension to the intermediate action feature to obtain the noisy intermediate action feature; De-noising the noisy intermediate action features in sequence by the at least two second denoisers to obtain the de-noised intermediate action features of the second dimension; The denoised intermediate action features are decoded by the second decoder to obtain an output action sequence matching the text.

6. The method according to any one of claims 1 to 5, wherein: The number of the second noise reducers is N, where N is an integer greater than or equal to 2; The step of sequentially denoising the noisy intermediate action features by the at least two second denoisers to obtain the denoised intermediate action features of the second dimension includes: The denoised intermediate action features are subjected to T by the i-th second denoiser i The denoising is performed once to obtain the intermediate action feature after the ith denoising, where i is a positive integer less than or equal to N, and the initial value of i is 1. i is a positive integer; If i is less than N, let i=i+1, and perform T again on the noise-added intermediate action features through the i-th second denoiser. i After the denoising is performed, the step of obtaining the intermediate action features after the i-th denoising is started; If i is equal to N, the i-th denoised intermediate action feature is determined as the denoised intermediate action feature.

7. A method for training an action generation model, executed by a computer device, wherein the action generation model includes a text encoder, a first diffusion model, and a second diffusion model, and the method includes: Acquire a training sample set of the action generation model, wherein the training sample set includes at least one action text pair, and each of the action text pairs includes a sample text and an original action sequence having a matching relationship; Generating text features of the sample text by the text encoder; Generate a first action sequence matching the sample text according to the text features in a feature space of a first dimension using the first diffusion model; Generate a match with the sample text according to the text features in a feature space of a second dimension through the second diffusion model, where the second dimension is greater than the first dimension; According to the first action sequence and the original action sequence, the parameters of the first diffusion model are adjusted, and according to the second action sequence and the original action sequence, the parameters of the second diffusion model are adjusted to obtain a trained action generation model.

8. The method according to claim 7, wherein: The first diffusion model includes a pre-trained first variational autoencoder and a first denoiser, wherein the first variational autoencoder includes a first encoder and a first decoder; The step of generating a first action sequence matching the sample text according to the text features in a feature space of the first dimension by using the first diffusion model includes: Generate, by means of the first encoder, a first action feature of a first random noise, wherein the dimension of the first action feature is the first dimension; A first action sequence matching the sample text is generated according to the first action feature and the text feature through the first noise reducer and the first decoder.

9. The method according to claim 7 or 8, wherein: The step of generating a first action sequence matching the sample text according to the first action feature and the text feature by the first noise reducer and the first decoder includes: adding first noise of the first dimension to the first action feature to obtain a noisy first action feature; De-noising the first action feature after adding noise according to the text feature by the first noise reducer to obtain the first action feature after de-noising of the first dimension; The first action feature after denoising is decoded by the first decoder to obtain a first action sequence matching the sample text.

10. The method according to any one of claims 7 to 9, wherein: The second diffusion model includes a pre-trained second variational autoencoder and at least two second denoisers, the second variational autoencoder including a second encoder and a second decoder; The step of generating a second action sequence matching the sample text according to the text features in the feature space of the second dimension by the second diffusion model includes: Generate, by the second encoder, a second action feature of a second random noise, wherein the dimension of the second action feature is the second dimension; By means of the at least two second noise reducers and the second decoder, according to the second action feature and the text feature The feature generates a second action sequence matching the sample text.

11. The method according to any one of claims 7 to 10, wherein: The step of generating a second action sequence matching the sample text according to the second action feature and the text feature by the at least two second noise reducers and the second decoder comprises: adding a second noise of the second dimension to the second action feature to obtain a noisy second action feature; De-noising the second action features after adding noise according to the text features in turn by the at least two second noise reducers to obtain the second action features after adding noise in the second dimension; The denoised second action feature is decoded by the second decoder to obtain a second action sequence matching the sample text.

12. The method according to any one of claims 7 to 11, wherein: The number of the second noise reducers is N, where N is an integer greater than or equal to 2; The step of denoising the second action features after adding noise in sequence according to the text features by the at least two second denoisers to obtain the second action features after denoising in the second dimension includes: The second action feature after the noise is T-processed according to the text feature by the i-th second noise reducer i The denoising is performed once to obtain the second action feature after the ith denoising, where i is a positive integer less than or equal to N, and the initial value of i is 1. i is a positive integer; If i is less than N, then let i=i+1, and perform T again on the second action feature after adding noise through the i-th second noise reducer. i After the denoising is performed, the step of obtaining the second action feature after the i-th denoising is started; If i is equal to N, the i-th second motion feature after denoising is determined as the second motion feature after denoising.

13. The method according to any one of claims 7 to 12, wherein: The adjusting the parameters of the first diffusion model according to the first action sequence and the original action sequence, and adjusting the parameters of the second diffusion model according to the second action sequence and the original action sequence to obtain a trained action generation model includes: Determining a predicted noise of the original action sequence according to the original action sequence; Calculating a first loss function value according to the first noise and the predicted noise, and calculating a second loss function value according to the second noise and the predicted noise; The parameters of the first denoiser are adjusted according to the first loss function value to obtain a trained first diffusion model, and the parameters of the at least two second denoisers are adjusted according to the second loss function value to obtain the trained second diffusion model.

14. An action generation device based on an action generation model, the action generation model comprising a text encoder, a first diffusion model and a second diffusion model, the device comprising: A text acquisition module, used to acquire text containing action information; A text feature generation module, used for generating text features of the text through the text encoder; An intermediate sequence generation module, configured to generate an intermediate action sequence according to the text features in a feature space of a first dimension by using the first diffusion model; The output sequence generation module is used to perform detail enhancement processing on the intermediate action sequence in a feature space of a second dimension through the second diffusion model to obtain an output action sequence matching the text, wherein the second dimension is greater than the first dimension.

15. A training device for an action generation model, the action generation model comprising a text encoder, a first diffusion model and a second diffusion model, the device comprising: A sample set acquisition module, used to acquire a training sample set of the action generation model, wherein the training sample set includes at least one action text pair, each of which includes a sample text and an original action sequence having a matching relationship; A text feature generation module, used for generating text features of the sample text through the text encoder; A first sequence generation module, configured to generate a first action sequence matching the sample text according to the text features in a feature space of a first dimension using the first diffusion model; A second sequence generation module, configured to generate a second action sequence matching the sample text according to the text features in a feature space of a second dimension through the second diffusion model, wherein the second dimension is greater than the first dimension; A parameter adjustment module is used to adjust the parameters of the first diffusion model according to the first action sequence and the original action sequence, and to adjust the parameters of the second diffusion model according to the second action sequence and the original action sequence to obtain a trained action generation model.

16. A computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the action generation method based on the action generation model as described in any one of claims 1 to 6, or to implement the action generation model training method as described in any one of claims 7 to 13.

17. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program is loaded and executed by a processor to implement the action generation method based on the action generation model as described in any one of claims 1 to 6, or to implement the action generation model training method as described in any one of claims 7 to 13.

18. A computer program product, comprising a computer program, wherein the computer program is loaded and executed by a processor to implement the action generation method based on the action generation model as described in any one of claims 1 to 6, or to implement the training method of the action generation model as described in any one of claims 7 to 13.

Images