3D human body action generation method and device, storage medium, electronic equipment and product

Abstract

The invention relates to the technical field of 3D action generation, and particularly provides a 3D human body action generation method and device, a storage medium, electronic equipment and a product, and the method can comprise the steps: obtaining a text feature vector corresponding to the text description of a human body action; layering and quantifying the text feature vector to obtain a basic action mark sequence and a residual action mark; acquiring a basic action sequence corresponding to the basic action mark sequence, and acquiring a residual action sequence corresponding to the residual action mark; and generating a 3D human body action sequence based on the basic action sequence and the residual action sequence. Some embodiments of the present application can improve the accuracy of 3D human motion generation.

Description

Technical Field

[0001] The present application relates to the technical field of 3D motion generation, and in particular to a method, device, storage medium, electronic device and product for generating 3D human motion. Background Art

[0002] In the field of text-driven 3D human motion generation, existing technologies mainly rely on autoregressive models and vector quantization models (VQ) to convert human motion sequences into discrete motion marker sequences, and generate subsequent markers from the previous markers through recursion or step-by-step generation to finally generate the entire motion sequence. However, due to the step-by-step generation method, each motion marker depends on the previous motion marker, and the generation process requires multiple iterations, which has a high time complexity and leads to low generation efficiency.

[0003] Therefore, how to provide a technical solution for an efficient method for generating 3D human motion has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of some embodiments of the present application is to provide a method, device, storage medium, electronic device and product for generating 3D human motion. Through the technical solutions of the embodiments of the present application, the efficiency of generating 3D human motion can be improved, the detailed motion can be accurately restored, and the motion generation quality can be high.

[0005] In a first aspect, some embodiments of the present application provide a method for generating 3D human motion, comprising: obtaining a text feature vector corresponding to a text description of a human motion; performing hierarchical quantization on the text feature vector to obtain a basic motion marker sequence and a residual motion marker; obtaining a basic motion sequence corresponding to the basic motion marker sequence, and obtaining a residual motion sequence corresponding to the residual motion marker; and generating a 3D human motion sequence based on the basic motion sequence and the residual motion sequence.

[0006] Some embodiments of the present application obtain a basic action tag sequence and a residual action tag by hierarchically quantizing the text feature vectors corresponding to the human body actions, and then obtain the basic action sequence and residual action sequence corresponding to the two, and finally generate a 3D human body action sequence through these two sequences. The embodiments of the present application can achieve accurate restoration of action details and improve generation quality by quantizing errors, while reducing time complexity and improving action generation efficiency.

[0007] In some embodiments, obtaining a basic action sequence corresponding to the basic action marker sequence includes: predicting the basic action marker sequence using a generative mask Transformer model to obtain the basic action sequence; obtaining a residual action sequence corresponding to the residual action marker includes: processing the residual action marker using a residual Transformer model to obtain the residual action sequence.

[0008] Some embodiments of the present application generate a mask Transformer model and a residual Transformer model to process the basic action mark sequence and the residual action mark respectively, so as to obtain the basic action sequence and the residual action sequence, thereby providing data support for accurately restoring human body movements.

[0009] In some embodiments, the generative mask Transformer model is obtained by training an initial mask Transformer model with a training data set consisting of an action text description, an action initial tag sequence, and an initial tag mask; the residual Transformer model is obtained by training an initial Transformer model with a training data set consisting of an action tag sequence and a corresponding action text feature vector.

[0010] Some embodiments of the present application obtain a generated mask Transformer model and a residual Transformer model through model training to achieve efficient and accurate processing of data.

[0011] In some embodiments, the hierarchical quantization of the text feature vector to obtain a basic action marker sequence and a residual action marker includes: performing base-layer quantization on the text feature vector to generate the basic action marker sequence; calculating the basic action marker sequence and the text feature vector to obtain a quantization error; and performing hierarchical quantization on the quantization error to generate multiple layers of the residual action marker.

[0012] Some embodiments of the present application perform base-layer quantization on the text feature vector, determine the quantization error through comparative calculation, and then perform hierarchical quantization on the quantization error to obtain a residual action marker, which can prevent information loss and facilitate subsequent accurate restoration of detailed actions.

[0013] In some embodiments, generating a 3D human motion sequence based on the basic motion sequence and the residual motion sequence includes: inputting the basic motion sequence and the residual motion sequence into a decoder, and outputting the 3D human motion sequence.

[0014] Some embodiments of the present application determine a 3D human motion sequence through a basic motion sequence and a residual motion sequence, thereby achieving accurate restoration of human motions and achieving high motion generation quality.

[0015] In a second aspect, some embodiments of the present application provide a device for generating 3D human motion, comprising: an acquisition module for acquiring a text feature vector corresponding to a text description of a human motion; a quantization module for hierarchically quantizing the text feature vector to obtain a basic motion marker sequence and a residual motion marker; a processing module for acquiring a basic motion sequence corresponding to the basic motion marker sequence, and acquiring a residual motion sequence corresponding to the residual action marker; and a generation module for generating a 3D human motion sequence based on the basic motion sequence and the residual motion sequence.

[0016] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0017] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement a method as described in any embodiment of the first aspect when executing the program.

[0018] In a fifth aspect, some embodiments of the present application provide a computer program product, wherein the computer program product comprises a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 System diagram for 3D human motion generation provided for some embodiments of the present application;

[0021] Figure 2 One of the flow charts of the method for generating 3D human motion provided in some embodiments of the present application;

[0022] Figure 3 A second flow chart of a method for generating 3D human motion provided in some embodiments of the present application;

[0023] Figure 4 A block diagram of a device for generating 3D human motion provided in some embodiments of the present application;

[0024] Figure 5 A schematic diagram of an electronic device is provided for some embodiments of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0027] In the related art, in the process of text-driven 3D human motion generation, the motion sequence is first converted into discrete tags by a vector quantization method, and then the generation model is used to gradually generate the action from front to back. In the process of gradually generating the entire action sequence, each tag depends on the previously generated tag. However, in the process of vector quantization in the prior art, the original action is converted into discrete tags, which easily leads to information loss, the quality of the generated action is limited, and the detailed action cannot be accurately restored. The action sequence is generated in a unidirectional (i.e., from front to back) manner through an automatic regression model. If there is an error in a certain step, the error of a certain step will be accumulated and propagated to the subsequent generation process, resulting in a gradual decrease in the generation quality, especially in the generation of long sequences. The problem is more obvious. Due to the use of a step-by-step generation method, each tag depends on the previous tag, the generation process requires multiple iterations, the time complexity is high, and the generation efficiency is low. Moreover, due to the limitations of the unidirectional generation model, the generated action is not sufficiently correlated in the global context, showing a weaker expression ability, especially when complex actions and long action sequences are generated, it is difficult to accurately capture details.

[0028] In view of this, some embodiments of the present application provide a method for generating 3D human motion, which obtains a basic motion sequence and a residual motion sequence by hierarchically quantizing and processing the text feature vectors corresponding to the text description of the human motion. A 3D human motion sequence is generated based on the two. The embodiments of the present application overcome the problem of loss of motion details in the traditional vector quantization process through residual vector quantization, and the generated motion is more accurate, especially when fine motion generation is required.

[0029] The following is combined with Figure 1The overall composition structure of the system for generating 3D human motion provided by some embodiments of the present application is exemplified.

[0030] like Figure 1 As shown, some embodiments of the present application provide a system for generating 3D human motion, which may include a terminal 100 and a server 200. Among them, the terminal 100 can send a text description of the human motion to the server 200. After receiving the text description, the server 200 converts it into a corresponding text feature vector. After that, the server 200 performs multi-level hierarchical quantization on the text feature vector to obtain a basic action tag sequence and a residual action tag. Then the server 200 processes the basic action tag sequence and the residual action tag based on the internally pre-deployed generated mask Transform model and the residual Transform model, respectively, to obtain a basic action sequence and a residual action series. Finally, a 3D human motion sequence is generated based on the two. The present application adopts a generated mask Transformer to predict the mask tags of the entire basic action tag sequence in parallel, and generates a basic action sequence through global context information, thereby avoiding the gradual accumulation of errors. This not only improves the generation quality, but also significantly improves the generation efficiency, so that complex actions can be efficiently generated in a small number of iterations.

[0031] In some embodiments of the present application, the terminal 100 may be a mobile terminal or a non-portable computer terminal, which is not specifically limited in the embodiments of the present application.

[0032] The following is combined with Figure 2 The implementation process of 3D human motion generation performed by the server 200 provided in some embodiments of the present application is exemplified.

[0033] Please see attached Figure 2 , Figure 2 A flow chart of a method for generating 3D human motion is provided for some embodiments of the present application. The method for generating 3D human motion may include:

[0034] S210, obtaining a text feature vector corresponding to the text description of the human body action.

[0035] For example, in some embodiments of the present application, a user can input a text description of a 3D human action in the terminal 100 through natural language text. For example, "a person walks forward and then turns around". The input text description is processed by the text encoding model to generate a text feature vector, which is used as the initial condition for the subsequent generation of the 3D human action. The text feature vector contains the semantic information of the action.

[0036] For example, there is a simple 3D human action description, represented as the change of joint position over time, for example, "a person walks forward two steps, turns around and walks backward". This action can be represented by a vector of the position of the joint at each time step. Assume that the time step of the action sequence is N=4, that is, 4 time frames, and the data dimension of each frame (that is, the position information of the joint) is D=3 (XYZ coordinates of each joint). First, use a 1D convolutional encoder to compress the original action sequence m1:4 into a low-dimensional latent vector representation b1:2. Assume that the time step N=4 is downsampled to n=2, and the dimension D=3 is compressed to d=2. After the 1D convolutional encoder, the latent vector b1:2 is generated (as a specific example of a text feature vector).

[0037] S220, performing hierarchical quantization on the text feature vector to obtain a basic action label sequence and a residual action label.

[0038] For example, in some embodiments of the present application, in order to prevent the loss of action information, after the base layer quantization of the text feature vector, it is also necessary to quantize the residual existing in the base layer quantization to obtain a basic action marker sequence and a residual action sequence, so that the action details can be accurately restored later.

[0039] In some embodiments of the present application, S220 may include: performing base-layer quantization on the text feature vector to generate the basic action marker sequence; calculating the basic action marker sequence and the text feature vector to obtain a quantization error; and performing hierarchical quantization on the quantization error to generate multiple layers of residual action markers.

[0040] For example, in some embodiments of the present application, the basic part of the action sequence in the text feature vector is first quantized to generate a basic action tag sequence, and then the quantization error part is further hierarchically quantized to generate multi-layer residual tags to retain the details of the action and reduce the quantization error.

[0041] Specifically, 3D human motion data is usually represented as the skeletal joint position information at each time step, and the skeletal state at each time step can be represented as a vector. This latent vector (that is, the text feature vector) is quantized into discrete tags through a preset code book, and the quantization result of the base layer is a discrete tag sequence (that is, the basic action tag sequence).

[0042] For example, the above-mentioned latent vector b1:2 is vector quantized. The present application provides a pre-trained code book, which contains discrete code vectors. By comparing the distance between each vector in b1:2 and the vector in the code book, the nearest code vector is selected for replacement. For example, b1=(0.9, 0.8) is closest to the code vector c1=(1.0, 1.0), and b2=(1.1, -0.9) is closest to the code vector c2=(1.0, -1.0), so that the basic action marker sequence t0 can be obtained. Although the basic action marker sequence t0 represents most of the information of the original action, there is still a quantization error, that is, the difference between the original latent vector and the quantized vector t0, and this part of the difference is the residual (that is, the quantization error).

[0043] The residual layer quantization is mainly to reduce the error layer by layer by capturing the details that cannot be expressed in the quantization of the basic part. After the basic action mark sequence is generated, the quantization error is calculated, that is, the residual. The first layer quantizes the residual again, similar to the quantization method of the base layer, and quantizes the residual vector through another code book. Continue to quantize through the code book of the second layer to generate the second layer of residual marks to capture more subtle action details. And so on, for high-order residual quantization. This process can be recursively performed multiple times, and each quantization will further reduce the residual of the previous layer until the expected accuracy is achieved. For example, a residual code r0 is used to quantize the first layer of residuals, that is, each vector of r0 is quantized to obtain a residual mark t1. If there are still some residuals that are not fully quantized after the first layer of residual quantization, higher-order residuals can be calculated (such as obtaining residual marks t2, etc.) until the residual is small enough and basically no information is lost. The number of residual quantizations can be selected according to actual needs, and the embodiments of the present application are not specifically limited here.

[0044] S230: Acquire a basic action sequence corresponding to the basic action label sequence, and acquire a residual action sequence corresponding to the residual action label.

[0045] In some embodiments of the present application, S230 may include: predicting the basic action label sequence using a generative mask Transformer model to obtain the basic action sequence.

[0046] For example, in some embodiments of the present application, after the above quantization, the generated mask Transformer (that is, the generated mask Transformer model) predicts the basic action sequence corresponding to the basic action mark in parallel through a random mask mechanism. In the training process, the generated mask Transformer model randomly masks a part of the action mark and uses context information to predict the mark of the mask position. When generating the action sequence, it starts from the fully masked sequence, gradually fills in the high-confidence marks, and finally generates a complete basic action sequence t0'.

[0047] In some embodiments of the present application, S230 may include: processing the residual action label using a residual Transformer model to obtain the residual action sequence.

[0048] For example, in some embodiments of the present application, after generating the basic action sequence, the residual transformer generates subsequent residual action sequences layer by layer. The residual transformer gradually predicts and generates detailed action tags based on the generated basic layer tags, and obtains residual action sequences t1' and t2', further improving the accuracy and detail of the generated actions.

[0049] Specifically, the main task of the generative masked transformer is to generate the basic action sequence, that is, the discrete tags of the initial layer (base layer) of the action sequence. The model uses a masking mechanism to generate action sequences from text descriptions in parallel, avoiding the cumulative error problem in traditional autoregressive models. The generative masked transformer generates the entire basic action sequence by gradually filling in the mask. During the generation process, it predicts the masked tags with reference to the text input and context information. Finally, it generates the base layer of the 3D action, which represents most of the global information of the action. The role of the residual transformer is to process the details in the action sequence, that is, the residual part that the base layer tags cannot fully capture. It generates a sequence of residual tags layer by layer to ensure that the details are gradually restored, so that the final action sequence has high accuracy. After the generative masked transformer generates the basic action sequence, the residual transformer generates higher-level residual tags layer by layer based on the base layer. The residual transformer predicts the residual tags of the next layer based on the output of each layer and the residual tags of the previous layer to ensure that the details of each layer can be gradually restored. In this way, through multi-level refinement, the final generated 3D action sequence includes not only global basic actions, but also rich local details.

[0050] That is to say, the process of action sequence generation mainly includes two steps:

[0051] Step 1: The generative mask Transformer first generates a basic action sequence to capture the main features of 3D human body movements. Step 2: After the basic action sequence is generated, the residual Transformer starts working. It starts from the basic action sequence and generates residual action sequences layer by layer, gradually supplementing the details that are not captured in the basic actions. That is, the generative mask Transformer is responsible for generating the overall basic action sequence, which is the "skeleton" of the action sequence. The residual Transformer is responsible for filling in the details in the "skeleton", that is, generating the residual of each layer. In this way, the generative mask Transformer and the residual Transformer work closely together to ensure that the generated 3D human body action sequence has both global information and rich details.

[0052] S240: Generate a 3D human motion sequence based on the basic motion sequence and the residual motion sequence.

[0053] For example, in some embodiments of the present application, the complete latent vector representation is restored by summing up the above-obtained t0', t1' and t2' layer by layer. The restored latent vector will be close to the original compressed vector b1:2, and finally a high-precision 3D human action sequence can be generated.

[0054] In some embodiments of the present application, S240 may include: inputting the basic motion sequence and the residual motion sequence into a decoder, and outputting the 3D human body motion sequence.

[0055] For example, in some embodiments of the present application, t0', t1' and t2' are decoded back to the original 3D human motion data (that is, 3D human motion sequence) through a decoder. In other words, the action sequence generated by the generative mask Transformer and the residual Transformer will be converted back to continuous 3D human motion data through a decoder. The action markers generated by the generative mask Transformer and the residual Transformer are discrete, representing the changes in the 3D action in the time step. Through the decoder, the discrete action markers are converted into continuous 3D joint position information, which can capture global and local detail information.

[0056] It can be seen that the residual Transformer in the above embodiment collaborates with the generative mask Transformer in order to generate both the basic action sequence and the detailed information contained in each layer of residual in the process of generating 3D human body motion. The collaboration between the two plays a vital role in the generation process of the model. In the process of 3D human body motion generation, the generation of action sequences requires layered processing. The generative mask Transformer and the residual Transformer are responsible for different parts respectively, and their collaboration ensures the global generation and local refined generation of actions.

[0057] In some embodiments of the present application, the generative mask Transformer model is obtained by training the initial mask Transformer model with a training data set consisting of an action text description, an action initial tag sequence, and an initial tag mask; the residual Transformer model is obtained by training the initial Transformer model with a training data set consisting of an action tag sequence and a corresponding action text feature vector.

[0058] The following is an example of the process of obtaining the generated mask Transformer and the residual Transformer.

[0059] The generated mask Transformer is obtained through the training process. The specific steps are as follows:

[0060] Input data: The input data is a text description (encoded as a text feature vector) and an initial tag sequence of the action sequence. The initial tag sequence is usually completely masked (i.e., the action sequence is empty at the beginning and all positions need to be predicted).

[0061] Training goal: The training goal is to use the masking mechanism to allow the model to gradually predict the masked action tags and restore the complete action sequence as much as possible. During training, the mask ratio is dynamically adjusted so that the model can learn how to generate a complete sequence under different masking conditions.

[0062] Masking strategy: The generative masked Transformer adopts the dynamic masking strategy in BERT. That is, during training, some tokens of the input sequence are randomly masked, and then the masked parts are predicted based on context and text information. In this way, the model can use both previous and next context information, rather than relying only on previous data like traditional autoregressive models.

[0063] Iterative filling: When actually generating an action sequence, the model starts from a fully masked sequence and gradually fills in high-confidence markers until the complete underlying action sequence is generated.

[0064] The residual Transformer is also obtained through a special training process. The specific steps are as follows:

[0065] Input data: The input data is the action tag sequence of the previous layer (such as basic action tags) and the text feature vector (embedded vector of text description).

[0066] Training objective: The purpose of training the residual transformer is to generate a residual action sequence for the corresponding layer. During training, the residual transformer learns how to generate residual tags that capture details from the tags generated by the previous layer. The generation of each layer of residual is based on the input of the current layer and the output of the previous layer.

[0067] Model structure: The structure of the residual Transformer is similar to that of the generative mask Transformer, using the Transformer architecture to handle local and global dependencies in time series. The difference is that the residual Transformer specializes in processing the residual part and learning detailed information.

[0068] It should be understood that the embodiment of the present application ensures the accuracy and precision of 3D human motion generation by generating the collaboration of the mask Transformer and the residual Transformer, and generating the basic actions and residual action details in layers. The mask Transformer is generated first to generate the basic action framework, and the residual Transformer fills in the details layer by layer, and finally outputs the complete 3D human motion through the decoder. Both of them are trained with a large amount of action data to ensure that they can cope with complex text descriptions and action generation tasks. The specific training process can be adaptively adjusted based on the actual application scenario, and the embodiment of the present application is not limited to this.

[0069] The following is combined with Figure 3 The specific process of generating 3D human motion provided by some embodiments of the present application is exemplified.

[0070] Please see attached Figure 3 , Figure 3 A flow chart of a method for generating 3D human motion is provided for some embodiments of the present application.

[0071] The above process is explained below as an example.

[0072] S310: Input a text description of 3D human body movements.

[0073] S320, generating a text feature vector corresponding to the text description.

[0074] S330, performing base-level quantization on the text feature vector to generate a base action tag sequence.

[0075] S340, calculating the basic action tag sequence and the text feature vector to obtain a quantization error.

[0076] S350, hierarchical quantization is performed on the quantization error to generate multiple layers of residual action labels.

[0077] S360, using the generative mask Transformer model to predict the basic action tag sequence to obtain the basic action sequence.

[0078] S370, using the residual Transformer model and the basic action sequence to process the residual action label to obtain a residual action sequence.

[0079] S380: Input the basic motion sequence and the residual motion sequence into a decoder, and output a 3D human motion sequence.

[0080] It should be noted that the specific implementation process of S310 to S380 can refer to the method embodiment provided above, and in order to avoid repetition, the detailed description is appropriately omitted here.

[0081] Through some of the above embodiments of the present application, it can be seen that the present application can achieve the following technical effects:

[0082] 1) Reduced quantization error and enhanced motion details: Through residual vector quantization, the problem of motion detail loss in the traditional vector quantization process is overcome. Multi-layer quantization allows each layer to capture different levels of motion details, thereby reducing quantization errors and improving motion reconstruction accuracy. Compared with existing technologies, this method generates more accurate motions, especially when fine motion generation is required.

[0083] 2) Eliminate cumulative errors and improve generation efficiency: Existing auto-regression generation methods are prone to accumulating errors in long-term sequence generation due to their reliance on previous and subsequent sequences, resulting in reduced generation quality. This application uses a generative mask Transformer to predict the mask tags of the entire action sequence in parallel, and generates an action sequence through global context information, avoiding the gradual accumulation of errors. This not only improves the generation quality, but also significantly improves the generation efficiency, allowing complex actions to be efficiently generated in a small number of iterations.

[0084] 3) Enhanced generation diversity and adaptability: This application is not only suitable for conventional text-driven 3D motion generation tasks, but also supports other application scenarios, such as text-based motion restoration. In complex text description scenarios, it can better capture the diversity and flexibility of motion. The generated motion can not only show complex global characteristics, but also accurately reflect local details.

[0085] In summary, this application successfully improves the accuracy and details of the generated 3D human motion by introducing residual vector quantization and generating mask Transformer, while improving the generation efficiency and reducing the error accumulation in the generation process. These technical advantages make this application applicable to various application scenarios that require high-quality motion generation and have broad application prospects.

[0086] Please refer to Figure 4 , Figure 4 The block diagram of the composition of the device for generating 3D human motion provided by some embodiments of the present application is shown. It should be understood that the device for generating 3D human motion corresponds to the above method embodiment and can perform each step involved in the above method embodiment. The specific functions of the device for generating 3D human motion can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0087] Figure 4 The device for generating 3D human motion includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the device for generating 3D human motion. The device for generating 3D human motion includes: an acquisition module 410, which is used to obtain a text feature vector corresponding to a text description of a human motion; a quantization module 420, which is used to perform hierarchical quantization on the text feature vector to obtain a basic motion marker sequence and a residual motion marker; a processing module 430, which is used to obtain a basic motion sequence corresponding to the basic motion marker sequence, and obtain a residual motion sequence corresponding to the residual motion marker; a generation module 440, which is used to generate a 3D human motion sequence based on the basic motion sequence and the residual motion sequence.

[0088] In some embodiments of the present application, the processing module 430 is used to predict the basic action label sequence using a generative mask Transformer model to obtain the basic action sequence; and process the residual action label using a residual Transformer model to obtain the residual action sequence.

[0089] In some embodiments of the present application, the generative mask Transformer model is obtained by training an initial mask Transformer model with a training data set consisting of an action text description, an action initial tag sequence, and an initial tag mask; the residual Transformer model is obtained by training an initial Transformer model with a training data set consisting of an action tag sequence and a corresponding action text feature vector.

[0090] In some embodiments of the present application, the quantization module 420 is used to perform base-layer quantization on the text feature vector to generate the basic action marker sequence; calculate the basic action marker sequence and the text feature vector to obtain a quantization error; and perform hierarchical quantization on the quantization error to generate multiple layers of residual action markers.

[0091] In some embodiments of the present application, the generating module 440 is used to input the basic motion sequence and the residual motion sequence into a decoder, and output the 3D human motion sequence.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0093] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the operations of the method corresponding to any of the above methods provided in the above embodiments.

[0094] Some embodiments of the present application further provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operations corresponding to any of the above methods provided in the above embodiments.

[0095] like Figure 5 As shown, some embodiments of the present application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 can implement a method as described in any of the above embodiments when reading the program from the memory 510 through a bus 530 and executing the program.

[0096] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0097] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the disclosed embodiment may be used to execute instructions in the memory 510 to implement the method shown above. The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0098] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0099] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0100] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for generating 3D human motion, characterized in that: include: Obtaining a text feature vector corresponding to a text description of a human action; hierarchically quantizing the text feature vector to obtain a basic action label sequence and a residual action label; Acquire a basic action sequence corresponding to the basic action mark sequence, and acquire a residual action sequence corresponding to the residual action mark; A 3D human motion sequence is generated based on the basic motion sequence and the residual motion sequence.

2. The method according to claim 1, characterized in that The acquiring of the basic action sequence corresponding to the basic action tag sequence includes: Predicting the basic action tag sequence using a generative mask Transformer model to obtain the basic action sequence; The acquiring a residual action sequence corresponding to the residual action label comprises: The residual action labels are processed using a residual Transformer model to obtain the residual action sequence.

3. The method according to claim 2, characterized in that The generated mask Transformer model is obtained by training the initial mask Transformer model with a training data set consisting of action text descriptions, action initial tag sequences, and initial tag masks; The residual Transformer model is obtained by training an initial Transformer model using a training data set consisting of an action tag sequence and a corresponding action text feature vector.

4. The method according to any one of claims 1 to 3, characterized in that The step of performing hierarchical quantization on the text feature vector to obtain a basic action label sequence and a residual action label comprises: Performing base-level quantization on the text feature vector to generate the base action tag sequence; Calculating the basic action tag sequence and the text feature vector to obtain a quantization error; The quantization error is quantized hierarchically to generate multiple layers of residual action labels.

5. The method according to any one of claims 1 to 3, characterized in that The generating a 3D human motion sequence based on the basic motion sequence and the residual motion sequence comprises: The basic motion sequence and the residual motion sequence are input into a decoder, and the 3D human motion sequence is output.

6. A device for generating 3D human motion, characterized in that: include: An acquisition module, used for acquiring a text feature vector corresponding to a text description of a human action; A quantization module, used for performing hierarchical quantization on the text feature vector to obtain a basic action label sequence and a residual action label; A processing module, configured to obtain a basic action sequence corresponding to the basic action mark sequence, and obtain a residual action sequence corresponding to the residual action mark; A generating module is used to generate a 3D human motion sequence based on the basic motion sequence and the residual motion sequence.

7. The device according to claim 6, characterized in that The processing module is used for: Predicting the basic action tag sequence using a generative mask Transformer model to obtain the basic action sequence; The residual action labels are processed using a residual Transformer model to obtain the residual action sequence.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program executes the method according to any one of claims 1 to 5 when executed by a processor.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program executes the method according to any one of claims 1 to 5 when being run by the processor.

10. A computer program product, characterized in that The computer program product comprises a computer program, wherein the computer program executes the method according to any one of claims 1 to 5 when executed by a processor.

Images