Action generation model training method, action generation method, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, and provides an action generation model training method and device, an action generation method and device and a medium. The target text is used for describing actions of the target object at a plurality of continuous moments; obtaining a target coding feature map based on the target text through an action generation model, and generating a target action sequence based on the target coding feature map; wherein the target coding feature map comprises joint vectors of joints of the target object at all moments; the joint vector is used for representing the spatial position of the joint; the target action sequence is used for representing actions of the target object at a plurality of continuous moments. The accuracy of action generation can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an action generation model training, action generation method, device and medium. Background Art

[0002] The technology of generating the action of an object to be processed (such as a human body) based on text description content has important value in many application fields such as film production and game production. At present, the existing action generation method can generate a sequence based on a given text prompt describing the action of the object to be processed. The sequence contains the posture of the object to be processed at each moment corresponding to the text prompt. However, the existing action generation method quantizes the posture of the entire object to be processed into a vector, so the posture representation of the object to be processed is rough, which leads to the problem of poor accuracy of action generation in this method. Summary of the invention

[0003] The present application provides an action generation model training, an action generation method, a device and a medium, which can improve the accuracy of action generation.

[0004] In a first aspect, the present application provides an action generation method, the method comprising:

[0005] Obtaining a target text; the target text is used to describe the actions of the target object at multiple consecutive moments;

[0006] Through the action generation model, based on the target text, a target coding feature map is obtained, and based on the target coding feature map, a target action sequence is generated; wherein, the target coding feature map includes: joint vectors of the joints of the target object at each of the moments; the joint vectors are used to characterize the spatial positions of the joints; and the target action sequence is used to characterize the actions of the target object at the multiple consecutive moments.

[0007] In a second aspect, the present application provides an action generation model training method, the method comprising:

[0008] Acquire a first sample data set; the first sample data set includes: sample text and a first sample encoding feature map corresponding to the sample text; the sample text is used to describe the action of the first sample object at multiple consecutive moments, and the first sample encoding feature map includes: joint vectors of the joints of the first sample object at each of the moments, and the joint vectors are used to represent the spatial position of the joints;

[0009] According to the first sample data set, a first preset model is trained to obtain an action generation model; the action generation model is used to obtain a target coding feature map based on a target text, and output a target action sequence based on the target coding feature map; wherein the target text is used to describe the action of the target object at multiple consecutive moments, and the target coding feature map includes: joint vectors of the joints of the target object at each of the said moments; the target action sequence is used to characterize the action of the target object at the said multiple consecutive moments.

[0010] In a third aspect, the present application provides a method for generating character actions in a virtual reality scene, the method comprising:

[0011] Obtaining a target text; the target text is used to describe the actions of the target character in the virtual reality scene at multiple consecutive moments;

[0012] Through the action generation model, based on the target text, a target coding feature map is obtained, and based on the target coding feature map, a target action sequence is generated; wherein, the target coding feature map includes: joint vectors of the joints of the target character at each of the moments; the joint vectors are used to characterize the spatial position of the joints; and the target action sequence is used to characterize the actions of the target character at the multiple consecutive moments.

[0013] In a fourth aspect, the present application provides a video generation method, the method comprising:

[0014] Obtaining a target text; the target text is used to describe the actions of the target object at multiple consecutive moments;

[0015] A target coding feature map is obtained based on the target text through an action generation model, and a target action sequence is generated based on the target coding feature map; wherein the target coding feature map includes: joint vectors of the joints of the target object at each time; the joint vectors are used to represent the spatial positions of the joints; and the target action sequence is used to represent the actions of the target object at the multiple consecutive times;

[0016] A target video is generated according to the target action sequence; the content of the target video is the actions of the target object at the multiple consecutive moments.

[0017] In a fifth aspect, the present application provides a method for determining a robot action, the method comprising:

[0018] Obtaining a target text; the target text is used to describe the action to be performed by the target robot;

[0019] Through the action generation model, based on the target text, a target coding feature map is obtained, and based on the target coding feature map, a target action sequence is generated; wherein, the target coding feature map includes: joint vectors of the joints of the target robot at multiple consecutive moments; the joint vectors are used to characterize the spatial positions of the joints; and the target action sequence is used to characterize the actions of the target robot at the multiple consecutive moments.

[0020] In a sixth aspect, the present application provides an electronic device, comprising: a processor and a memory; the processor is communicatively connected with the memory;

[0021] The memory stores computer instructions;

[0022] The processor executes the computer instructions stored in the memory to implement the method as described in any one of the first to fifth aspects.

[0023] In a seventh aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the first to fifth aspects.

[0024] In an eighth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method as described in any one of the first to fifth aspects.

[0025] The action generation model training, action generation method, device and medium provided by the present application are based on the target text for describing the action of the target object at multiple continuous moments, and the target coding feature map obtained may include the joint vectors of the joints of the target object at each moment. Through the above method, the spatial position of each joint of the target object can be quantified as a joint vector. Therefore, compared with the prior art in which the overall human posture is represented by a vector, the present application improves the precision of quantifying the action posture of the target object, and the joint vectors of different joints in the target coding feature map can be used to describe the spatial positions of different joints, thereby improving the accuracy of quantifying the action of the target object at multiple continuous moments, and then the accuracy of the subsequent target action sequence (i.e., action generation) generated for representing the action of the target object at the multiple continuous moments based on the target coding feature map can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0027] Figure 1 A flowchart of an action generation method provided by this application;

[0028] Figure 2 A flowchart of an action generation model training method provided in this application;

[0029] Figure 3 A flowchart of a mask processing process provided by this application;

[0030] Figure 4 A flowchart of adding attention provided for this application;

[0031] Figure 5 A schematic diagram of the training process of an action quantization model provided in this application;

[0032] Figure 6 A schematic diagram of the training process of another action quantization model provided in this application;

[0033] Figure 7 A flowchart of an action generation model provided in this application;

[0034] Figure 8 A flowchart of a method for generating character actions in a virtual reality scene provided by the present application;

[0035] Fig. 9 A schematic diagram of a process of video generation method provided in this application;

[0036] Fig.10 A schematic diagram of a flow chart of a robot action determination method provided in this application;

[0037] Fig.11 A schematic diagram of the structure of an action generation model training device 60 provided in the present application;

[0038] Fig.12 A schematic diagram of the structure of an action generating device 70 provided in the present application;

[0039] Fig.13 A schematic diagram of the structure of a character action generation device 80 in a virtual reality scene provided by the present application;

[0040] Fig.14A schematic diagram of the structure of a video generating device 90 provided in the present application;

[0041] Fig.15 A schematic diagram of the structure of a robot action determination device 150 provided in the present application;

[0042] Fig.16 A schematic diagram of the hardware structure of an electronic device provided in this application.

[0043] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The technology of generating the action of the object to be processed based on the text description content has important value in many application fields such as film production and game production. Taking the above-mentioned object to be processed as a human body as an example, the existing action generation method can generate a sequence based on the given text prompts describing the human body action. The sequence contains the posture of the human body at each moment corresponding to the text prompt.

[0046] At present, the existing motion generation methods mainly include motion masking (MoMask), multi-modal motion (MMM), text-to-motion generative pre-trained transformer (T2M-GPT), etc. However, the existing motion generation method quantizes the posture of the entire human body into a vector, that is, the vector is used to represent the overall posture of the human body, and does not pay attention to the specific joint position of the human body. Therefore, the existing human body posture representation is rough, there is no spatial relationship between joints, and there is a problem of poor accuracy in motion generation.

[0047] Considering the above problems existing in the existing motion generation methods, this application proposes a method of quantizing each joint into a vector and generating motion based on the quantization result. Compared with the prior art of representing the overall human posture by a vector, this application quantizes the spatial position of a joint into a vector, achieving more refined posture quantization and improving the accuracy of motion generation based on the quantization result.

[0048] Optionally, the execution subject of the action generation method provided in the present application may be, for example, any electronic device with processing capabilities, such as a terminal or a server.

[0049] The following specific embodiments are combined to describe the action generation method of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0050] Figure 1 This is a flow chart of an action generation method provided by this application. Figure 1 As shown, the method comprises the following steps:

[0051] S101: Obtain a target text, wherein the target text may be used to describe actions of a target object at multiple consecutive moments.

[0052] Exemplarily, the target object may be any object such as a human body or an animal that can change its motion by changing the position of its joints. In addition, the present application does not limit the language, length, etc. of the target text.

[0053] Optionally, the electronic device may receive the target text input by the user through an application programming interface (API) or a graphical user interface (GUI). Alternatively, the electronic device may also receive the user's voice data, and then parse the content described by the voice data as the target text. Alternatively, the electronic device may obtain the target text by referring to the method for obtaining the target text in the existing action generation method.

[0054] S102. Generate a target encoding feature map based on the target text through an action generation model.

[0055] Among them, the above-mentioned target coding feature map may include: the joint vectors of the joints of the above-mentioned target object at each moment. Among them, the joint vector can be used to characterize the spatial position of the joint. Exemplarily, assuming that the number of joints of the above-mentioned target object is M (M can be an integer greater than or equal to 1), and the number of the above-mentioned continuous moments is N (N can be an integer greater than 1), optionally, the target coding feature map may include, for example: M×N joint vectors. Among them, any joint vector can be used to characterize the spatial position of a joint of the target object at a moment. That is to say, for any moment in the above-mentioned N continuous moments, the position of each joint of the target object can be represented by a joint vector. It should be understood that the present application does not limit the values ​​of the above-mentioned M and N. Taking the above-mentioned target object as a human body as an example, in some embodiments, the number of joints of the human body can be, for example, 22, that is, for any moment in the above-mentioned N continuous moments, the position of each joint of the human body can be represented by 22 joint vectors. In addition, the present application does not limit the dimension of any joint vector.

[0056] Optionally, the above-mentioned action generation model can be, for example, pre-configured in the electronic device. In some embodiments, the action generation model can be a trained machine learning model, such as a neural network model. The action generation model can be used to obtain a target encoding feature map based on the above-mentioned target text, and generate a target action sequence based on the target encoding feature map.

[0057] S103, generating a target action sequence through an action generation model based on the above target encoding feature map.

[0058] The target action sequence may be used to characterize the action of the target object at a plurality of consecutive moments. Exemplarily, the target action sequence may include data for representing the position of each joint of the target object at each moment.

[0059] It should be understood that the application scenario of the target action sequence is not limited in the present application. Exemplarily, the target action sequence can be used for video rendering, so as to render a video including the action of the target object based on the target action sequence. Optionally, the method for performing video rendering based on the target action sequence can refer to the existing method for rendering the output video based on the action sequence generated, which will not be repeated here.

[0060] In the present embodiment, based on the target text for describing the action of the target object at multiple continuous moments, the target coding feature map obtained may include the joint vectors of the joints of the target object at each moment. Through the above method, the spatial position of each joint of the target object can be quantified as a joint vector. Therefore, compared with the prior art in which the posture of the overall object is represented by a vector, the present application improves the precision of quantifying the action posture of the target object, and the joint vectors of different joints in the target coding feature map can be used to describe the spatial positions of different joints, thereby improving the accuracy of quantifying the action of the target object at multiple continuous moments, and then the accuracy of the target action sequence (i.e., action generation) generated for representing the action of the target object at the multiple continuous moments can be improved based on the target coding feature map.

[0061] The following is an exemplary description of how the action generation model obtains a target encoding feature map based on the target text, and generates a target action sequence based on the target encoding feature map:

[0062] As a possible implementation manner, the action generation model may include, for example: a text feature extraction module, a joint feature extraction module and a decoding module.

[0063] The action generation model may, for example, extract text features from the target text through the above-mentioned text feature extraction module to obtain text features of the target text.

[0064] In some embodiments, the above-mentioned text feature can also be called a text embedding feature. The text feature can be used to characterize the semantics of the target text, that is, the text feature captures the semantic information of the target text. Optionally, the text feature extraction module extracts text features from the target text, for example, by referring to any existing method for extracting text embedding features of text, which will not be described in detail here. Optionally, the above-mentioned text feature can be, for example, a vector used to characterize the semantics of the target text.

[0065] Then, the action generation model can encode the text features of the target text through the above-mentioned joint feature extraction module to obtain the above-mentioned target encoding feature map.

[0066] Exemplarily, the joint feature extraction module can be a machine learning model pre-trained according to the sample encoding feature map and the text features of the sample text. Through the joint feature extraction module, the text features can be encoded to obtain the joint vectors corresponding to the joints of the target object described by the target text as the target encoding feature map.

[0067] Then, the action generation model can decode the above target encoding feature map through the above decoding module to obtain the target action sequence.

[0068] Exemplarily, the decoding module may be, for example, a decoder in a preset action quantization model. The preset action quantization model may be, for example, a pre-trained machine learning model. For example, the action quantization model may be a preset machine learning model trained according to a sample action sequence. Among them, the preset machine learning model may, for example, include an encoding layer and a decoding layer. The trained encoding layer may be an encoder of the action quantization model. The trained decoding layer may, for example, be a decoder of the action quantization model. The above-mentioned encoder may be used to encode the action sequence based on the input action sequence to obtain an encoding feature map. The encoding feature map may, for example, refer to the aforementioned target encoding feature map. The above-mentioned decoder may be used to decode the encoding feature map based on the input encoding feature map (for example, the aforementioned target encoding feature map) to obtain an action sequence (for example, the aforementioned target action sequence).

[0069] In this embodiment, the action generation model can obtain the text features of the target text through the text feature extraction module, and then encode the text features through the joint feature extraction module to obtain a target encoding feature map, thereby realizing the representation of the action of the target object through the target encoding feature map, and then based on the decoding module, the target encoding feature map is decoded to obtain the target action sequence, thereby realizing action generation.

[0070] As a possible implementation, the above-mentioned action generation model may be, for example, a pre-trained machine learning model. Taking the action generation model as a pre-trained machine learning model as an example, the training method of the action generation model is exemplarily described below:

[0071] First, it should be understood that the execution subject of the training method of the action generation model can also be any electronic device with processing capabilities, such as a terminal or a server. Optionally, the electronic device that executes the model training method and the execution subject of the action generation method provided by this application can be the same electronic device or different electronic devices, and this application does not limit this.

[0072] Figure 2 This is a flow chart of a method for training an action generation model provided in this application. Figure 2 As shown, the method may include the following steps:

[0073] S201: Obtain a first sample data set.

[0074] The first sample data set may include, for example, a sample text and a first sample encoding feature map corresponding to the sample text. The sample text may be used to describe the actions of the first sample object at multiple consecutive moments. Exemplarily, the multiple consecutive moments may be, for example, N consecutive moments. N may be an integer greater than 1.

[0075] It should be understood that the present application does not limit the type of the above-mentioned first sample object. For example, the first sample object can be any object such as a human body, an animal, etc. that can change its action by changing the joint position. Optionally, the above-mentioned first sample object can be related to the scenario to which the action generation model is required to be applied. For example, taking the need to use the action generation model in a human body action generation scenario as an example, the above-mentioned first sample object can be, for example, a human body. Optionally, the first sample object can include, for example, different human bodies. Taking the need to use the action generation model in an action generation scenario of a target type of animal as an example, the above-mentioned first sample object can be, for example, an animal of the target type. Optionally, the first sample object can include, for example, different animals belonging to the target type. In addition, it should be understood that the present application does not limit the language, length, etc. of the above-mentioned sample text.

[0076] The first sample encoding feature map may include: the joint vectors of the joints of the first sample object at each moment. The joint vectors may be used to characterize the spatial position of the joints. Exemplarily, taking the number of joints of the first sample object as M (M may be an integer greater than or equal to 1), and the number of consecutive moments as N (N may be an integer greater than 1), the first sample encoding feature map may include, for example, M×N joint vectors. Any joint vector may be used to characterize the spatial position of a joint of the first sample object at a moment.

[0077] Among them, M is the number of joints of the first sample object. The N is the N of the actions of the first sample object described by the sample text at N consecutive moments. That is to say, for any of the above N consecutive moments, the position of each joint of the first sample object can be represented by a joint vector. It should be understood that the present application does not limit the values ​​of the above M and N. Taking the above-mentioned first sample object as a human body as an example, in some embodiments, the number of joints of the first sample object can be, for example, 22, that is, for any of the above N consecutive moments, the position of each joint of the first sample object can be represented by 22 joint vectors. In addition, the present application does not limit the dimension of any joint vector.

[0078] Optionally, the electronic device may obtain the first sample data set through an API, or a GUI, for example.

[0079] S202. According to the first sample data set, a first preset model is trained to obtain an action generation model.

[0080] Optionally, the first preset model may include, for example, any existing neural network model to be trained, such as a Transformer neural network. In some embodiments, the first preset model may include sub-models to be trained and modules that do not require training. The module that does not require training may be, for example, a pre-trained machine learning model, or may be a non-machine learning model. The electronic device may train the first preset model according to the first sample data set, so that the action generation model obtained by training can be used to obtain a target coding feature map based on the target text, and output a target action sequence based on the target coding feature map. The target text and the target action sequence may refer to the aforementioned embodiments, which will not be described in detail here.

[0081] Optionally, the electronic device may, for example, combine training methods such as adding an attention mechanism to train the first preset model according to the above-mentioned first sample data set to obtain the action generation model.

[0082] In this embodiment, the first sample encoding feature map may include a joint vector for characterizing the spatial position of the joint of the first sample object. Therefore, the action generation model obtained by training the first sample data set including the first sample encoding feature map can quantize the joint of the target object described by the target text into a joint vector. That is to say, a joint vector is used to characterize the spatial position of a joint. Through the above method, it is realized to obtain an action generation model based on a joint vector for characterizing the spatial position of a joint. Compared with the prior art that uses a vector to characterize the overall human posture, the present application improves the precision of quantizing the action posture of the first sample object. Through the joint vectors of different joints in the first sample encoding feature map, the spatial positions of different joints can be described, thereby improving the accuracy of the action generation model obtained by training based on the first sample encoding feature map, and then the accuracy of action generation based on the action generation model can be improved.

[0083] The following is a detailed description of how the electronic device obtains the first sample data set:

[0084] As a possible implementation, as mentioned above, the electronic device may, for example, obtain the first sample data set through an API or a GUI. Optionally, the first sample data set may, for example, be pre-stored in the electronic device. The electronic device may obtain the first sample data set from data stored in the electronic device. Alternatively, the electronic device may, for example, also receive the first sample data set input by a user.

[0085] As another possible implementation, the electronic device may first obtain the first sample action sequence corresponding to the aforementioned sample text. The first sample action sequence may be used to characterize the actions of the first sample object at multiple consecutive moments. The sample text and the first sample action sequence may correspond one to one. In other words, the first sample action sequence characterizes the actions of the first sample object described by the sample text at multiple consecutive moments.

[0086] Optionally, taking the example of a first sample data set including multiple sample texts, the corresponding multiple sample texts respectively correspond to first sample action sequences, wherein different first sample action sequences are used to represent the actions of the first sample object at multiple consecutive moments, and the number of moments in the multiple consecutive moments may be the same or different, and the present application does not limit this.

[0087] Optionally, the sample text and the first sample action sequence corresponding to the sample text may be, for example, a public dataset, or a dataset made by researchers. For example, the sample text and the first sample action sequence corresponding to the sample text may belong to a HumanML3D (name of a dataset) dataset or a KIT-ML (name of a dataset) dataset, etc.

[0088] Exemplarily, the electronic device may obtain the first sample action sequence corresponding to the sample text through an API or GUI, etc. Alternatively, the electronic device may obtain the first sample action sequence corresponding to the sample text by referring to any existing method for obtaining a data set, which will not be described in detail here.

[0089] After obtaining the first sample action sequence corresponding to the sample text, the electronic device may encode the first sample action sequence to obtain the first sample encoding feature map. After obtaining the first sample encoding feature map, the electronic device may, for example, establish a mapping relationship between the first sample encoding feature map and the corresponding sample text. Then, based on the mapping relationship, the electronic device may use the first sample encoding feature map and the corresponding sample text to construct the first sample data set.

[0090] In this embodiment, a first sample encoding feature map can be obtained by encoding the first sample action sequence corresponding to the sample text. The first sample encoding feature map and the aforementioned sample text can constitute the aforementioned first sample data set, thereby achieving the acquisition of the first sample data set for training the action generation model, and training the first preset model through the first sample encoding feature map so that the action generation model can obtain the target encoding feature map based on the target text, which lays the foundation for obtaining the target encoding feature map based on the action generation model.

[0091] In some embodiments, the electronic device may, for example, first identify the data used to characterize the spatial position of each joint at each moment from the first sample action sequence. Then, for the data used to characterize the spatial position of any joint at any moment, the electronic device may encode the data (for example, referring to the method of data feature extraction) and obtain the vector after the data is encoded as a joint vector. Then, the electronic device may obtain the first sample encoding feature map based on the joint vectors of each joint at each moment.

[0092] In some embodiments, the electronic device may also use, for example, an encoder in a preset motion quantization model to encode the first sample motion sequence to obtain the first sample encoding feature map.

[0093] Optionally, the encoder in the action quantization model may be preset in the electronic device, for example. The electronic device may call the encoder, input the first sample action sequence into the encoder, and output the first sample encoding feature map through the encoder.

[0094] Optionally, in some embodiments, the above-mentioned action quantization model may further include a decoder. The decoder may be used to decode the target encoded feature map to obtain a target action sequence. In other words, the decoder may be used in the action generation process as a decoding module of the action generation model.

[0095] Through the above method, the first sample action sequence is encoded by the encoder of the preset action quantization model without the need for additional storage of encoding rules. While obtaining the first sample encoding feature map, the flexibility and universality of the encoding are improved because no fixed encoding rules are required. By obtaining the first sample encoding feature map, the foundation is laid for training the action generation model based on the first sample encoding feature map.

[0096] The following is a detailed description of how to obtain the above action quantization model:

[0097] In some embodiments, the above-mentioned action quantization model can be obtained, for example, by model training the second preset model based on the second sample action sequence. It should be understood that the execution subject of the action quantization model obtained by model training the second preset model based on the second sample action sequence can be any electronic device with processing capabilities. Optionally, the electronic device mentioned here and the electronic device for executing the action generation model training method described in the aforementioned embodiment can be the same electronic device, or can be a different electronic device, and this application does not limit this.

[0098] Taking the example that the electronic device that performs “training the second preset model based on the second sample action sequence to obtain the action quantization model” is the same electronic device as the electronic device for performing the action generation model training method, the electronic device may also first obtain at least one second sample action sequence. The second sample action sequence may be used to characterize the action of the second sample object at multiple consecutive moments.

[0099] Optionally, the second sample object and the first sample object may be the same object (e.g., the human body of the same person), or different objects (e.g., the human bodies of different people). The first sample object and the second sample object may be of the same type. For example, the first sample object and the second sample object are both human bodies, or both are animals, etc.

[0100] Optionally, the implementation manner in which the electronic device acquires the at least one second sample action sequence may refer to the method for acquiring the first sample data set described in the aforementioned embodiment, for example, and will not be described in detail here.

[0101] Then, the electronic device may train the second preset model according to the at least one second sample action sequence to obtain an action quantization model.

[0102] The second preset model may include a coding layer and a decoding layer, and the coding layer and the decoding layer may both be two-dimensional convolutional layers. By using two-dimensional convolutional layers as coding layers and decoding layers, the coding layers and the decoding layers perform data processing similar to image processing, thereby improving the efficiency of coding and decoding, that is, improving the efficiency of subsequent encoding by the encoder using the motion quantization model, and also improving the efficiency of subsequent decoding by the decoder based on the motion quantization model.

[0103] Exemplarily, the second preset model may be any existing network model that can be used to train an encoder and a decoder. For example, the second preset model may be a vector quantized variational auto encoder (VQVAE).

[0104] In some embodiments, the electronic device may, for example, repeatedly perform the following steps until a second preset iteration condition is reached to obtain a well-trained motion quantization model:

[0105] Step 1: The electronic device may encode the second sample action sequence based on the above encoding layer to obtain a second sample encoding feature map. The second sample encoding feature map may include: joint vectors of the joints of the second sample object at each time. In other words, any joint vector is used to represent the spatial position of a joint of the second sample object at a time.

[0106] Optionally, the second sample action sequence may be input to the encoding layer. The encoding layer may encode the second sample action sequence and output an encoding result. The encoding result may include initial joint vectors of the joints of the second sample object at each time. Any initial joint vector is used to represent the initial spatial position of a joint of the second sample object at a time.

[0107] Exemplarily, the coding layer can directly use the coding result as the above-mentioned second sample coding feature map. Alternatively, taking the example that the coding layer also includes a code book, the coding layer can use the coding result as the initial sample coding feature map. Then, for each initial joint vector in the initial sample coding feature map, the coding layer can determine the vector with the highest similarity to the initial joint vector from the code book as the replaced joint vector corresponding to the initial joint vector. Then, the coding layer can obtain the above-mentioned second sample coding feature map based on the replaced joint vector corresponding to each initial joint vector.

[0108] Step 2: The electronic device can decode the second sample encoding feature map based on the above decoding layer to obtain a predicted action sequence.

[0109] Optionally, the above-mentioned decoding layer decodes the second sample encoded feature map to obtain an implementation method of predicting the action sequence. This can refer to any existing model training process, in which a convolutional layer is used to perform decoding processing to obtain a prediction result, and will not be repeated here.

[0110] Step 3: The electronic device can determine the loss function of the second preset model according to the predicted action sequence and the second sample action sequence, and update the parameters of the second preset model according to the loss function of the second preset model. The second preset model when the second preset iteration condition is met can be an action quantization model.

[0111] Exemplarily, the second preset iteration condition may refer to any iteration condition used in an existing model training process, such as reaching a preset number of training rounds, etc., which will not be described in detail in this application.

[0112] Optionally, the implementation method of calculating the loss function of the second preset model based on the above-mentioned predicted action sequence and the second sample action sequence, as well as the method of training and updating parameters of the next round of the second preset model based on the loss function, can refer to the existing model training method and will not be repeated here.

[0113] In this embodiment, the second preset model is trained by the second sample action sequence to obtain the action quantization model. The two-dimensional convolution layer is used as the encoding layer and the decoding layer, so as to improve the efficiency of the action quantization model encoder for encoding and the efficiency of the decoder for decoding. The action quantization model obtained by training the second preset model by the second sample action sequence can be used to encode the first sample action sequence, which lays the foundation for obtaining the first sample encoding feature map to train the action generation model.

[0114] The following is a detailed description of how the electronic device trains the first preset model according to the first sample data set to obtain the action generation model:

[0115] As a possible implementation manner, the above-mentioned first preset model may, for example, include: a text feature extraction module, a decoding module and a first sub-model to be trained.

[0116] In some embodiments, the electronic device may, for example, extract text features from the sample text using a text feature extraction module to obtain text features of the sample text.

[0117] Then, the electronic device can train the first sub-model to be trained according to the text features of the sample text and the first sample encoding feature map to obtain a joint feature extraction module. In other words, the trained first sub-model can be used as the joint feature extraction module of the action generation model.

[0118] Optionally, in some embodiments, the action generation model may include: a text feature extraction module, a joint feature extraction module and a decoding module. In the process of applying the action generation model, as described above, the text feature extraction module can extract text features of the target text to obtain text features of the target text. The joint feature extraction module can encode the text features of the target text to obtain a target encoding feature map. The decoding module can be used to decode the target encoding feature map to obtain a target action sequence. Among them, the decoding module can, for example, be a decoder of the aforementioned action quantization model.

[0119] Through the above method, the first sub-model is used as the module to be trained in the first preset model. During the training process of the first preset model, there is no need to train the text feature extraction module or the decoding module. In the training process of the first preset model, it is actually only necessary to train the first sub-model. Therefore, the efficiency of training the first preset model is improved, that is, the efficiency of obtaining a trained action generation model is improved.

[0120] In some embodiments, the electronic device may, for example, repeatedly perform the following steps in the process of training the first sub-model to be trained according to the text features of the sample text and the first sample encoding feature map until a first preset iteration condition is reached:

[0121] Step 1: The electronic device performs mask processing on at least one first sample coding feature map to obtain a masked coding feature map.

[0122] Step 2: The electronic device obtains a predicted coding feature map based on the masked coding feature map and the text features of the sample text.

[0123] Step 3: The electronic device calculates the loss function of the first sub-model based on the first sample coding feature map and the predicted coding feature map, and updates the parameters of the first sub-model according to the loss function of the first sub-model. The first sub-model obtained when the first preset iteration condition is reached is the joint feature extraction module.

[0124] Exemplarily, the first preset iteration condition can refer to any iteration condition used in an existing model training process, such as reaching a preset number of training rounds, etc., which will not be described in detail in this application. Optionally, the implementation method of calculating the loss function of the first sub-model based on the above-mentioned first sample coding feature map and the predicted coding feature map, as well as the method of training and updating parameters of the next round of the first sub-model based on the loss function, can refer to the existing model training method, which will not be described in detail here.

[0125] In this embodiment, during the training process of the first sub-model to be trained, the prediction of the prediction coding feature map is performed after the first sample coding feature map is masked. Because the spatial position of the joint is unknown when the actual action occurs, the joint vector included in the first sample coding feature map is masked by masking the first sample coding feature map, so as to achieve a scene where the joint vector is unknown. Therefore, the training of the first sub-model is achieved by a training method that is more in line with the actual action generation scene. Therefore, the trained first sub-model, that is, the joint feature extraction module in the action generation model is obtained by training in combination with the actual action generation scene features, so that the accuracy of the subsequent application of the action generation model to the actual scene for action generation can be improved.

[0126] The following describes in detail how the electronic device performs mask processing on the first sample coding feature map to obtain the masked coding feature map, and how to obtain the predicted coding feature map based on the masked coding feature map and the text features of the sample text:

[0127] In some embodiments, the electronic device may, for example, perform mask processing in the time dimension and / or mask processing in the space dimension on at least one first sample coding feature map to obtain a masked coding feature map.

[0128] The mask processing of the time dimension may include, for example, masking a joint vector corresponding to at least one of the plurality of consecutive moments. The mask processing of the space dimension may include, for example, masking a joint vector corresponding to at least one joint of the first sample object at at least one moment.

[0129] Optionally, the electronic device may, for example, perform mask processing on a portion of the first sample coding feature graph in the time dimension, and / or, perform mask processing on a portion of the space dimension, to obtain a masked coding feature graph. That is, there may be a portion of the first sample coding feature graph that has not been masked. In this implementation, the electronic device may, for example, obtain a predicted coding feature graph based on the masked coding feature graph, the first sample coding feature graph that has not been masked, and the aforementioned sample text. Alternatively, the electronic device may also perform mask processing on a time dimension, and / or, perform mask processing on a space dimension, to obtain a masked coding feature graph. In this implementation, the electronic device may, for example, obtain a predicted coding feature graph based on the masked coding feature graph and the aforementioned sample text.

[0130] Taking the example of an electronic device performing mask processing on the time dimension of at least one first sample coding feature map to obtain a masked coding feature map, the electronic device can, for example, perform mask processing on any first sample coding feature map in the at least one first sample coding feature map, and on the joint vectors corresponding to at least one moment in multiple consecutive moments in the first sample coding feature map to obtain a masked coding feature map.

[0131] Taking the example of an electronic device performing spatial dimension masking processing on at least one first sample coding feature map to obtain a masked coding feature map, the electronic device can, for example, perform mask processing on a joint vector corresponding to at least one joint of the first sample object at at least one moment for any first sample coding feature map in the at least one first sample coding feature map to obtain a masked coding feature map.

[0132] Taking the example of an electronic device performing mask processing of the time dimension and the space dimension on at least one first sample coding feature map to obtain a masked coding feature map, optionally, the electronic device may, for example, first perform mask processing of the time dimension and then perform mask processing of the space dimension. Alternatively, the electronic device may, for example, first perform mask processing of the space dimension and then perform mask processing of the time dimension.

[0133] Taking the example of an electronic device first performing mask processing of the time dimension and then performing mask processing of the space dimension (the implementation method of the electronic device first performing mask processing of the space dimension and then performing mask processing of the time dimension can refer to the following embodiment, which will not be repeated here), optionally, the electronic device can, for example, perform mask processing on the joint vector corresponding to at least one of the multiple consecutive moments in the first sample coding feature map to obtain a coding feature map masked in the time dimension. Then, the electronic device can perform mask processing on the joint vector corresponding to at least one joint of the first sample object at at least one moment in the coding feature map masked in the time dimension to obtain a masked coding feature map.

[0134] It should be understood that the present application does not limit the method for determining at least one of the above-mentioned multiple continuous moments. For example, the electronic device can randomly determine at least one of the multiple continuous moments from the above-mentioned multiple moments. In addition, the specific number referred to by the at least one can also be random, for example. Or, the specific number referred to by the at least one can be preconfigured. In addition, it should be understood that the present application does not limit the method for determining the above-mentioned at least one joint. For example, the electronic device can randomly determine at least one joint from the multiple joints of the above-mentioned first sample object. In addition, the specific number referred to by the at least one can also be random, for example. Or, the specific number referred to by the at least one can be preconfigured. In addition, for "the joint vector corresponding to at least one joint at at least one moment", it should be understood that the number of "at least one moment" mentioned here corresponding to different joints and which specific moment can be different or the same, and the present application does not limit this.

[0135] For example, Figure 3 The following is a flowchart of a mask processing process provided by this application. Figure 3 As shown, the horizontal axis represents each moment (i.e., the time dimension), the vertical axis represents each joint (i.e., the spatial dimension), and each square represents a joint vector. Figure 3 As shown, the electronic device can mask the joint vectors corresponding to the k1 moment and the k2 moment in the first sample coding feature map to obtain a coding feature map masked in the time dimension (light-colored squares represent masked joint vectors, and dark-colored squares represent unmasked joint vectors). Then, the electronic device can, for example, mask the joint vectors corresponding to the j1 joint of the first sample object at three moments other than the k1 moment and the k2 moment in the coding feature map masked in the time dimension, and mask the joint vectors corresponding to the j2 joint at one moment other than the k1 moment and the k2 moment, and mask the joint vectors corresponding to the j3 joint at one moment other than the k1 moment and the k2 moment, to obtain a coding feature map masked in the spatiotemporal dimension (i.e., a masked coding feature map).

[0136] In this embodiment, masking of the time dimension is achieved by masking the joint vector corresponding to at least one of the multiple consecutive moments in the first sample coding feature map, and masking of the space dimension is achieved by masking the joint vector corresponding to at least one joint of the first sample object at at least one moment in the coding feature map masked in the time dimension. Therefore, through the above method, after masking the time dimension and the space dimension of the first sample coding feature map, prediction of the prediction coding feature map is achieved, and training of the action generation model is achieved through a masking method that is more in line with the actual action generation scene (because when the actual action occurs, the joint positions in time and space may be unknown), so that the action generation model is trained in combination with the characteristics of the time and space dimensions of the first sample object, so that the accuracy of subsequent action generation based on the action generation model can be improved.

[0137] As mentioned above, the electronic device can obtain a predicted coding feature map based on the masked coding feature map and the sample text. In some embodiments, the electronic device can, for example, add attention in the time dimension and / or in the space dimension to the masked coding feature map based on the text features of the sample text to obtain a coding feature map after adding the attention mechanism. Then, the electronic device can obtain the predicted coding feature map based on the coding feature map after adding the attention mechanism.

[0138] Exemplarily, the above-mentioned attention addition in the time dimension may include, for example: adding attention between joint vectors corresponding to different moments among the above-mentioned multiple moments. Exemplarily, the above-mentioned attention addition in the space dimension may include, for example: adding attention between joint vectors corresponding to different joints among the multiple joints of the first sample object. It should be understood that the implementation method of adding attention in the time dimension or the implementation method of adding attention in the space dimension may refer to any existing attention mechanism adding method, which will not be repeated here.

[0139] Optionally, the electronic device may, for example, add attention in the time dimension to the masked coding feature map based on the text features of the sample text, and obtain the coding feature map after adding the attention mechanism. Alternatively, the electronic device may, based on the text features of the sample text, add attention in the space dimension to the masked coding feature map, and obtain the coding feature map after adding the attention mechanism. Alternatively, the electronic device may, for example, also add attention in the time dimension and attention in the space dimension to the masked coding feature map based on the text features of the sample text, and obtain the coding feature map after adding the attention mechanism.

[0140] In this embodiment, by adding attention in the time dimension and / or adding attention in the space dimension to the masked coding feature map, a connection between different joint vectors in the masked coding feature map can be established, because in real scenes, the spatial position of a joint at different times (time dimension) and the spatial position between different joints (spatial dimension) are usually related. Therefore, by adding attention in the time dimension and / or adding attention in the space dimension, the coding feature map after adding the attention mechanism carries the characteristics of the spatial position of the joint in the time dimension and / or the space dimension, which improves the accuracy of the expression of the spatial position of the joint, thereby improving the accuracy of the subsequent predicted coding feature map obtained based on the coding feature map after adding the attention mechanism, thereby improving the training efficiency of the first sub-model.

[0141] Optionally, taking the case where attention is added to the time dimension and the space dimension of the masked coding feature map based on the text features of the sample text to obtain the coding feature map after adding the attention mechanism, the electronic device can, for example, expand the masked coding feature map into a one-dimensional feature map based on the above text features, and add one-dimensional attention to the one-dimensional feature map based on the position embedding feature to obtain the one-dimensional attention mechanism processing result. Among them, the above position embedding feature describes the position of each joint vector in the first sample coding feature map. Therefore, through the position embedding feature, after the masked coding feature map is expanded into a one-dimensional feature map, it can still be known which position (for example, the row and column) of each joint vector in the original first sample coding feature map is the vector, so as to improve the accuracy of the one-dimensional attention addition, and then improve the accuracy of other subsequent processing based on the one-dimensional attention mechanism processing result.

[0142] Then, the electronic device can add attention in the spatial dimension to the above one-dimensional attention mechanism processing result based on the position embedding feature, and obtain the spatial dimension attention mechanism processing result. Then, the electronic device can add attention in the time dimension to the spatial dimension attention mechanism processing result, and obtain the encoded feature map after adding the attention mechanism.

[0143] For example, Figure 4 A flowchart of adding attention provided by this application. Figure 4 As shown, T represents the text feature of the sample text. The electronic device can expand the masked encoding feature map into a one-dimensional feature map, and based on the position of each joint vector described by the position embedding feature in the first sample encoding feature map, perform one-dimensional attention mechanism addition, spatial dimension attention addition, and time dimension attention addition, thereby obtaining the encoding feature map after adding the attention mechanism.

[0144] It should be understood that the implementation method of adding attention in the spatial dimension to the processing result of the one-dimensional attention mechanism, and the implementation method of adding attention in the temporal dimension to the processing result of the spatial dimension attention mechanism, for example, can refer to the existing method of adding attention mechanism, and will not be repeated here.

[0145] In this embodiment, through the text features and position embedding features of the sample text, one-dimensional attention, spatial dimension attention, and time dimension attention can be added to the masked coding feature map to obtain the coding feature map after adding the attention mechanism. Through the above method, the addition of the attention mechanism to the masked coding feature map in both the time and space dimensions is achieved, which improves the accuracy of the predicted coding feature map in the time and space dimensions, and further improves the accuracy of the action generation model trained based on the predicted coding feature map.

[0146] Taking the above-mentioned first preset model as the Transformer neural network model based on the attention mechanism and the above-mentioned second preset model as VQVAE as an example, Figure 5 A schematic diagram of the training process of an action quantization model provided in this application. Figure 6 A schematic diagram of the training process of another action quantization model provided in this application. Figure 5 and Figure 6 As shown, during the training process of the action quantization model, the VQVAE model can encode the input second sample action sequence through an encoding layer (for example, a two-dimensional convolutional network) to achieve joint vector quantization and obtain a second sample encoding feature map. Then, the VQVAE model can decode the second sample encoding feature map through a decoding layer (for example, a two-dimensional convolutional network) to obtain a predicted action sequence, and then train the VQVAE model based on the predicted action sequence and the second sample action sequence. The trained VQVAE model can be used as the above-mentioned action quantization model.

[0147] like Figure 6 As shown, the joint vector output by the encoding layer can be used as the initial joint vector (such as Figure 6 In the initial sample coding feature map shown in , each square represents an initial joint vector), the coding layer can determine the vector with the highest similarity to the initial joint vector from the coding book as the target joint vector corresponding to the initial joint vector, and then construct the above-mentioned second sample coding feature map based on the target joint vector.

[0148] The trained encoding layer can be used as an encoder of the motion quantization model, and the trained decoding layer can be used as a decoder of the motion quantization model.

[0149] The encoder can be used to encode the first sample action sequence to obtain a first sample encoding feature map. The first sample encoding feature map and the corresponding sample text can be used as a first sample data set for training the Transformer model. The trained Transformer model can be used as a joint feature extraction module of the action generation model.

[0150] Figure 7 This is a flow chart of an action generation model provided in this application. Figure 7 As shown, during the training of the Transformer model, the first sample coding feature map can be masked in two dimensions, time dimension and space dimension (time-space) (the specific implementation method can refer to the method described in the aforementioned embodiment, for example), to obtain the masked coding feature map. The Transformer model can also add an attention mechanism in two dimensions, time-space, to the masked coding feature map based on the text features of the sample text and the position embedding features, and predict the coding feature map based on the addition of the attention mechanism to obtain a predicted coding feature map. The addition of the attention mechanism in two dimensions, time-space, not only considers whether the generated action sequence conforms to the input text in terms of time series, but also considers whether the generated joints are reasonable in spatial structure, thereby improving the accuracy of the action generation model training, and then improving the accuracy of subsequent action generation based on the action generation model.

[0151] The decoder of the above-mentioned action quantization model can be used as a decoding module of the action generation model, which is used to decode the target encoded feature map output by the joint feature extraction module to obtain the target action sequence.

[0152] Through the above method, each individual joint of the human body can be quantized into a vector, that is, a joint vector. Through this quantization method, compared with quantizing the entire posture into a vector, the complexity of a single joint is significantly lower, so the present application simplifies the quantization process and improves the quantization efficiency. In addition, by quantizing each individual joint of the human body into a vector, the spatial structural characteristics of the human body can be maintained, and the changes in the joint positions over time are combined. The characteristics of the space-time structure are guaranteed by the above-mentioned first sample encoding feature map, that is, the spatial relationship between the joints and the time movement pattern are retained, thereby improving the accuracy of the action generation model.

[0153] In addition, for the first sample encoded feature map, various 2D operations (such as 2D convolution, 2D position encoding, and 2D attention mechanism, etc.) widely used in two-dimensional (2D) images are used to improve the flexibility of action generation model training. For example, both the encoding layer and the decoding layer can be a two-dimensional convolutional network for efficient feature extraction, similar to processing two-dimensional images.

[0154] The inventors also verified the accuracy of the action generation model through experiments on different data sets. Compared with the existing action generation methods, the action generation model provided by the present application reduced the Fréchet Inception Distance (FID) by 26.6% on the HumanML3D data set and reduced the FID by 29.9% on the KIT-ML data set, verifying the accuracy of the action generation model.

[0155] It should be understood that the present application does not limit the application scenarios of the action generation model. For example, the action generation model can be used in the fields of film production, game production, virtual reality, and robotics. The following uses several application scenarios as examples to illustrate the application of the action generation model:

[0156] Figure 8 This is a flowchart of a method for generating character actions in a virtual reality scene provided by this application. Figure 8 As shown, the method may include the following steps:

[0157] S301, obtaining a target text, wherein the target text is used to describe the actions of a target character in a virtual reality scene at multiple consecutive moments.

[0158] It should be understood that the present application does not limit the execution subject of the "character action generation method in a virtual reality scene". Still taking the execution subject as any electronic device with processing capability as an example, the implementation method of the electronic device obtaining the target text can refer to the method described in the above embodiment, or refer to any existing implementation method of obtaining the target text used to describe the actions of the target character in a virtual reality scene at multiple consecutive moments.

[0159] S302. Obtain a target coding feature map based on the target text through an action generation model, and generate a target action sequence based on the target coding feature map.

[0160] The target encoding feature map may include: joint vectors of the target person's joints at each moment. The joint vectors may be used to characterize the spatial position of the joints. The target action sequence is used to characterize the actions of the target person at multiple consecutive moments.

[0161] The action generation model may refer to the action generation model described in any of the aforementioned embodiments, and will not be described in detail here.

[0162] In the present embodiment, based on the target text for describing the actions of the target person in the virtual reality scene at multiple consecutive moments, the target coding feature map obtained may include the joint vectors of the joints of the target person at each moment. Through the above method, the spatial position of each joint of the target person can be quantified as a joint vector. Therefore, compared with the prior art in which the overall human posture is represented by a vector, the present application improves the precision of quantifying the action posture of the target person, and the joint vectors of different joints in the target coding feature map can be used to describe the spatial positions of different joints, thereby improving the accuracy of quantifying the actions of the target person at multiple consecutive moments, and then the accuracy of the subsequent generation of the target action sequence (i.e., action generation) for representing the actions of the target person at the multiple consecutive moments based on the target coding feature map can be improved.

[0163] Fig. 9 A schematic diagram of a video generation method provided in this application. Fig. 9 As shown, the method may include the following steps:

[0164] S401: Obtain target text, wherein the target text is used to describe the actions of the target object at multiple consecutive moments.

[0165] It should be understood that the present application does not limit the execution subject of the "video generation method". Still taking the execution subject as any electronic device with processing capability as an example, the implementation method and target object of the electronic device to obtain the target text can refer to the above embodiments.

[0166] S402. Obtain a target coding feature map based on the target text through an action generation model, and generate a target action sequence based on the target coding feature map.

[0167] The target encoding feature map includes: the joint vectors of the joints of the target object at each moment. The joint vectors are used to represent the spatial position of the joints. The target action sequence is used to represent the actions of the target object at multiple consecutive moments.

[0168] The action generation model may refer to the action generation model described in any of the aforementioned embodiments, and will not be described in detail here.

[0169] S403: Generate a target video according to the target action sequence.

[0170] The content of the target video is the actions of the target object at multiple consecutive moments.

[0171] Optionally, the electronic device may generate a target video based on a target action sequence by referring to any existing method for generating a video based on an action sequence.

[0172] It should be understood that the application scenario of the video generation method is not limited in the present application. Taking the application of the video generation method to movie production as an example, the target video can be, for example, a movie clip. Taking the application of the video generation method to game production as an example, the target video can be, for example, a game clip, etc.

[0173] Optionally, after generating the target video, the electronic device may also output the target video, for example, so that the user can view the target video.

[0174] In the present embodiment, based on the target text for describing the action of the target object at multiple continuous moments, the target coding feature map obtained may include the joint vectors of the joints of the target object at each moment. Through the above method, the spatial position of each joint of the target object can be quantified as a joint vector. Therefore, compared with the prior art in which a vector is used to represent the overall human posture, the present application improves the precision of quantifying the action posture of the target object, and the joint vectors of different joints in the target coding feature map can be used to describe the spatial positions of different joints, thereby improving the accuracy of quantifying the action of the target object at multiple continuous moments, and then the accuracy of the target action sequence generated based on the target coding feature map for representing the action of the target object at the multiple continuous moments can be improved. The target video is generated by a target action sequence with higher accuracy, so the accuracy of the content of the target video can be improved, and the user experience is improved.

[0175] Fig.10 This is a flow chart of a method for determining robot actions provided in this application. Fig.10 As shown, the method may include the following steps:

[0176] S501, obtaining a target text, wherein the target text can be used to describe an action to be performed by the target robot.

[0177] It should be understood that the present application does not limit the execution subject of the "robot action determination method". Exemplarily, the execution subject can be, for example, the target robot or other electronic devices with processing capabilities. Taking the execution subject of the method as an electronic device as an example, the implementation method of the electronic device obtaining the target text can refer to the aforementioned embodiment. Alternatively, taking the execution subject of the method as a target robot as an example, the implementation method of the target robot obtaining the target text can refer to the existing method of a robot obtaining a text describing the action to be performed by the robot.

[0178] S502. Obtain a target coding feature map based on the target text through an action generation model, and generate a target action sequence based on the target coding feature map.

[0179] The target coding feature map may include, for example, joint vectors of the joints of the target robot at multiple consecutive moments. The joint vectors may be used to characterize the spatial position of the joints. The target action sequence may be used to characterize the actions of the target robot at multiple consecutive moments.

[0180] The action generation model may refer to the action generation model described in any of the aforementioned embodiments, and will not be described in detail here.

[0181] Taking the target robot as an example, after generating the above-mentioned target action sequence, the target robot can, for example, perform the action represented by the target action sequence. Alternatively, in some embodiments, the target robot can also determine whether to perform the action represented by the target action sequence based on the target action sequence and information about the environment in which the target robot is located (such as the position and shape of obstacles, etc.). If it is determined to be performed, the action represented by the target action sequence can be performed. If it is determined not to be performed, for example, a prompt message can be output to prompt the user that the action described in the target text cannot be performed.

[0182] Alternatively, taking the execution subject of the method as an electronic device as an example, the electronic device can, after generating the target action sequence, send the target action sequence to a target robot, and the target robot can, for example, receive the target action sequence and execute the action represented by the target action sequence.

[0183] In the present embodiment, based on the target text for describing the action to be performed by the target robot, the target coding feature map obtained may include the joint vectors of the joints of the target robot at each moment. Through the above method, the spatial position of each joint of the target robot can be quantified as a joint vector. Therefore, compared with the prior art in which a vector is used to represent the overall human posture, the present application improves the precision of quantifying the action posture of the target robot, and the joint vectors of different joints in the target coding feature map can be used to describe the spatial positions of different joints, thereby improving the accuracy of quantifying the action of the target robot at multiple consecutive moments, and then the accuracy of the target action sequence generated based on the target coding feature map to represent the action of the target robot at the multiple consecutive moments can be improved.

[0184] Fig.11 Schematic diagram of the structure of an action generation model training device 60 provided in this application. Fig.11 As shown, the action generation model training device 60 may include: an acquisition module 61 and a training module 62.

[0185] The acquisition module 31 is used to acquire a first sample data set. The first sample data set includes: a sample text and a first sample encoding feature map corresponding to the sample text; the sample text is used to describe the action of the first sample object at multiple consecutive moments, and the first sample encoding feature map includes: a joint vector of the joint of the first sample object at each of the moments, and the joint vector is used to represent the spatial position of the joint.

[0186] The training module 62 is used to train the first preset model according to the first sample data set to obtain an action generation model. The action generation model is used to obtain a target coding feature map based on the target text, and output a target action sequence based on the target coding feature map. The target text is used to describe the action of the target object at multiple consecutive moments, and the target coding feature map includes: the joint vectors of the joints of the target object at each of the moments; the target action sequence is used to characterize the action of the target object at the multiple consecutive moments.

[0187] Taking the first preset model including: a text feature extraction module, a decoding module and a first sub-model to be trained as an example, optionally, the training module 62 is specifically used to extract text features of the sample text through the text feature extraction module to obtain the text features of the sample text; according to the text features of the sample text and the first sample encoding feature map, the first sub-model to be trained is trained to obtain a joint feature extraction module. Among them, the action generation model includes: the text feature extraction module, the joint feature extraction module and the decoding module; the text feature extraction module is used to extract text features of the target text to obtain the text features of the target text; the joint feature extraction module is used to encode the text features of the target text to obtain the target encoding feature map; the decoding module is used to decode the target encoding feature map to obtain the target action sequence.

[0188] Optionally, the training module 62 is specifically used to repeatedly perform the following steps until a first preset iteration condition is reached: masking at least one of the first sample coding feature maps to obtain a masked coding feature map; obtaining a predicted coding feature map based on the masked coding feature map and the text features of the sample text; calculating the loss function of the first sub-model based on the first sample coding feature map and the predicted coding feature map, and updating the parameters of the first sub-model according to the loss function of the first sub-model. The first sub-model obtained when the first preset iteration condition is reached is the joint feature extraction module.

[0189] Optionally, the training module 62 is specifically used to perform mask processing of the time dimension and / or the space dimension on at least one of the first sample coding feature maps to obtain the masked coding feature map. The mask processing of the time dimension includes: masking the joint vector corresponding to at least one moment in a plurality of consecutive moments; the mask processing of the space dimension includes: masking the joint vector corresponding to at least one joint of the first sample object at at least one moment.

[0190] Optionally, the training module 62 is specifically used to mask the joint vector corresponding to at least one of the multiple consecutive moments in the first sample coding feature map to obtain a coding feature map masked in the time dimension; and mask the joint vector corresponding to at least one joint of the first sample object at at least one moment in the coding feature map masked in the time dimension to obtain the masked coding feature map.

[0191] Optionally, the training module 62 is specifically used to add attention in the time dimension and / or in the space dimension to the masked coding feature map according to the text features of the sample text, so as to obtain the coding feature map after adding the attention mechanism; and obtain the predicted coding feature map according to the coding feature map after adding the attention mechanism.

[0192] Optionally, the training module 62 is specifically used to expand the masked coded feature map into a one-dimensional feature map according to the text feature, and add one-dimensional attention to the one-dimensional feature map based on the position embedding feature to obtain a one-dimensional attention mechanism processing result; based on the position embedding feature, add spatial dimension attention to the one-dimensional attention mechanism processing result to obtain a spatial dimension attention mechanism processing result; add time dimension attention to the spatial dimension attention mechanism processing result to obtain the coded feature map after adding the attention mechanism. Wherein, the position embedding feature is used to describe the position of each joint vector in the first sample coded feature map.

[0193] Optionally, the acquisition module 61 is specifically used to acquire a first sample action sequence corresponding to the sample text, and encode the first sample action sequence to obtain the first sample encoding feature map, wherein the first sample action sequence is used to represent the action of the first sample object at multiple consecutive moments.

[0194] Optionally, the acquisition module 61 is specifically configured to encode the first sample action sequence using an encoder in a preset action quantization model to obtain the first sample encoding feature map.

[0195] Optionally, the acquisition module 61 is further used to acquire at least one second sample action sequence. The second sample action sequence is used to characterize the action of the second sample object at multiple consecutive moments. Optionally, the training module 62 is further used to train the second preset model according to the at least one second sample action sequence to obtain the action quantization model. The second preset model includes a coding layer and a decoding layer, and both the coding layer and the decoding layer are two-dimensional convolutional layers.

[0196] Optionally, the training module 62 is specifically used to repeatedly perform the following steps until a second preset iteration condition is reached: encoding the second sample action sequence based on the encoding layer to obtain a second sample encoding feature map, decoding the second sample encoding feature map based on the decoding layer to obtain a predicted action sequence; determining the loss function of the second preset model based on the predicted action sequence and the second sample action sequence, and updating the parameters of the second preset model according to the loss function of the second preset model. Wherein, the second sample encoding feature map includes: the joint vectors of the joints of the second sample object at each of the said moments; wherein, the second preset model when the second preset iteration condition is reached is the action quantization model.

[0197] The action generation model training device 60 provided in the present application is used to execute the aforementioned action generation model training method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0198] Fig.12 Schematic diagram of the structure of an action generation device 70 provided in this application. Fig.12 As shown, the action generating device 70 may include: an acquisition module 71 and a processing module 72 .

[0199] The acquisition module 71 is used to acquire the target text, wherein the target text is used to describe the actions of the target object at multiple consecutive moments.

[0200] The processing module 72 is used to obtain a target coding feature map based on the target text through an action generation model, and generate a target action sequence based on the target coding feature map. The target coding feature map includes: joint vectors of the joints of the target object at each time; the joint vectors are used to represent the spatial positions of the joints; and the target action sequence is used to represent the actions of the target object at the multiple consecutive times.

[0201] Taking the action generation model including: a text feature extraction module, a joint feature extraction module and a decoding module as an example, optionally, the processing module 72 is specifically used to extract text features of the target text through the text feature extraction module to obtain the text features of the target text; encode the text features through the joint feature extraction module to obtain the target encoding feature map; decode the target encoding feature map through the decoding module to obtain the target action sequence.

[0202] The action generation device 70 provided in the present application is used to execute the aforementioned action generation method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail.

[0203] Fig.13 Schematic diagram of the structure of a character action generation device 80 in a virtual reality scene provided by the present application. Fig.13 As shown, the character action generation device 80 in the virtual reality scene may include: an acquisition module 81 and a processing module 82.

[0204] The acquisition module 81 is used to acquire the target text, wherein the target text is used to describe the actions of the target character in the virtual reality scene at multiple consecutive moments.

[0205] The processing module 82 is used to obtain a target coding feature map based on the target text through an action generation model, and generate a target action sequence based on the target coding feature map. The target coding feature map includes: joint vectors of the joints of the target person at each time; the joint vectors are used to represent the spatial positions of the joints; and the target action sequence is used to represent the actions of the target person at the multiple consecutive times.

[0206] The character action generation device 80 in a virtual reality scene provided in the present application is used to execute the aforementioned character action generation method embodiment in a virtual reality scene. Its implementation principle and technical effect are similar and will not be described in detail.

[0207] Fig.14 FIG. 9 is a schematic diagram of the structure of a video generating device 90 provided in this application. Fig.14 As shown, the video generating device 90 may include: an acquisition module 91 , a processing module 92 and a generating module 93 .

[0208] The acquisition module 91 is used to acquire the target text; the target text is used to describe the actions of the target object at multiple consecutive moments.

[0209] The processing module 92 is used to obtain a target coding feature map based on the target text through an action generation model, and generate a target action sequence based on the target coding feature map. The target coding feature map includes: joint vectors of the joints of the target object at each time; the joint vectors are used to represent the spatial position of the joints; and the target action sequence is used to represent the actions of the target object at the multiple consecutive times.

[0210] The generating module 93 is used to generate a target video according to the target action sequence, wherein the content of the target video is the action of the target object at the plurality of consecutive moments.

[0211] The video generating device 90 provided in the present application is used to execute the aforementioned video generating method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail.

[0212] Fig.15 Schematic diagram of the structure of a robot action determination device 150 provided in this application. Fig.15 As shown, the robot action determination device 150 may include: an acquisition module 151 and a processing module 152.

[0213] The acquisition module 151 is used to acquire a target text, wherein the target text is used to describe the action to be performed by the target robot.

[0214] The processing module 152 is used to obtain a target coding feature map based on the target text through an action generation model, and generate a target action sequence based on the target coding feature map. The target coding feature map includes: joint vectors of the joints of the target robot at multiple consecutive moments; the joint vectors are used to represent the spatial positions of the joints; and the target action sequence is used to represent the actions of the target robot at the multiple consecutive moments.

[0215] The robot action determination device 150 provided in the present application is used to execute the aforementioned robot action determination method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0216] Fig.16 A schematic diagram of the hardware structure of an electronic device provided in this application. Fig.16The electronic device 160 shown includes a memory 161, a processor 162, and a communication interface 163. The memory 161, the processor 162, and the communication interface 163 are connected to each other in communication. For example, the memory 161, the processor 162, and the communication interface 163 can be connected to each other through a network connection. Alternatively, the electronic device 160 may further include a bus 164. The memory 161, the processor 162, and the communication interface 163 are connected to each other in communication through the bus 164. Fig.16 The electronic device 160 is a memory 161 , a processor 162 , and a communication interface 163 that are communicatively connected to each other via a bus 164 .

[0217] The memory 161 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 161 may store a program. When the program stored in the memory 161 is executed by the processor 162, the processor 162 and the communication interface 163 are used to execute at least one of the methods of action generation model training, action generation, robot action determination, video generation, and character action generation in a virtual reality scene described in any of the aforementioned embodiments. The memory may also store data required for at least one of the methods of action generation model training, action generation, robot action determination, video generation, and character action generation in a virtual reality scene.

[0218] The processor 162 may be a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

[0219] The processor 162 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, at least one of the action generation model training, action generation, robot action determination, video generation, and character action generation in a virtual reality scene of the present application can be completed by an integrated logic circuit of hardware or software instructions in the processor 162. The above-mentioned processor 162 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and may implement or execute the methods, steps, and logic block diagrams disclosed in the following embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the following embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 161, and the processor 162 reads the information in the memory 161 and completes the road information processing of the present application in combination with its hardware.

[0220] The communication interface 163 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the electronic device 160 and other devices or a communication network. For example, a data set can be acquired through the communication interface 163 .

[0221] When the electronic device 160 includes a bus 164 , the bus 164 may include a path for transmitting information between various components of the electronic device 160 (eg, the memory 161 , the processor 162 , and the communication interface 163 ).

[0222] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the above embodiments.

[0223] The present application also provides a program product, which includes an execution instruction, which is stored in a readable storage medium. At least one processor of the electronic device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the electronic device implements at least one method of action generation model training, action generation, robot action determination, video generation, and character action generation in a virtual reality scene provided by the various embodiments described above.

[0224] The term "plurality" in this article refers to two or more than two. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0225] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an action generation model, characterized in that: The method comprises: Acquire a first sample data set; the first sample data set includes: sample text and a first sample encoding feature map corresponding to the sample text; the sample text is used to describe the action of the first sample object at multiple consecutive moments, and the first sample encoding feature map includes: joint vectors of the joints of the first sample object at each of the moments, and the joint vectors are used to represent the spatial position of the joints; According to the first sample data set, a first preset model is trained to obtain an action generation model; the action generation model is used to obtain a target coding feature map based on a target text, and output a target action sequence based on the target coding feature map; wherein the target text is used to describe the action of the target object at multiple consecutive moments, and the target coding feature map includes: joint vectors of the joints of the target object at each of the said moments; the target action sequence is used to characterize the action of the target object at the said multiple consecutive moments.

2. The method according to claim 1, characterized in that The first preset model includes: a text feature extraction module, a decoding module and a first sub-model to be trained. The first preset model is trained according to the first sample data set to obtain an action generation model, including: By means of the text feature extraction module, text feature extraction is performed on the sample text to obtain text features of the sample text; According to the text features of the sample text and the first sample encoding feature map, the first sub-model to be trained is trained to obtain a joint feature extraction module; The action generation model comprises: the text feature extraction module, the joint feature extraction module and the decoding module; The text feature extraction module is used to extract text features from the target text to obtain text features of the target text; the joint feature extraction module is used to encode the text features of the target text to obtain the target encoding feature map; the decoding module is used to decode the target encoding feature map to obtain the target action sequence.

3. The method according to claim 2, characterized in that The step of training the first sub-model to be trained according to the text features of the sample text and the first sample encoding feature map to obtain a joint feature extraction module includes: Repeat the following steps until the first preset iteration condition is reached: Performing mask processing on at least one of the first sample coding feature maps to obtain a masked coding feature map; Obtaining a predicted coding feature map according to the masked coding feature map and the text features of the sample text; Calculating a loss function of the first sub-model based on the first sample coding feature map and the predicted coding feature map, and updating parameters of the first sub-model according to the loss function of the first sub-model; Among them, the first sub-model obtained when the first preset iteration condition is reached is the joint feature extraction module.

4. The method according to claim 3, characterized in that: The performing masking on at least one of the first sample coding feature maps to obtain a masked coding feature map includes: Mask processing of the time dimension and / or mask processing of the space dimension are performed on at least one of the first sample coding feature maps to obtain the masked coding feature map; the mask processing of the time dimension includes: masking processing of the joint vector corresponding to at least one moment among the multiple consecutive moments; the mask processing of the space dimension includes: masking processing of the joint vector corresponding to at least one joint of the first sample object at at least one moment.

5. The method according to claim 4, characterized in that Performing mask processing of the time dimension and the space dimension on at least one of the first sample coding feature maps to obtain the masked coding feature map includes: Perform mask processing on the joint vector corresponding to the at least one moment in the first sample encoding feature map to obtain an encoding feature map masked in the time dimension; The joint vector corresponding to the at least one joint of the first sample object at at least one moment in the coded feature map of the time dimension mask is masked to obtain the masked coded feature map.

6. The method according to any one of claims 3 to 5, characterized in that: The step of obtaining a predicted coding feature map according to the masked coding feature map and the text features of the sample text includes: According to the text features of the sample text, adding attention in the time dimension and / or adding attention in the space dimension to the masked encoding feature map to obtain an encoding feature map after adding the attention mechanism; According to the encoding feature map after adding the attention mechanism, the predicted encoding feature map is obtained.

7. The method according to claim 6, characterized in that According to the text features of the sample text, attention is added in the time dimension and attention is added in the space dimension to the masked encoding feature map to obtain an encoding feature map after adding the attention mechanism, including: According to the text features, the masked encoded feature map is expanded into a one-dimensional feature map, and based on the position embedding features, one-dimensional attention is added to the one-dimensional feature map to obtain a one-dimensional attention mechanism processing result; Based on the position embedding feature, adding spatial dimension attention to the one-dimensional attention mechanism processing result to obtain the spatial dimension attention mechanism processing result; Adding time dimension attention to the processing result of the spatial dimension attention mechanism to obtain the encoded feature map after adding the attention mechanism; The position embedding feature is used to describe the position of each joint vector in the first sample encoding feature map.

8. The method according to any one of claims 1 to 5, characterized in that: The obtaining of the first sample data set includes: Acquire a first sample action sequence corresponding to the sample text; the first sample action sequence is used to represent the action of the first sample object at multiple consecutive moments; The first sample action sequence is encoded to obtain the first sample encoding feature map.

9. The method according to claim 8, characterized in that The step of encoding the first sample action sequence to obtain the first sample encoding feature graph includes: The first sample action sequence is encoded using an encoder in a preset action quantization model to obtain the first sample encoding feature map.

10. The method according to claim 9, characterized in that The method further comprises: Acquire at least one second sample action sequence; the second sample action sequence is used to represent the action of the second sample object at multiple consecutive moments; According to the at least one second sample action sequence, a second preset model is trained to obtain the action quantization model; the second preset model includes a coding layer and a decoding layer, and both the coding layer and the decoding layer are two-dimensional convolutional layers.

11. The method according to claim 10, characterized in that The step of training the second preset model according to the at least one second sample action sequence to obtain the action quantization model includes: Repeat the following steps until the second preset iteration condition is reached: Encoding the second sample action sequence based on the encoding layer to obtain a second sample encoding feature map, wherein the second sample encoding feature map includes: joint vectors of the joints of the second sample object at each of the moments; Decoding the second sample encoding feature map based on the decoding layer to obtain a predicted action sequence; Based on the predicted action sequence and the second sample action sequence, determining a loss function of the second preset model, and updating parameters of the second preset model according to the loss function of the second preset model; Among them, the second preset model when the second preset iteration condition is reached is the action quantization model.

12. An action generation method, characterized in that: The method comprises: Obtaining a target text; the target text is used to describe the actions of the target object at multiple consecutive moments; Through the action generation model, based on the target text, a target coding feature map is obtained, and based on the target coding feature map, a target action sequence is generated; wherein, the target coding feature map includes: joint vectors of the joints of the target object at each of the moments; the joint vectors are used to characterize the spatial positions of the joints; and the target action sequence is used to characterize the actions of the target object at the multiple consecutive moments.

13. The method according to claim 12, characterized in that The action generation model includes: a text feature extraction module, a joint feature extraction module and a decoding module. The action generation model is used to obtain a target encoding feature map based on the target text, and based on the target encoding feature map, generating a target action sequence includes: By means of the text feature extraction module, text feature extraction is performed on the target text to obtain text features of the target text; By means of the joint feature extraction module, the text features are encoded to obtain the target encoding feature map; The target coding feature map is decoded by the decoding module to obtain the target action sequence.

14. A method for generating character actions in a virtual reality scene, characterized in that: The method comprises: Obtaining a target text; the target text is used to describe the actions of the target character in the virtual reality scene at multiple consecutive moments; Through the action generation model, based on the target text, a target coding feature map is obtained, and based on the target coding feature map, a target action sequence is generated; wherein, the target coding feature map includes: joint vectors of the joints of the target character at each of the moments; the joint vectors are used to characterize the spatial position of the joints; and the target action sequence is used to characterize the actions of the target character at the multiple consecutive moments.

15. A video generation method, characterized in that: The method comprises: Obtaining a target text; the target text is used to describe the actions of the target object at multiple consecutive moments; A target coding feature map is obtained based on the target text through an action generation model, and a target action sequence is generated based on the target coding feature map; wherein the target coding feature map includes: joint vectors of the joints of the target object at each time; the joint vectors are used to represent the spatial positions of the joints; and the target action sequence is used to represent the actions of the target object at the multiple consecutive times; A target video is generated according to the target action sequence; the content of the target video is the actions of the target object at the multiple consecutive moments.

16. A method for determining a robot action, characterized in that: The method comprises: Obtaining a target text; the target text is used to describe the action to be performed by the target robot; Through the action generation model, based on the target text, a target coding feature map is obtained, and based on the target coding feature map, a target action sequence is generated; wherein, the target coding feature map includes: joint vectors of the joints of the target robot at multiple consecutive moments; the joint vectors are used to characterize the spatial positions of the joints; and the target action sequence is used to characterize the actions of the target robot at the multiple consecutive moments.

17. An electronic device, characterized in that: include: Processor and memory; The processor is communicatively connected to the memory; The memory stores computer instructions; The processor executes the computer instructions stored in the memory to implement the method according to any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 16.

19. A computer program product, characterized in that The computer program product comprises a computer program, which implements the method according to any one of claims 1 to 16 when executed by a processor.

Images