Method for training action generation model, action generation method and device

Abstract

The present disclosure provides a method for training an action generation model, an action generation method and device, and relates to the technical field of computers. The method comprises: obtaining a first audio sample, a first action image sample, a second audio sample and a second action image sample; training an audio encoder and an audio quantizer using the first audio sample to obtain a trained audio encoder and audio quantizer; training an action quantizer and an action decoder using the first action image sample to obtain a trained action quantizer and action decoder; training an audio-action converter using the second audio sample and the second action image sample to obtain a trained audio-action converter; and sequentially connecting the trained audio encoder, the audio quantizer, the audio-action converter, the action quantizer and the action decoder to generate an action generation model. The method solves the problems of insufficient model training and poor generalization performance in related technologies, and improves the generalization performance of the model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method for an action generation model, an action generation method, a training device for an action generation model, an action generation device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Generating actions based on music to drive virtual characters to sing and dance can be applied to animation generation, human-computer interaction, and multimedia intelligent creation. Related technologies train a generative model based on music-action pairwise data, and then use this model to obtain actions that match the music. However, models trained on music-action pairwise data suffer from the difficulty of collecting these pairs, leading to insufficient training and poor generalization performance in models trained with limited pairs of data.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] This disclosure provides a training method for an action generation model, an action generation method, a training device for an action generation model, an action generation device, an electronic device, and a computer-readable storage medium, to at least solve the problems of insufficient model training and poor generalization performance in related technologies.

[0005] According to one aspect of the present disclosure, a method for training an action generation model is provided, comprising: acquiring a first audio sample, a first motion image sample, a second audio sample, and a second motion image sample corresponding to the second audio sample; sequentially inputting the first audio sample into an audio encoder, an audio quantizer, and an audio decoder to obtain an audio reconstruction sequence; training the audio encoder and the audio quantizer based on the first audio sample and the audio reconstruction sequence to obtain a trained audio encoder and a trained audio quantizer; sequentially inputting the first motion image sample into an action encoder, an action quantizer, and an action decoder to obtain an action reconstruction sequence; training the action quantizer and the action decoder based on the first motion image sample and the action reconstruction sequence to obtain a trained action quantizer and a trained action decoder; inputting the second audio sample into an audio-action converter to obtain a predicted action sequence; training the audio-action converter based on the second motion image sample and the predicted action sequence to obtain a trained audio-action converter; and sequentially connecting the trained audio encoder, the trained audio quantizer, the trained audio-action converter, the trained action quantizer, and the trained action decoder to obtain an action generation model.

[0006] In some embodiments of this disclosure, the step of sequentially inputting the first audio sample into an audio encoder, an audio quantizer, and an audio decoder to obtain an audio reconstruction sequence, and training the audio encoder and the audio quantizer based on the first audio sample and the audio reconstruction sequence to obtain a trained audio encoder and a trained audio quantizer sample, includes: obtaining a feature sequence of the first audio sample; inputting the feature sequence of the first audio sample into the audio encoder to obtain an encoding vector of the feature sequence of the first audio sample; inputting the encoding vector of the feature sequence of the first audio sample into the audio quantizer to obtain a quantized encoding vector of the feature sequence of the first audio sample; inputting the quantized encoding vector of the feature sequence of the first audio sample into the audio decoder to obtain an audio reconstruction sequence of the first audio sample; calculating a first loss based on the feature sequence of the first audio sample, the encoding vector of the feature sequence of the first audio sample, the quantized encoding vector of the feature sequence of the first audio sample, and the audio reconstruction sequence; adjusting the parameters of the audio encoder, the parameters of the audio quantizer, and the parameters of the audio decoder based on the first loss until the calculated first loss is less than a preset value, thereby obtaining the trained audio encoder, the trained audio quantizer, and the trained audio decoder.

[0007] In some embodiments of this disclosure, the step of sequentially inputting the first motion image sample into a motion encoder, a motion quantizer, and a motion decoder to obtain a motion reconstruction sequence, and training the motion quantizer and the motion decoder based on the first motion image sample and the motion reconstruction sequence to obtain a trained motion quantizer and a trained motion decoder includes: acquiring a feature sequence of the first motion image sample; inputting the feature sequence of the first motion image sample into the motion encoder to obtain an encoding vector of the feature sequence of the first motion image sample; inputting the encoding vector of the feature sequence of the first motion image sample into the motion quantizer to obtain a quantized encoding vector of the feature sequence of the first motion image sample; inputting the quantized encoding vector of the feature sequence of the first motion image sample into the motion decoder to obtain a motion reconstruction sequence of the first motion image sample; calculating a second loss based on the feature sequence of the first motion image sample, the encoding vector of the feature sequence of the first motion image sample, the quantized encoding vector of the first motion image sample, and the motion reconstruction sequence; adjusting the parameters of the motion encoder, the parameters of the motion quantizer, and the parameters of the motion decoder based on the second loss until the calculated second loss is less than a preset value, thereby obtaining a trained motion encoder, a trained motion quantizer, and a trained motion decoder.

[0008] In some embodiments of this disclosure, the step of inputting the second audio sample into an audio-action converter to obtain a predicted action sequence, and training the audio-action converter based on the second action image sample and the predicted action sequence to obtain a trained audio-action converter includes: acquiring the feature sequence of the second audio sample and the feature sequence of the second action image sample; encoding and quantizing the feature sequence of the second audio sample to obtain a quantized encoding vector of the feature sequence of the second audio sample; encoding and quantizing the feature sequence of the second action image sample to obtain a quantized encoding vector of the feature sequence of the second action image sample; inputting the quantized encoding vector of the feature sequence of the second audio sample into the encoder of the audio-action converter to obtain an encoding vector; inputting the encoding vector into the decoder of the audio-action converter to obtain the predicted action sequence; calculating a third loss based on the predicted action sequence and the quantized encoding vector of the feature sequence of the second action image sample; adjusting the parameters of the encoder and the decoder of the audio-action converter based on the third loss until the calculated third loss is less than a preset value, thereby obtaining the trained audio-action converter.

[0009] In some embodiments of this disclosure, the predicted action sequence includes multiple action symbols; wherein, the step of inputting the encoded vector into the decoder in the audio action converter to obtain the predicted action sequence includes: for the current action symbol, inputting the encoded vector and the action prefix corresponding to the current action symbol into the decoder in the audio action converter to obtain probability estimates of each candidate action symbol, wherein the action prefix corresponding to the current action symbol is the action symbol preceding the current action symbol in the predicted action sequence; and determining the current action symbol from the candidate action symbols based on the probability estimates of each candidate action symbol.

[0010] According to another aspect of the present disclosure, an action generation method is provided, the method comprising: acquiring an audio feature sequence to be processed; inputting the audio feature sequence to be processed into an action generation model obtained according to any of the above methods to obtain an action feature sequence corresponding to the audio feature sequence to be processed.

[0011] According to another aspect of the present disclosure, a training apparatus for an action generation model is provided, comprising: a sample acquisition module, configured to acquire a first audio sample, a first motion image sample, a second audio sample, and a second motion image sample corresponding to the second audio sample; a first training module, configured to sequentially input the first audio sample into an audio encoder, an audio quantizer, and an audio decoder to obtain an audio reconstruction sequence, and train the audio encoder and the audio quantizer based on the first audio sample and the audio reconstruction sequence to obtain a trained audio encoder and a trained audio quantizer; and a second training module, configured to sequentially input the first motion image sample into an action encoder, an action quantizer, and an action decoder to obtain an action image sample. A third training module is used to input the second audio sample into the audio-action converter to obtain a predicted action sequence, and to train the audio-action converter based on the second action image sample and the predicted action sequence to obtain a trained audio-action converter. A model generation module is used to sequentially connect the trained audio encoder, the trained audio quantizer, the trained audio-action converter, the trained action quantizer, and the trained action decoder to obtain an action generation model.

[0012] In some embodiments of this disclosure, the first training module is further configured to: acquire the feature sequence of the first audio sample; input the feature sequence of the first audio sample to the audio encoder to obtain the encoding vector of the feature sequence of the first audio sample; input the encoding vector of the feature sequence of the first audio sample to the audio quantizer to obtain the quantized encoding vector of the feature sequence of the first audio sample; input the quantized encoding vector of the feature sequence of the first audio sample to the audio decoder to obtain the audio reconstruction sequence of the first audio sample; calculate a first loss based on the feature sequence of the first audio sample, the encoding vector of the feature sequence of the first audio sample, the quantized encoding vector of the feature sequence of the first audio sample, and the audio reconstruction sequence; adjust the parameters of the audio encoder, the parameters of the audio quantizer, and the parameters of the audio decoder based on the first loss until the calculated first loss is less than a preset value, thereby obtaining the trained audio encoder, the trained audio quantizer, and the trained audio decoder.

[0013] In some embodiments of this disclosure, the second training module is further configured to: acquire the feature sequence of the first motion image sample; input the feature sequence of the first motion image sample to the motion encoder to obtain the encoding vector of the feature sequence of the first motion image sample; input the encoding vector of the feature sequence of the first motion image sample to the motion quantizer to obtain the quantized encoding vector of the feature sequence of the first motion image sample; input the quantized encoding vector of the feature sequence of the first motion image sample to the motion decoder to obtain the motion reconstruction sequence of the first motion image sample; calculate a second loss based on the feature sequence of the first motion image sample, the encoding vector of the feature sequence of the first motion image sample, the quantized encoding vector of the first motion image sample, and the motion reconstruction sequence; adjust the parameters of the motion encoder, the parameters of the motion quantizer, and the parameters of the motion decoder based on the second loss until the calculated second loss is less than a preset value, thereby obtaining the trained motion encoder, the trained motion quantizer, and the trained motion decoder.

[0014] In some embodiments of this disclosure, the third training module is further configured to: acquire the feature sequence of the second audio sample and the feature sequence of the second motion image sample; encode and quantize the feature sequence of the second audio sample to obtain a quantized encoding vector of the feature sequence of the second audio sample; encode and quantize the feature sequence of the second motion image sample to obtain a quantized encoding vector of the feature sequence of the second motion image sample; input the quantized encoding vector of the feature sequence of the second audio sample to the encoder in the audio motion converter to obtain an encoding vector; input the encoding vector to the decoder in the audio motion converter to obtain the predicted motion sequence; calculate a third loss based on the predicted motion sequence and the quantized encoding vector of the feature sequence of the second motion image sample; adjust the parameters of the encoder and the decoder in the audio motion converter according to the third loss until the calculated third loss is less than a preset value, thereby obtaining the trained audio motion converter.

[0015] In some embodiments of this disclosure, the predicted action sequence includes multiple action symbols; wherein, the third training module is further configured to: for the current action symbol, input the encoding vector and the action prefix corresponding to the current action symbol into the decoder in the audio action converter to obtain probability estimates of each candidate action symbol, wherein the action prefix corresponding to the current action symbol is the action symbol preceding the current action symbol in the predicted action sequence; and determine the current action symbol from the candidate action symbols based on the probability estimates of each candidate action symbol.

[0016] According to another aspect of the present disclosure, an action generation apparatus is provided, the apparatus comprising: an audio sequence acquisition module for acquiring an audio feature sequence to be processed; and an action sequence generation module for inputting the audio feature sequence to be processed into an action generation model obtained according to any of the above methods to obtain an action feature sequence corresponding to the audio feature sequence to be processed.

[0017] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the above-described training method for the action generation model or to implement the above-described action generation method.

[0018] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the above-described training method for the action generation model or implement the above-described action generation method.

[0019] The technical solution provided by the embodiments of this disclosure brings at least the following beneficial effects: It trains an audio encoder and an audio quantizer using audio samples, trains an action quantizer and an action decoder using motion image samples, and trains an audio-action converter using pairs of audio samples and motion image samples. Then, it sequentially connects the trained audio encoder, audio quantizer, audio-action converter, action quantizer, and action decoder to obtain an action generation model. Thus, by using audio samples and motion image samples to train the audio encoder and audio quantizer for processing audio data, and the action quantizer and action decoder for processing motion data, and using paired samples of audio actions to train the audio-action converter, the entire action generation model is trained. This solves the problems of insufficient model training and poor generalization performance in related technologies. Furthermore, the entire action generation model is trained in a semi-supervised manner, improving the model's generalization performance.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0022] Figure 1 This is a schematic diagram of the system architecture of a training method for an action generation model according to an exemplary embodiment;

[0023] Figure 2 This is a flowchart illustrating a training method for an action generation model according to an exemplary embodiment;

[0024] Figure 3 This is a flowchart illustrating, according to an exemplary embodiment, how to train an audio encoder and an audio quantizer using a first audio sample to obtain the trained audio encoder and the trained audio quantizer.

[0025] Figure 4 This is a flowchart illustrating, according to an exemplary embodiment, how to train a trained motion quantizer and a trained motion decoder using a first motion image sample.

[0026] Figure 5 This is a flowchart illustrating the training of an audio motion converter according to an exemplary embodiment;

[0027] Figure 6 This is a schematic diagram of the structure of an action generation model according to an exemplary embodiment;

[0028] Figure 7This is a flowchart illustrating a method for generating actions based on an action generation model according to an exemplary embodiment;

[0029] Figure 8 This is a block diagram of a training apparatus for an action generation model according to an exemplary embodiment;

[0030] Figure 9 This is a block diagram of an action generation device based on an action generation model, according to an exemplary embodiment.

[0031] Figure 10 This is a schematic diagram illustrating the structure of an electronic device suitable for implementing embodiments of the present disclosure, according to an exemplary embodiment. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0033] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0034] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0035] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0036] In this specification, the terms “a,” “an,” “the,” “the,” and “at least one” are used to indicate the presence of at least one element / component / etc.; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0037] Figure 1 This is a schematic diagram of the system architecture of a training method for an action generation model according to an exemplary embodiment. For example... Figure 1 As shown, the system architecture may include a server 101, a network 102, and a client 103. The network 102 serves as the medium for providing a communication link between the client 103 and the server 101. The network 102 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0038] In an exemplary embodiment, the client 103 that transmits data with the server 101 may be, but is not limited to, terminal devices such as smartphones, tablets, laptops, smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, and smart wearable devices. Alternatively, the client 103 may also be a personal computer, such as a laptop computer or a desktop computer. Optionally, the operating system running on the electronic device may include, but is not limited to, Android, iOS, Linux, and Windows.

[0039] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some practical applications, server 101 can also be a server for a network platform, such as a trading platform, live streaming platform, social platform, or audio platform, etc., which is not limited in this embodiment. The server can be a single server or a cluster of multiple servers; the specific architecture of the server is not limited in this disclosure.

[0040] In this embodiment of the disclosure, the client 103 can be a terminal device with various applications installed, such as a terminal device with an animation production application installed, where users can display the generated animation through the animation production application provided by the client 103, or a terminal device with a multimedia application installed, where users can display virtual characters singing and dancing on the display page of the multimedia application provided by the client 103.

[0041] In an exemplary embodiment, the training process of the action generation model by the server 101 may be as follows: The server 101 acquires a first audio sample, a first motion image sample, a second audio sample, and a second motion image sample corresponding to the second audio sample; the server 101 sequentially inputs the first audio sample into an audio encoder, an audio quantizer, and an audio decoder to obtain an audio reconstruction sequence, and trains the audio encoder and audio quantizer based on the first audio sample and the audio reconstruction sequence to obtain a trained audio encoder and a trained audio quantizer; the server 101 sequentially inputs the first motion image sample into an action encoder, an action quantizer, and an action decoder to obtain an action reconstruction sequence, and trains the action quantizer and action decoder based on the first motion image sample and the action reconstruction sequence to obtain a trained action quantizer and a trained action decoder; the server 101 inputs the second audio sample into an audio-action converter to obtain a predicted action sequence, and trains the audio-action converter based on the second motion image sample and the predicted action sequence to obtain a trained audio-action converter; the server 101 sequentially connects the trained audio encoder, the trained audio quantizer, the trained audio-action converter, the trained action quantizer, and the trained action decoder to obtain the action generation model.

[0042] In addition, it should be noted that, Figure 1 The example shown is merely one application environment of the model generation method provided in this disclosure. Figure 1 The number of clients 103, networks 102, and servers 101 shown is merely illustrative; any number of clients, networks, and servers can be used as needed.

[0043] To enable those skilled in the art to better understand the technical solutions of this disclosure, the following will describe in more detail each step of the training method for the action generation model in the example embodiments of this disclosure, in conjunction with the accompanying drawings and examples.

[0044] Figure 2 This is a flowchart illustrating a training method for an action generation model according to an exemplary embodiment. Figure 2 The execution subject of the method provided in the embodiments can be any electronic device, such as... Figure 1 The server 101 in this embodiment is not limited thereto. Figure 2As shown, the training method for the action generation model may include the following steps.

[0045] Step S201: Obtain the first audio sample, the first motion image sample, the second audio sample, and the second motion image sample corresponding to the second audio sample.

[0046] In step S202, the first audio sample is sequentially input into the audio encoder, audio quantizer, and audio decoder to obtain the audio reconstruction sequence. The audio encoder and audio quantizer are trained based on the first audio sample and the audio reconstruction sequence to obtain the trained audio encoder and trained audio quantizer.

[0047] Step S203: The first motion image sample is sequentially input into the motion encoder, motion quantizer and motion decoder to obtain the motion reconstruction sequence. The motion quantizer and motion decoder are trained based on the first motion image sample and the motion reconstruction sequence to obtain the trained motion quantizer and trained motion decoder.

[0048] Step S204: Input the second audio sample into the audio action converter to obtain the predicted action sequence. Train the audio action converter based on the second action image sample and the predicted action sequence to obtain the trained audio action converter.

[0049] In step S205, the trained audio encoder, trained audio quantizer, trained audio motion converter, trained motion quantizer, and trained motion decoder are connected sequentially to obtain the motion generation model.

[0050] The first audio sample can be understood as pure audio training data, that is, it only contains audio data; the first motion image sample can be understood as pure motion image training data, that is, it only contains motion image data; the second audio sample and the second motion image sample are paired training data. The second motion image sample corresponding to the second audio sample can be understood as motion image data that matches the audio data. For example, if the audio being played is classical music, and the matching motion can be classical dance, then classical dance will be displayed.

[0051] The training method for the action generation model provided in this disclosure uses audio samples to train an audio encoder and an audio quantizer, uses motion image samples to train an action quantizer and an action decoder, and uses pairs of audio samples and motion image samples to train an audio-motion converter. The trained audio encoder, audio quantizer, audio-motion converter, action quantizer, and action decoder are then sequentially connected to obtain the action generation model. Thus, by using audio samples and motion image samples to train the audio encoder and audio quantizer for processing audio data, and the action quantizer and action decoder for processing motion data, and using paired samples of audio actions to train the audio-motion converter, the entire action generation model is trained. This solves the problems of insufficient model training and poor generalization performance in related technologies. Furthermore, the entire action generation model is trained in a semi-supervised manner, improving the model's generalization performance.

[0052] To facilitate understanding, the following will be combined with the appendix. Figure 3 To be continued Figure 7 The above steps will be explained in more detail.

[0053] In this embodiment, the first audio sample and the first motion image sample can be trained based on a vector quantization variational autoencoder. The function of the vector quantization variational autoencoder is to learn the representation of the input information by using the input information as the learning target. The vector quantization variational autoencoder includes an encoder, a decoder, and a quantizer. The encoder encodes the input features into latent vectors, the quantizer performs clustering and vector quantization processing on the encoded latent vectors, outputting discretized latent vectors, and the decoder decodes the discretized latent vectors to reconstruct the features.

[0054] Specifically, the vector quantization variational autoencoder is constructed using a pyramid-structured convolutional neural network. The encoder and decoder are both 3-layer 1D convolutional neural networks. The quantizer can use the K-means algorithm for quantization. Let Enc represent the encoder, x be the input feature of the encoder, Quantize represent the quantizer, and the vector quantization codebook be [e1, e2, ..., e...]. k Then, after inputting feature x into the encoder for encoding and quantizer processing, we obtain...

[0055] Quantize(Enc(x)) = e k k = arg min|\Enc(x) - e j |\ Formula 1

[0056] In Formula 1, e k This represents the quantized code, where the quantization criterion is the code closest to the encoder, i.e., e. jIt is a random vector that follows a certain distribution, representing a latent variable, and argmin represents taking the value (Enc(x) - e). j The minimum value in the vector quantization variational autoencoder (i.e., encoder-quantizer-decoder model) is given by k, where k is the index of the minimum value. Then, the quantized encoding is input into the decoder to reconstruct the features. The overall loss function of the vector quantization variational autoencoder is:

[0057]

[0058] In Formula 2, Let represent the overall loss function, Enc(x) represent the encoded vector output after inputting feature x into the encoder, and e represent the quantized encoded vector of the input feature x. The features to be reconstructed are represented by sg, which indicates the gradient cessation operation, and β represents a constant, such as 0.2. The overall loss function can be viewed as three parts: the first part is the reconstruction loss, used to update the encoder and decoder; the second part is the loss for the latent vector, used to update the value of the latent vector; and the third part is a commitment loss, which encourages the encoder's output to stay close to its chosen latent vector, avoiding frequent fluctuations in output from one latent vector to another.

[0059] In this embodiment of the disclosure, an audio vector quantization variational autoencoder can be trained using a first audio sample. The audio vector quantization variational autoencoder includes an audio encoder, an audio quantizer, and an audio decoder. Therefore, a trained audio encoder, a trained audio quantizer, and a trained audio decoder can be obtained. Subsequently, an action generation model can be generated using the trained audio encoder and the trained audio quantizer.

[0060] Figure 3 This is a flowchart illustrating, according to an exemplary embodiment, the training of an audio encoder and an audio quantizer using a first audio sample. Figure 3 As shown, it was trained according to the following process:

[0061] Step S301: Obtain the feature sequence of the first audio sample;

[0062] Step S302: Input the feature sequence of the first audio sample into the audio encoder to obtain the encoding vector of the feature sequence of the first audio sample;

[0063] Step S303: Input the encoding vector of the feature sequence of the first audio sample into the audio quantizer to obtain the quantized encoding vector of the feature sequence of the first audio sample;

[0064] Step S304: Input the quantized encoding vector of the feature sequence of the first audio sample into the audio decoder to obtain the audio reconstruction sequence of the first audio sample;

[0065] Step S305: Calculate the first loss based on the feature sequence of the first audio sample, the encoding vector of the feature sequence of the first audio sample, the quantization encoding vector of the feature sequence of the first audio sample, and the audio reconstruction sequence;

[0066] Step S306: Adjust the parameters of the audio encoder, the audio quantizer, and the audio decoder according to the first loss until the calculated first loss is less than a preset value, and obtain the trained audio encoder, the trained audio quantizer, and the trained audio decoder.

[0067] In step S301, the feature sequence of the first audio sample is obtained. This feature sequence is a vector sequence representing audio features extracted from the audio waveform, such as Mel-Filterbank features and MFCC (Mel frequency cepstral coefficients) features. For example, with a frame length of 25ms and a frame shift of 10ms, extracting 80-dimensional Mel-Filterbank features from 100s of audio can yield 1000 feature vectors.

[0068] In step S302, the feature sequence of the first audio sample is input to the audio encoder, and the input feature sequence is encoded into a latent vector by the audio encoder, thus obtaining the encoded vector of the feature sequence of the first audio sample.

[0069] In step S303, the latent vector output by the encoder is clustered and vector quantized using an audio quantizer, outputting a discrete quantity after quantization, which is the quantized encoding vector of the feature sequence of the first audio sample. After training, the quantized encoding vector of the feature sequence of the audio sample can be clustered to extract representative semantic features.

[0070] In step S304, the quantized encoding vector of the feature sequence of the first audio sample is reconstructed into a feature sequence by the audio decoder.

[0071] In steps S305 and S306, the feature sequence of the first audio sample, the encoding vector of the feature sequence of the first audio sample, the quantization encoding vector of the first audio sample, and the reconstructed feature sequence of the first audio sample are substituted into Formula 2 above to calculate the first loss. Then, the parameters of the audio encoder, the audio quantizer, and the audio decoder are adjusted according to the first loss until the calculated first loss is less than the preset value (i.e., the loss threshold of the audio vector quantization variational autoencoder) and training stops, resulting in the trained audio encoder, the trained audio quantizer, and the trained audio decoder.

[0072] In this embodiment of the disclosure, a motion vector quantization variational autoencoder can be trained using a first motion image sample. The motion vector quantization variational autoencoder includes a motion encoder, a motion quantizer, and a motion decoder. Therefore, a trained motion encoder, a trained motion quantizer, and a trained motion decoder can be obtained. Subsequently, a motion generation model can be generated using the trained motion quantizer and the trained motion decoder.

[0073] Figure 4 This is a flowchart illustrating, according to an exemplary embodiment, the training of a motion quantizer and a motion decoder using first motion image samples. Figure 4 As shown, it was trained according to the following process:

[0074] Step S401: Obtain the feature sequence of the first action image sample;

[0075] Step S402: Input the feature sequence of the first motion image sample into the motion encoder to obtain the encoding vector of the feature sequence of the first motion image sample;

[0076] Step S403: Input the encoding vector of the feature sequence of the first action image sample into the action quantizer to obtain the quantized encoding vector of the feature sequence of the first action image sample.

[0077] Step S404: Input the quantized encoding vector of the feature sequence of the first action image sample into the action decoder to obtain the action reconstruction sequence of the first action image sample;

[0078] Step S405: Calculate the second loss based on the feature sequence of the first action image sample, the encoding vector of the feature sequence of the first action image sample, the quantization encoding vector of the first action image sample, and the action reconstruction sequence;

[0079] Step S406: Adjust the parameters of the action encoder, the action quantizer, and the action decoder according to the second loss until the calculated second loss is less than a preset value, and obtain the trained action encoder, the trained action quantizer, and the trained action decoder.

[0080] In step S401, the feature sequence of the first action image sample is obtained. This feature sequence is a vector sequence representing action features extracted from the action posture.

[0081] In step S402, the feature sequence of the first motion image sample is input to the motion encoder, and the input feature sequence is encoded into latent variables by the motion encoder, thus obtaining the encoding vector of the feature sequence of the first motion image sample.

[0082] In step S403, the latent vectors output by the encoder are clustered and quantized using an action quantizer, outputting discrete quantities after quantization, which are the quantized encoding vectors of the feature sequences of the first action image sample. After training, the quantized encoding vectors of the feature sequences of the action image samples can be clustered to identify representative action features.

[0083] In step S404, the quantized encoding vector of the feature sequence of the first action image sample is reconstructed into a feature sequence by the action decoder.

[0084] In steps S405 and S406, the feature sequence of the first action image sample, the encoding vector of the feature sequence of the first action image sample, the quantization encoding vector of the first action image sample, and the reconstructed feature sequence of the first action image sample are substituted into Formula 2 above to calculate the second loss. Then, the parameters of the action encoder, the parameters of the action quantizer, and the parameters of the action decoder are adjusted according to the second loss until the calculated second loss is less than the preset value (i.e., the loss threshold of the action vector quantization variational autoencoder) and training stops, thus obtaining the trained action encoder, the trained action quantizer, and the trained action decoder.

[0085] In this example embodiment, the audio vector quantization variational autoencoder and the motion vector quantization variational autoencoder can be trained using the same network structure. During training, they are trained using pure audio training data (i.e., the first audio sample) and pure motion training data (i.e., the second motion image sample), respectively, which can make good use of the latent space and discretize the data well. It should be noted that, in addition to using convolutional neural networks, the vector quantization variational autoencoder can also use recurrent neural networks, Transformer networks, etc.

[0086] In this embodiment, an audio-action converter is trained using second audio samples and second motion image samples. The audio-action converter, used to convert audio sequences into motion sequences, is an attention-based sequence conversion model, which can be divided into an encoder and a decoder. The encoder consists of multiple layers, each comprising a multi-head self-attention mechanism, a feedforward neural network, and a normalization layer. The decoder can be a label-synchronized autoregressive decoder or a frame-synchronized Transducer decoder.

[0087] Figure 5 This is a flowchart illustrating the training of an audio motion converter according to an exemplary embodiment. Figure 5 As shown, the audio motion converter can be trained according to the following process:

[0088] Step S501: Obtain the feature sequence of the second audio sample and the feature sequence of the second motion image sample;

[0089] Step S502: Encode and quantize the feature sequence of the second audio sample to obtain the quantized encoding vector of the feature sequence of the second audio sample;

[0090] Step S503: Encode and quantize the feature sequence of the second action image sample to obtain the quantized encoding vector of the feature sequence of the second action image sample;

[0091] Step S504: Input the quantized encoding vector of the feature sequence of the second audio sample into the encoder in the audio motion converter to obtain the encoding vector;

[0092] Step S505: Input the encoded vector into the decoder in the audio motion converter to obtain the predicted motion sequence;

[0093] Step S506: Calculate the third loss based on the quantized encoding vector of the feature sequence of the predicted action sequence and the second action image sample;

[0094] Step S507: Adjust the parameters of the encoder and decoder in the audio motion converter according to the third loss until the calculated third loss is less than the preset value, and obtain the trained audio motion converter.

[0095] The audio-action converter can be a Transformer model, including a Transformer encoder and a Transformer decoder. Specifically, it can be trained using a second audio sample and a second action image sample corresponding to the second audio sample, that is, it can be trained using paired training samples.

[0096] In step S501, the feature sequence of the second audio sample is obtained, which is a vector sequence representing audio features extracted from the audio waveform; the feature sequence of the second motion image sample is obtained, which is a vector sequence representing motion features extracted from the motion posture.

[0097] In step S502, after obtaining the feature sequence of the second audio sample, the obtained feature sequence is encoded using the trained audio encoder to obtain an encoding vector. Then, the obtained encoding vector is input into the trained audio quantizer, and the input encoding vector is clustered and vector quantized to obtain the quantized encoding vector of the feature sequence of the second audio sample.

[0098] In step S503, after obtaining the feature sequence of the second action image sample, the obtained feature sequence is encoded using the trained action encoder to obtain an encoding vector. Then, the obtained encoding vector is input into the trained action quantizer, and the input encoding vector is clustered and vector quantized to obtain the quantized encoding vector of the feature sequence of the second action image sample.

[0099] In steps S504 and S505, the quantized encoding vector of the feature sequence of the acquired second audio sample is input into the Transformer encoder for encoding to obtain the encoding vector. That is, the quantized encoding vector of the feature sequence of the second audio sample is abstracted by the Transformer encoder. Then, the obtained encoding vector is input into the Transformer decoder for decoding to obtain the predicted action sequence. That is, the discrete symbols of each action corresponding to the second audio sample are generated by the Transformer decoder.

[0100] The predicted action sequence includes multiple action symbols. In this embodiment, step S505 may further include: for the current action symbol, inputting the encoded vector and the action prefix corresponding to the current action symbol into the decoder in the audio action converter to obtain the probability estimate of each candidate action symbol, wherein the action prefix corresponding to the current action symbol is the action symbol preceding the current action symbol in the predicted action sequence; and determining the current action symbol from the candidate action symbols based on the probability estimate of each candidate action symbol.

[0101] Assuming the Transformer decoder uses a transformer decoder, the input to the transformer decoder consists of two parts: an action prefix and the encoder output, which are combined using an attention mechanism. The action prefix refers to the string of symbols preceding the current action symbol. The Transformer decoder uses the action prefix y1,…,y1 corresponding to the current action symbol. s-1 The current action symbol y is estimated from the features h output by the encoder. s The probability of:

[0102] P(y s )=P(y s | y1,…, y s-1 Formula 3 (h)

[0103] In formula 3, y s Let P(y) represent the s-th action symbol. s ) represents the action symbol y sThe probability of the next action symbol is given by the Transformer encoder output, h. The Transformer decoder consists of D (D is any positive integer greater than 1) serially connected decoder modules. The Transformer encoder output h is input into each Transformer decoder module, and then the last Transformer decoder module outputs the probability estimate of the next action symbol. Finally, the action symbols with the highest estimated probabilities are concatenated to obtain the predicted action sequence corresponding to the second audio sample.

[0104] In steps S506 and S507, a third loss is calculated based on the quantized encoding vector of the feature sequence of the predicted action sequence and the second action image sample. For example, the difference between the quantized encoding vector of the feature sequence of the predicted action sequence and the second action image sample can be calculated. Then, the parameters of the Transformer encoder and the Transformer decoder are adjusted according to the third loss until the calculated third loss is less than a preset value (i.e., the loss threshold of the audio action converter) and training stops, thus obtaining the trained audio action converter.

[0105] Furthermore, in this embodiment of the disclosure, the maximum likelihood estimation criterion can be used during the training of the audio-action converter:

[0106]

[0107] After training, the audio action converter can estimate the probability P(y) of the next action symbol given a prefix and audio. s |y1,…,y s-1 When generating predicted action sequences, the beam search technique can be used to find the action symbol sequence with the highest probability, and then input into the action quantizer and action decoder to generate action feature sequences.

[0108] In this example embodiment, the audio-action converter is trained using audio-action pair data, and in addition to the Transformer model, the audio-action converter can also use models such as recurrent neural networks.

[0109] Figure 6 This is a schematic diagram of the structure of an action generation model according to an exemplary embodiment. Figure 6 In this model, the structure of the motion generation model may include: an audio encoder 601, an audio quantizer 602, an audio motion converter 603, a motion quantizer 604, and a motion decoder 605.

[0110] Among them, the audio encoder 601 and the audio quantizer 602 are the encoder and quantizer in the audio vector quantization variational autoencoder, and the audio vector quantization variational autoencoder is trained using pure audio data (i.e., the first audio sample). Figure 6 As shown, the audio vector quantization variational autoencoder may also include an audio decoder 606.

[0111] The motion quantizer 604 and motion decoder 605 are the quantizer and decoder in the motion vector quantization variational autoencoder, and the motion vector quantization variational autoencoder is trained using pure motion data (i.e., the first motion image sample). Figure 6 In this process, the motion vector quantization variational autoencoder may also include a motion encoder 607.

[0112] The audio-action converter 603 is obtained using a small number of paired audio-action data sequences (i.e., second audio samples and second action image samples corresponding to the second audio samples). The audio-action converter 603 includes an encoder and a decoder. The encoder in the audio-action converter abstractly represents the quantized encoding vector of the audio feature sequence obtained through the audio quantizer. The decoder in the audio-action converter decodes the abstractly represented quantized encoding vector, generating discrete action symbols one by one, thus obtaining the predicted action sequence. The generated discrete action symbols are input into the action quantizer for clustering and vector quantization processing, and then input into the action decoder to generate action features.

[0113] In this embodiment, the audio vector quantization variational autoencoder and the action vector quantization variational autoencoder are trained using unsupervised data, while the audio action converter is trained using a small amount of paired data. Thus, the entire network is trained in a semi-supervised manner, which can utilize a large amount of data to generate an action generation model and improve the generalization performance of the model.

[0114] In generation Figure 6 After the motion generation model is shown, audio data can be input into the motion generation model, and motion data matching the music data can be generated through the motion generation model. Figure 6 In the diagram, the dashed arrow indicates the process of generating an action that matches the audio using the action generation model. Figure 7 This is a flowchart illustrating a method for generating actions based on an action generation model, according to an exemplary embodiment. Figure 7 As shown, the method for generating actions based on an action generation model may include the following steps.

[0115] Step S701: Obtain the audio feature sequence to be processed.

[0116] In this step, the audio to be processed refers to the audio that needs to be processed, that is, the action that needs to be generated to match the audio. The audio feature sequence to be processed is a vector sequence representing the audio features extracted from the audio waveform to be processed.

[0117] Step S702: Input the audio feature sequence to be processed into the action generation model to obtain the action feature sequence corresponding to the audio feature sequence to be processed.

[0118] Specifically, the audio feature sequence to be processed is input into... Figure 6 In the audio encoder of the action generation model shown, the audio feature sequence to be processed is encoded to obtain an encoded vector. Then, the encoded vector is input to an audio quantizer for clustering and vector quantization, outputting a quantized encoded vector. Next, the quantized encoded vector is input to an audio action converter. The encoder in the audio action converter encodes the quantized encoded vector to obtain a new encoded vector. The decoder in the audio action converter decodes the new encoded vector to obtain a predicted action sequence. Then, the predicted action sequence is input to an action quantizer, which performs clustering quantization on the predicted action sequence to obtain a clustered quantized predicted action sequence. The action decoder decodes the clustered quantized predicted action sequence to obtain the action feature sequence.

[0119] In this example embodiment, an action generation model can be used to process the audio feature sequence to be processed, generating an action feature sequence corresponding to the audio feature sequence. This action generation model includes an audio encoder and audio quantizer in an audio vector quantization variational autoencoder, an action quantizer and action decoder in an action vector quantization variational autoencoder, and an audio action converter. It can capture typical features of the audio, reduce the representation space, and exhibits good generalization performance, resulting in a high degree of matching between the generated actions and the audio to be processed, and producing richer and smoother actions.

[0120] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0121] Figure 8 This is a block diagram of a training apparatus for an action generation model according to an exemplary embodiment. (Refer to...) Figure 8The training device 800 for the action generation model includes: a sample acquisition module 810, a first training module 820, a second training module 830, a third training module 840, and a model generation module 850.

[0122] The sample acquisition module 810 is used to: acquire a first audio sample, a first motion image sample, a second audio sample, and a second motion image sample corresponding to the second audio sample; the first training module 820 is used to: input the first audio sample sequentially into an audio encoder, an audio quantizer, and an audio decoder to obtain an audio reconstruction sequence, and train the audio encoder and audio quantizer based on the first audio sample and the audio reconstruction sequence to obtain a trained audio encoder and a trained audio quantizer; the second training module 830 is used to: input the first motion image sample sequentially into an motion encoder, a motion quantizer, and a motion decoder to obtain a motion reconstruction sequence, and train the motion quantizer and motion decoder based on the first motion image sample and the motion reconstruction sequence to obtain a trained motion quantizer and a trained motion decoder; the third training module 840 is used to: input the second audio sample into an audio motion converter to obtain a predicted motion sequence, and train the audio motion converter based on the second motion image sample and the predicted motion sequence to obtain a trained audio motion converter; the model generation module 850 is used to: sequentially connect the trained audio encoder, the trained audio quantizer, the trained audio motion converter, the trained motion quantizer, and the trained motion decoder to obtain a motion generation model.

[0123] In some embodiments of this disclosure, the first training module 820 is further configured to: acquire a feature sequence of a first audio sample; input the feature sequence of the first audio sample into an audio encoder to obtain an encoding vector of the feature sequence of the first audio sample; input the encoding vector of the feature sequence of the first audio sample into an audio quantizer to obtain a quantized encoding vector of the feature sequence of the first audio sample; input the quantized encoding vector of the feature sequence of the first audio sample into an audio decoder to obtain an audio reconstruction sequence of the first audio sample; calculate a first loss based on the feature sequence of the first audio sample, the encoding vector of the feature sequence of the first audio sample, the quantized encoding vector of the feature sequence of the first audio sample, and the audio reconstruction sequence; adjust the parameters of the audio encoder, the parameters of the audio quantizer, and the parameters of the audio decoder based on the first loss until the calculated first loss is less than a preset value, thereby obtaining a trained audio encoder, a trained audio quantizer, and a trained audio decoder.

[0124] In some embodiments of this disclosure, the second training module 830 is further configured to: acquire a feature sequence of a first motion image sample; input the feature sequence of the first motion image sample into a motion encoder to obtain an encoding vector of the feature sequence of the first motion image sample; input the encoding vector of the feature sequence of the first motion image sample into a motion quantizer to obtain a quantized encoding vector of the feature sequence of the first motion image sample; input the quantized encoding vector of the feature sequence of the first motion image sample into a motion decoder to obtain a motion reconstruction sequence of the first motion image sample; calculate a second loss based on the feature sequence of the first motion image sample, the encoding vector of the feature sequence of the first motion image sample, the quantized encoding vector of the first motion image sample, and the motion reconstruction sequence; adjust the parameters of the motion encoder, the parameters of the motion quantizer, and the parameters of the motion decoder based on the second loss until the calculated second loss is less than a preset value, thereby obtaining a trained motion encoder, a trained motion quantizer, and a trained motion decoder.

[0125] In some embodiments of this disclosure, the third training module 840 is further configured to: acquire the feature sequence of the second audio sample and the feature sequence of the second motion image sample; encode and quantize the feature sequence of the second audio sample to obtain a quantized encoding vector of the feature sequence of the second audio sample; encode and quantize the feature sequence of the second motion image sample to obtain a quantized encoding vector of the feature sequence of the second motion image sample; input the quantized encoding vector of the feature sequence of the second audio sample to the encoder in the audio motion converter to obtain an encoding vector; input the encoding vector to the decoder in the audio motion converter to obtain a predicted motion sequence; calculate a third loss based on the predicted motion sequence and the quantized encoding vector of the feature sequence of the second motion image sample; adjust the parameters of the encoder and the decoder in the audio motion converter according to the third loss until the calculated third loss is less than a preset value, thereby obtaining a trained audio motion converter.

[0126] In some embodiments of this disclosure, the predicted action sequence includes multiple action symbols. The third training module 840 is further configured to: for the current action symbol, input the encoded vector and the action prefix corresponding to the current action symbol into the decoder of the audio action converter to obtain probability estimates for each candidate action symbol, wherein the action prefix corresponding to the current action symbol is the action symbol preceding the current action symbol in the predicted action sequence; and determine the current action symbol from the candidate action symbols based on the probability estimates of each candidate action symbol.

[0127] Figure 9 This is a block diagram of an action generation device based on an action generation model, according to an exemplary embodiment. (Refer to...) Figure 9The motion generation device 900 includes an audio sequence acquisition module 910 and an motion sequence generation module 920.

[0128] The audio sequence acquisition module 910 is used to acquire the audio feature sequence to be processed; the action sequence generation module 920 is used to input the audio feature sequence to be processed into the action generation model to obtain the action feature sequence corresponding to the audio feature sequence to be processed.

[0129] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0130] Figure 10 This is a schematic diagram illustrating the structure of an electronic device suitable for implementing embodiments of the present disclosure, according to an exemplary embodiment. It should be noted that... Figure 10 The illustrated electronic device 1000 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0131] like Figure 10 As shown, the electronic device 1000 is manifested in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, and a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010).

[0132] The storage unit stores program code that can be executed by the processing unit 1010, causing the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1010 can perform, as follows: Figure 2The steps shown are as follows: Step S201: Obtain a first audio sample, a first motion image sample, a second audio sample, and a second motion image sample corresponding to the second audio sample; Step S202: Input the first audio sample sequentially into an audio encoder, an audio quantizer, and an audio decoder to obtain an audio reconstruction sequence; train the audio encoder and audio quantizer based on the first audio sample and the audio reconstruction sequence to obtain a trained audio encoder and a trained audio quantizer; Step S203: Input the first motion image sample sequentially into an motion encoder, motion quantizer, and motion decoder to obtain a motion reconstruction sequence; train the motion quantizer and motion decoder based on the first motion image sample and the motion reconstruction sequence to obtain a trained motion quantizer and a trained motion decoder; Step S204: Input the second audio sample into an audio motion converter to obtain a predicted motion sequence; train the audio motion converter based on the second motion image sample and the predicted motion sequence to obtain a trained audio motion converter; Step S205: Connect the trained audio encoder, trained audio quantizer, trained audio motion converter, trained motion quantizer, and trained motion decoder sequentially to obtain a motion generation model.

[0133] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203.

[0134] Storage unit 1020 may also include a program / utility 10204 having a set (at least one) program module 10205, such program module 10205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0135] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0136] Electronic device 1000 can also communicate with one or more external devices 1060 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1040. As shown, network adapter 1040 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0137] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0138] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0140] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0141] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0142] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0143] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0144] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0145] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0146] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0147] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for an action generation model, characterized in that, include: Obtain a first audio sample, a first motion image sample, a second audio sample, and a second motion image sample corresponding to the second audio sample; The process involves: acquiring the feature sequence of the first audio sample; inputting the feature sequence of the first audio sample into the audio encoder to obtain the encoding vector of the feature sequence of the first audio sample; inputting the encoding vector of the feature sequence of the first audio sample into the audio quantizer to obtain the quantized encoding vector of the feature sequence of the first audio sample; inputting the quantized encoding vector of the feature sequence of the first audio sample into the audio decoder to obtain the audio reconstruction sequence of the first audio sample; calculating a first loss based on the feature sequence of the first audio sample, the encoding vector of the feature sequence of the first audio sample, the quantized encoding vector of the feature sequence of the first audio sample, and the audio reconstruction sequence; adjusting the parameters of the audio encoder, the audio quantizer, and the audio decoder based on the first loss until the calculated first loss is less than a preset value, thereby obtaining the trained audio encoder, the trained audio quantizer, and the trained audio decoder. The first motion image sample is sequentially input into the motion encoder, motion quantizer and motion decoder to obtain the motion reconstruction sequence. The motion quantizer and motion decoder are trained based on the first motion image sample and the motion reconstruction sequence to obtain the trained motion quantizer and trained motion decoder. The second audio sample is input into the audio action converter to obtain the predicted action sequence. The audio action converter is trained based on the second action image sample and the predicted action sequence to obtain the trained audio action converter. The trained audio encoder, the trained audio quantizer, the trained audio motion converter, the trained motion quantizer, and the trained motion decoder are connected sequentially to obtain the motion generation model.

2. The method according to claim 1, characterized in that, The step of sequentially inputting the first motion image sample into a motion encoder, a motion quantizer, and a motion decoder to obtain a motion reconstruction sequence, and training the motion quantizer and the motion decoder based on the first motion image sample and the motion reconstruction sequence to obtain a trained motion quantizer and a trained motion decoder includes: Obtain the feature sequence of the first action image sample; The feature sequence of the first motion image sample is input into the motion encoder to obtain the encoding vector of the feature sequence of the first motion image sample; The encoding vector of the feature sequence of the first action image sample is input into the action quantizer to obtain the quantized encoding vector of the feature sequence of the first action image sample. The quantized encoding vector of the feature sequence of the first action image sample is input into the action decoder to obtain the action reconstruction sequence of the first action image sample; The second loss is calculated based on the feature sequence of the first action image sample, the encoding vector of the feature sequence of the first action image sample, the quantization encoding vector of the first action image sample, and the action reconstruction sequence. The parameters of the motion encoder, the motion quantizer, and the motion decoder are adjusted according to the second loss until the calculated second loss is less than a preset value, thus obtaining the trained motion encoder, the trained motion quantizer, and the trained motion decoder.

3. The method according to claim 1, characterized in that, The step of inputting the second audio sample into the audio action converter to obtain a predicted action sequence, and training the audio action converter based on the second action image sample and the predicted action sequence to obtain the trained audio action converter includes: Obtain the feature sequence of the second audio sample and the feature sequence of the second motion image sample; The feature sequence of the second audio sample is encoded and quantized to obtain the quantized encoding vector of the feature sequence of the second audio sample; The feature sequence of the second action image sample is encoded and quantized to obtain the quantized encoding vector of the feature sequence of the second action image sample; The quantized encoding vector of the feature sequence of the second audio sample is input into the encoder in the audio motion converter to obtain the encoding vector; The encoded vector is input into the decoder in the audio-action converter to obtain the predicted action sequence; The third loss is calculated based on the quantized encoding vector of the feature sequence of the predicted action sequence and the second action image sample; The parameters of the encoder and decoder in the audio motion converter are adjusted according to the third loss until the calculated third loss is less than a preset value, thus obtaining the trained audio motion converter.

4. The method according to claim 3, characterized in that, The predicted action sequence includes multiple action symbols; The step of inputting the encoded vector into the decoder of the audio motion converter to obtain the predicted motion sequence includes: For the current action symbol, the encoded vector and the action prefix corresponding to the current action symbol are input into the decoder in the audio action converter to obtain the probability estimate of each candidate action symbol, wherein the action prefix corresponding to the current action symbol is the action symbol preceding the current action symbol in the predicted action sequence; The current action symbol is determined from the candidate action symbols based on the probability estimates of each candidate action symbol.

5. An action generation method, characterized in that, The method includes: Obtain the audio feature sequence to be processed; The audio feature sequence to be processed is input into the action generation model generated by the method according to any one of claims 1 to 4 to obtain the action feature sequence corresponding to the audio feature sequence to be processed.

6. A training device for an action generation model, characterized in that, The device includes: The sample acquisition module is used to acquire a first audio sample, a first motion image sample, a second audio sample, and a second motion image sample corresponding to the second audio sample; A first training module is configured to: acquire the feature sequence of the first audio sample; input the feature sequence of the first audio sample into the audio encoder to obtain the encoding vector of the feature sequence of the first audio sample; input the encoding vector of the feature sequence of the first audio sample into the audio quantizer to obtain the quantized encoding vector of the feature sequence of the first audio sample; input the quantized encoding vector of the feature sequence of the first audio sample into the audio decoder to obtain the audio reconstruction sequence of the first audio sample; calculate a first loss based on the feature sequence of the first audio sample, the encoding vector of the feature sequence of the first audio sample, the quantized encoding vector of the feature sequence of the first audio sample, and the audio reconstruction sequence; adjust the parameters of the audio encoder, the audio quantizer, and the audio decoder based on the first loss until the calculated first loss is less than a preset value, thereby obtaining the trained audio encoder, the trained audio quantizer, and the trained audio decoder. The second training module is used to input the first motion image sample into the motion encoder, motion quantizer and motion decoder in sequence to obtain the motion reconstruction sequence, and to train the motion quantizer and motion decoder according to the first motion image sample and the motion reconstruction sequence to obtain the trained motion quantizer and trained motion decoder. The third training module is used to input the second audio sample into the audio action converter to obtain the predicted action sequence, and to train the audio action converter based on the second action image sample and the predicted action sequence to obtain the trained audio action converter. The model generation module is used to sequentially connect the trained audio encoder, the trained audio quantizer, the trained audio motion converter, the trained motion quantizer, and the trained motion decoder to obtain an motion generation model.

7. An action generation device, characterized in that, The device includes: The audio sequence acquisition module is used to acquire the audio feature sequence to be processed; An action sequence generation module is used to input the audio feature sequence to be processed into an action generation model generated by the method according to any one of claims 1 to 4, so as to obtain an action feature sequence corresponding to the audio feature sequence to be processed.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the action generation model as described in any one of claims 1 to 4, or to implement the action generation method as described in claim 5.

9. A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a training method for an action generation model as described in any one of claims 1 to 4, or to perform an action generation method as described in claim 5.

Images