Audio-based attitude generation model training method and attitude generation method

Abstract

The invention provides a training method of a posture generation model based on audio and a posture generation method, and relates to the technical field of computers, in particular to the technical field of artificial intelligence such as computer vision, deep learning and large models. The method comprises the following steps: firstly, processing a sample video, determining an audio token sequence corresponding to the sample video and a noise-containing attitude hidden variable corresponding to the sample video, and then determining an attitude reference hidden variable corresponding to the sample video and a mask vector corresponding to the attitude reference hidden variable; the method comprises the following steps: firstly, generating an initial generation model, inputting a noise-containing attitude hidden variable, an attitude reference hidden variable, a mask vector and an audio token sequence into the initial generation model to obtain an initially generated and output predicted noise, then correcting the initial generation model based on the difference between the predicted noise and an actual noise to obtain a corrected generation model, and finally, generating an audio signal based on the corrected generation model. And returning to execute the noise prediction step until the target generation model is obtained.

Description

Training methods and pose generation methods for audio-based pose generation models Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence technology such as computer vision, deep learning, and large models. Specifically, it relates to a training method and a pose generation method for an audio-based pose generation model. Background Technology

[0002] Currently, music-driven dance video generation is typically broken down into two steps: generating dance posture sequences based on music; and generating realistic dance videos based on dance posture sequences.

[0003] The image or video generation technology in the second step of the existing technology is relatively mature, and can generate high-fidelity, identity-preserving videos given a pose sequence and a reference image. However, the first step—how ​​to generate a pose sequence that is temporally coherent, rhythmically follows the music, and has the same proportions as the reference image for characters in complex and varied music and real-world scene video distributions—is a problem that urgently needs to be solved. Summary of the Invention

[0004] This disclosure aims to at least partially address one of the technical problems in the related art.

[0005] The first aspect of this disclosure proposes a training method for an audio-based pose generation model, comprising: processing a sample video to determine an audio token sequence corresponding to the sample video and a noisy pose latent variable corresponding to the sample video; determining a pose reference latent variable corresponding to the sample video and a mask vector corresponding to the pose reference latent variable, wherein the mask vector is used to mark the position of the real reference frame in the pose reference latent variable; inputting the noisy pose latent variable, the pose reference latent variable, the mask vector, and the audio token sequence into an initial generation model to obtain the predicted noise of the initial generation output; correcting the initial generation model based on the difference between the predicted noise and the actual noise to obtain a corrected generation model; and returning to the step of performing noise prediction based on the corrected generation model until a target generation model is obtained.

[0006] A second aspect of this disclosure proposes an audio-based pose generation method, comprising: receiving a pose sequence generation request, wherein the generation request includes a reference image and an audio sequence; performing pose estimation and encoding processing on the reference image to determine a first pose reference latent variable and a first mask vector; performing audio encoding on the audio sequence to obtain an audio token sequence corresponding to the audio sequence; inputting a preset pure noise pose latent variable, the first pose reference latent variable, the first mask vector, and the audio token sequence into a generation model to obtain a pose latent variable output by the generation model, wherein the generation model is trained based on the method described in the first aspect of this disclosure; and decoding the pose latent variable to obtain a pose sequence corresponding to the audio sequence.

[0007] A third aspect of this disclosure provides a training apparatus for an audio-based pose generation model, comprising: a first processing module for processing a sample video to determine an audio token sequence corresponding to the sample video and a noisy pose latent variable corresponding to the sample video; a determination module for determining a pose reference latent variable corresponding to the sample video and a mask vector corresponding to the pose reference latent variable, wherein the mask vector is used to mark the position of the real reference frame in the pose reference latent variable; a first input module for inputting the noisy pose latent variable, the pose reference latent variable, the mask vector, and the audio token sequence into an initial generation model to obtain the predicted noise of the initial generation output; a correction module for correcting the initial generation model based on the difference between the predicted noise and the actual noise to obtain a corrected generation model; and a second processing module for returning to the noise prediction step based on the corrected generation model until a target generation model is obtained.

[0008] A fourth aspect of this disclosure provides an audio-based pose generation apparatus, comprising: a receiving module for receiving a pose sequence generation request, wherein the generation request includes a reference image and an audio sequence; a third processing module for performing pose estimation and encoding processing on the reference image to determine a first pose reference latent variable and a first mask vector; an encoding module for performing audio encoding on the audio sequence to obtain an audio token sequence corresponding to the audio sequence; a second input module for inputting a preset pure noise pose latent variable, the first pose reference latent variable, the first mask vector, and the audio token sequence into a generation model to obtain a pose latent variable output by the generation model, wherein the generation model is trained based on the method described in the first aspect of this disclosure; and a decoding module for decoding the pose latent variable to obtain a pose sequence corresponding to the audio sequence.

[0009] A fifth aspect of this disclosure provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the training method for an audio-based pose generation model as proposed in the first aspect of this disclosure and the audio-based pose generation method as proposed in the second aspect of this disclosure.

[0010] A sixth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method for an audio-based pose generation model as proposed in the first aspect of this disclosure and the audio-based pose generation method as proposed in the second aspect of this disclosure.

[0011] A seventh aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the training method for an audio-based pose generation model as proposed in the first aspect of this disclosure and the audio-based pose generation method as proposed in the second aspect of this disclosure.

[0012] The training method and pose generation method for the audio-based pose generation model provided in this disclosure have the following beneficial effects: In the embodiments of this disclosure, the sample video is first processed to determine the audio token sequence corresponding to the sample video and the pose latent variable containing noise corresponding to the sample video. Then, the pose reference latent variable corresponding to the sample video and the mask vector corresponding to the pose reference latent variable are determined. The pose latent variable containing noise, the pose reference latent variable, the mask vector and the audio token sequence are input into the initial generation model to obtain the predicted noise of the initial generation output. Then, based on the difference between the predicted noise and the actual noise, the initial generation model is corrected to obtain the corrected generation model. Finally, based on the corrected generation model, the step of performing noise prediction is returned until the target generation model is obtained. Thus, the model was trained based on multimodal feature fusion and constraints of reference latent variables and masks. This enabled the model to learn the ability to control the pose proportion and maintain the consistency between the pose and the audio during training. This effectively solved the problems of audio and pose being out of sync and pose proportion being inconsistent with the reference pose in the dance videos generated by the model. Furthermore, based on the difference between the predicted noise and the actual noise output by the model, the model was iteratively corrected to obtain the target generation model. This improved the stability and quality of the model training process, thereby enhancing the performance of the target generation model.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 is a flowchart illustrating a training method for an audio-based posture generation model provided in an embodiment of this disclosure; Figure 2 is a flowchart illustrating the determination of noisy posture latent variables in a training method for an audio-based posture generation model provided in an embodiment of this disclosure; Figure 3 is a flowchart illustrating a training method for an audio-based posture generation model provided in an embodiment of this disclosure; Figure 4 is a flowchart illustrating a training method for an audio-based posture generation model provided in an embodiment of this disclosure; Figure 5 is a flowchart illustrating a training method for an audio-based posture generation model provided in an embodiment of this disclosure; Figure 6 is a flowchart illustrating a training method for an audio-based posture generation model provided in an embodiment of this disclosure; Figure 7 is a schematic diagram of the system architecture for a music-based posture generation model proposed in an embodiment of this disclosure; Figure 8 is a schematic diagram of the system framework principle of the training method for a music-based posture generation model proposed in this disclosure; Figure 9 is a schematic diagram of the system application process of the music-based posture generation method proposed in this disclosure; Figure 10 is a schematic diagram of the structure of a training device for an audio-based posture generation model provided in an embodiment of this disclosure; Figure 11 is a schematic diagram of the structure of an audio-based posture generation device provided in an embodiment of this disclosure; Figure 12 shows a block diagram of an exemplary electronic device suitable for implementing embodiments of this disclosure. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, such as computer vision, deep learning, and large models.

[0017] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence.

[0018] Computer vision refers to machine vision that uses cameras and computers to identify, track, and measure targets instead of human eyes, and further processes the images to make them more suitable for human observation or transmission to instruments for detection.

[0019] Deep learning (DL) learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize data such as text, images, and sound.

[0020] Large models, also known as Foundation Models, are models that extract knowledge from hundreds of millions of corpora or images, learn, and then produce large models with hundreds of millions of parameters.

[0021] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0022] The training method and pose generation method of the audio-based pose generation model of this disclosure are described below with reference to the accompanying drawings.

[0023] Figure 1 is a flowchart illustrating a training method for an audio-based pose generation model provided in an embodiment of this disclosure.

[0024] As shown in Figure 1, the training method of the audio-based pose generation model may include the following steps: Step 101, process the sample video to determine the audio token sequence corresponding to the sample video and the pose latent variables containing noise corresponding to the sample video.

[0025] It should be noted that the pose generation model trained by the audio-based pose generation model training method proposed in this disclosure can be used in automatic dance content generation scenarios in video platforms or virtual human live streaming scenarios, automatic dance video generation scenarios based on brand image in marketing or advertising creative, unified music-driven dance capability scenarios across multiple product lines such as keyboard skins, expressions, and virtual human images, etc. Specific application scenarios can be set according to actual needs, and this disclosure does not limit them.

[0026] It should be noted that the method of acquiring sample videos can be set according to actual needs. For example, dance videos and corresponding audio can be collected from the Internet or proprietary data sources. By utilizing a large amount of real-world dance data, it is helpful to train the model's generalization ability across different music styles, rhythms, and character proportions. This can effectively reduce the engineering costs of repeatedly tuning parameters or retraining for a single style. This disclosure does not limit this aspect.

[0027] The audio token sequence may include tokens related to the rhythm, melody, and style of the audio, and the specific details can be determined according to actual needs. This disclosure does not impose any restrictions on this.

[0028] It should be noted that the specific implementation of determining the audio token sequence corresponding to the sample video can be set according to actual needs. For example, the audio token sequence can be determined by processing the audio in the sample video using a pre-trained audio encoder, etc. This disclosure does not limit this.

[0029] The specific type of the pre-trained audio encoder can be set according to actual needs. For example, the pre-trained audio encoder can be a generative music model, etc., and this disclosure does not limit it.

[0030] Among them, latent posture variables are low-dimensional vector representations that cannot be directly observed but can be inferred from posture data, and can include essential features of human posture.

[0031] Among them, the pose latent variables containing noise can be used to train the model, and can also be called "noisy pose latent", etc., which are not limited in this disclosure.

[0032] In some possible implementations, the pose latent variables of the sample video can be constructed by using standard linear or cosine noise scheduling during the forward noise addition stage of the model training process, without limiting the scope of this disclosure.

[0033] Step 102: Determine the pose reference latent variables and the mask vector corresponding to the pose reference latent variables in the sample video.

[0034] The mask vector is used to mark the location of the real reference frame in the attitude reference latent variables.

[0035] Among them, the real reference frame can be the frame in the sample video where the actual labeled pose is located.

[0036] In this disclosure, the specific implementation form of the pose reference latent variable corresponding to the sample video can be set according to actual needs. For example, pose estimation can be performed on the sample video, selecting a single-frame pose, a multi-frame average pose, or a representative pose in the sample video (the specific pose selected can be determined according to actual needs, and is not limited here), converting it to one-hot form and performing VAE encoding to obtain the pose reference latent variable. This disclosure does not limit this.

[0037] One-hot encoding is a coding method used for digitizing categorical features. Its core principle is that each category corresponds to a unique binary bit activation.

[0038] VAE stands for Variational Autoencoder.

[0039] By determining the pose reference latent variables, the model can learn the proportions of the human figure and the control of camera distance during training, thereby ensuring that the proportions of the human figure and the camera distance remain consistent in the pose sequences generated by the model, thus improving the viewing experience of downstream pose-driven videos.

[0040] In this disclosure, when determining the mask vector corresponding to the attitude reference latent variable, a binary mask with the same size as the attitude reference latent variable can be determined to mark the location of the true reference frame in the latent variable, thereby prompting the model to fit the reference attitude as closely as possible at these locations.

[0041] It should be noted that the encoding method of the mask vector is not restricted and can be set according to actual needs. For example, different weights can be assigned to different positions instead of just binary labels of 0 or 1, etc., and this disclosure does not limit this.

[0042] Step 103: Input the pose latent variables containing noise, the pose reference latent variables, the mask vector, and the audio token sequence into the initial generation model to obtain the prediction noise of the initial generation output.

[0043] The specific type and architecture of the initial generation model can be set according to actual needs. For example, the initial generation model can be a Transformer-based diffusion network such as a Diffusion Transformer (Dit) network, etc., and this disclosure does not limit it.

[0044] The prediction noise refers to the noise in the pose latent variables predicted by the initial generative model, which includes noise.

[0045] In this disclosure, by inputting pose latent variables containing noise, pose reference latent variables, mask vectors, and audio token sequences into the initial generation model, the predicted noise output by the initial generation model is obtained. This enables the initial generation model to be trained through multimodal feature fusion and constraints on reference latent variables and mask vectors, thereby improving the model training efficiency and effectiveness.

[0046] Step 104: Based on the difference between the predicted noise and the actual noise, the initial generation model is corrected to obtain the corrected generation model.

[0047] Among them, actual noise can be the noise actually added to the latent variables of posture during the forward noise addition phase of the training process.

[0048] It should be noted that the specific implementation method for determining the difference between predicted noise and actual noise can be set according to actual needs. For example, the difference between predicted noise and actual noise can be determined using a standard mean squared error loss function, and this disclosure does not limit this.

[0049] In this disclosure, after obtaining the predicted noise output by the initial generation model, the difference between the predicted noise and the actual noise can be determined first, and then the initial generation model can be corrected based on the determined difference to obtain the corrected generation model.

[0050] Step 105: Based on the corrected generative model, return to the step of performing noise prediction until the target generative model is obtained.

[0051] The target generation model can be a pre-trained audio-based pose generation model.

[0052] In this disclosure, by returning to the noise prediction step based on the modified generative model, the predicted noise gradually approaches the actual noise until the target generative model is obtained, thereby improving the model training effect and stability and ensuring the generation quality of the target generative model generated during training.

[0053] In this embodiment, the sample video is first processed to determine the audio token sequence corresponding to the sample video and the pose latent variable containing noise corresponding to the sample video. Then, the pose reference latent variable corresponding to the sample video and the mask vector corresponding to the pose reference latent variable are determined. The pose latent variable containing noise, the pose reference latent variable, the mask vector and the audio token sequence are input into the initial generation model to obtain the predicted noise of the initial generation output. Then, based on the difference between the predicted noise and the actual noise, the initial generation model is corrected to obtain the corrected generation model. Finally, based on the corrected generation model, the step of performing noise prediction is returned until the target generation model is obtained. Therefore, by iteratively training the initial generative model using audio token sequences based on sample videos, noisy pose latent variables, pose reference latent variables, and their corresponding mask vectors, the model is trained based on multimodal feature fusion and constraints of reference latent variables and masks. This enables the model to learn pose proportion control and maintain the consistency between pose and audio during training, effectively solving the problems of audio and pose asynchrony and pose proportion inconsistency with the reference pose in the generated dance videos. Furthermore, based on the difference between the predicted noise and the actual noise in the model output, the model is continuously corrected to obtain the target generative model, improving the stability and quality of the model training process, and thus enhancing the performance of the target generative model.

[0054] Figure 2 is a flowchart illustrating the process of determining noisy pose latent variables in a training method for an audio-based pose generation model provided in an embodiment of this disclosure.

[0055] As shown in Figure 2, the process of determining the pose latent variables containing noise in the training method of the audio-based pose generation model may include the following steps: Step 201, perform pose estimation on the sample video and determine the coordinates of each key point in each frame of the sample video.

[0056] It should be noted that the specific implementation of pose estimation for the sample video can be set according to actual needs. For example, a preset pose estimation model can be used to estimate the pose of the sample video, and this disclosure does not limit this.

[0057] The specific type and structure of the attitude estimation model can be set according to actual needs. For example, the attitude estimation model can be a two-dimensional attitude estimation model or a three-dimensional attitude estimation model; this disclosure does not limit it in this regard.

[0058] Among them, each key point in each frame of the image can be a key point of human posture in each frame of the image, which can include key points of human posture such as head, neck, shoulders, chest, waist, upper limbs and lower limbs, etc. The specific settings can be set according to actual needs, and this disclosure does not limit them.

[0059] In this disclosure, by performing pose estimation on sample videos, the coordinates of each key point in each frame of the sample video are determined, thereby obtaining the pose sequence corresponding to the sample video, providing a data basis for subsequent generation of pose latent variables.

[0060] In this disclosure, the pose of each frame of an image can be obtained by performing pose estimation on the sample video, and then the coordinates of each key point in each frame of the image can be determined based on the pose.

[0061] Step 202: Encode the coordinates of each key point, and determine the first orientation vector and first confidence level, the second orientation vector and second confidence level corresponding to each key point.

[0062] Among them, the first direction attitude vector is the attitude vector in the horizontal direction, that is, the attitude vector in the x-axis direction.

[0063] Among them, the second direction attitude vector is the vertical attitude vector, that is, the attitude vector in the y-axis direction.

[0064] Among them, confidence level can be used to judge the credibility of attitude vector.

[0065] It should be noted that the specific implementation of encoding the coordinates of each key point and determining the first orientation vector and the second orientation vector corresponding to each key point can be determined according to actual needs, and this disclosure does not limit it.

[0066] For example, in some possible implementations, one-hot encoding can be used to encode the coordinates of each keypoint to determine the corresponding pose vector: The coordinates of all keypoints are normalized to eliminate the influence of image size. The normalized x and y values ​​in the [0,1] interval are divided into multiple equal-width intervals, each corresponding to a discrete category. The discrete category corresponding to the x-coordinate and the discrete category corresponding to the y-coordinate of each keypoint are determined. Based on the discrete category of the x-coordinate of each keypoint, a one-hot vector corresponding to the x-coordinate is generated, and based on the discrete category of the y-coordinate, a one-hot vector corresponding to the y-coordinate is generated. In the one-hot vector, only the position corresponding to the category is 1, and the rest are 0. Finally, the one-hot vectors corresponding to the x-coordinates of all keypoints are concatenated in a fixed order (such as the keypoint numbering order) to obtain the first-direction pose vector, and the one-hot vectors corresponding to the y-coordinates are concatenated in a fixed order (such as the keypoint numbering order) to obtain the second-direction pose vector. This disclosure does not limit the scope of the implementation.

[0067] In some possible implementations, the pose vector representation can be not only one-hot but also other encoded representations, depending on the specific needs. For example, the one-hot representation can be replaced with soft one-hot encoding, Gaussian label smoothing, or low-resolution heatmap representation, etc., as long as the pose sequence is still converted into a "multi-channel image" format and encoded via VAE. This disclosure does not limit this.

[0068] In some possible implementations, discrete intervals with different resolutions may be used for the first and second directions, or the coordinates may be nonlinearly quantized (e.g., using a finer scale for the vertical direction). This disclosure does not limit this.

[0069] In this disclosure, when using a pose estimation model to estimate the pose of a sample video and determine the coordinates of each key point in each frame of the sample video, the confidence scores corresponding to the first direction coordinates and the second direction coordinates of each key point output by the model can also be obtained. Then, when encoding the coordinates of each key point and determining the first direction pose vector corresponding to each key point, the confidence scores of the first direction coordinates of all key points can be aggregated to obtain the first confidence score. Similarly, the confidence scores of the second direction coordinates of all key points can be aggregated to obtain the second confidence score. This disclosure does not limit this aspect.

[0070] It should be noted that the specific implementation of confidence score aggregation can be set according to actual needs. For example, confidence scores can be aggregated using methods such as arithmetic average or weighted average, and this disclosure does not limit this.

[0071] For example, let the two-dimensional pose of a single frame be... Where K is the number of keypoints and 3 is the feature dimension of the keypoints. When using one-hot split-axis encoding, the horizontal direction is first divided into W equal-width intervals, each interval corresponding to a discrete category (W can be set as needed, for example, W can be 512, no restriction is made here). For the x-coordinate of each keypoint, a sparse vector of length W is constructed, which is the first-direction pose vector: The remaining positions are 0. Similarly, construct the y-coordinate. ,have: .

[0072] Among them, the kth key point is For normalized coordinates, For confidence level, It is the encoding vector of the x-coordinate of the k-th key point. The element value at each position, similarly... For the first The element value at each position.

[0073] In some possible implementations, when encoding the coordinates of each keypoint, especially when the image includes hand keypoints, issues such as finger sticking and tremors can easily occur after downsampling or encoding due to the large number of closely spaced keypoints in the hand. Therefore, a better approach is to first enlarge the hand keypoints away from the wrist, obtaining updated coordinates, and then encode these updated coordinates. This hand enlargement strategy allows the hand keypoints to occupy a larger pixel span, improving hand detail resolution without modifying the network structure, ensuring hand stability during training, and enhancing hand movement quality.

[0074] Step 203: Stack all pose vectors corresponding to all key points in each frame of the image in the channel dimension and the time dimension respectively to obtain the pose tensor corresponding to the sample video.

[0075] The channel dimension can be an image channel, which can be set according to actual needs. For example, the channel dimension can be 3-dimensional, and this disclosure does not limit it.

[0076] In this disclosure, after obtaining the first orientation vector and the second orientation vector corresponding to each key point in each frame of the image, the orientation vectors corresponding to all key points in each frame of the image can be stacked in the channel dimension to obtain the representation of each frame of the image. Then, the representations of each frame of the image are stacked in the time dimension to obtain the orientation tensor of the multi-channel image corresponding to the sample video.

[0077] For example, after obtaining the first-direction pose vector and confidence level, and the second-direction pose vector and confidence level for each keypoint in each frame, the first-direction pose vectors and second-direction pose vectors of all keypoints in each frame can be stacked according to the channel dimension to obtain a single-frame representation: Then, all the obtained single-frame representations are stacked along the time dimension T to obtain the pose tensor corresponding to the sample video: .

[0078] Step 204: Encode and add noise to the pose tensor to obtain the pose latent variables containing noise corresponding to the sample video.

[0079] In this disclosure, the pose tensor is encoded to convert it into a low-dimensional pose latent variable, resulting in a smaller dimension and a more regular structure. Then, the pose latent variable is subjected to noise processing to obtain the pose latent variable containing noise for the points corresponding to the sample video.

[0080] It should be noted that the specific implementation of encoding the attitude tensor can be set according to actual needs. For example, the attitude tensor can be encoded using VAE encoding, and this disclosure does not limit this.

[0081] For example, when encoding the pose tensor, a pre-trained image VAE encoder can be used for latent variable encoding: first, channel grouping and 3-channel pseudo-image construction are performed, that is, the C channels are divided into G groups of 3 channels each (if the last group has less than 3 channels, the last channel is copied to make up the difference), forming G images of size . "Pseudo-images": .

[0082] Then VAE encoding and downsampling are performed, i.e., using a pre-trained image VAE encoder. Each pseudo-image is encoded to obtain a 4-channel pose latent variable, with the spatial size downsampled by a factor of 8: .

[0083] Then, implicit variable concatenation is performed, that is, all... By concatenating the data along the channel dimension, we obtain the pose latent variables corresponding to the sample video: .

[0084] In this way, this disclosure eliminates the need to retrain the VAE for each number of keypoints, and allows for direct reuse of a unified image VAE capability.

[0085] In some possible implementations, the step of using a VAE encoder to generate latent variables in this disclosure can also replace the VAE with other image encoders, such as Vector Quantized Variational Autoencoder (VQ-VAE), etc. This disclosure does not limit this.

[0086] It should be noted that the specific implementation of adding noise to the latent attitude variables can be set according to actual needs. This disclosure does not limit it in this regard.

[0087] For example, standard linear or cosine noise scheduling can be applied to attitude latent variables to construct noisy data, i.e., attitude latent variables containing noise: ,in, For noise step The corresponding signal coefficients, For noise step The corresponding noise figure, Standard Gaussian noise, It follows a multivariate standard normal distribution. For noise step index, It is a discrete uniform distribution. This represents the total number of noise steps.

[0088] In this disclosure, by using one-hot representation and VAE encoding, the model can significantly reduce the output dimension and training cost while ensuring quality. At the same time, it can directly reuse existing generative models (such as DiT) + VAE image generation framework, reducing the R&D cost and maintenance burden of custom networks.

[0089] In this embodiment, pose estimation is first performed on the sample video to determine the coordinates of keypoints in each frame. Then, the coordinates of each keypoint are encoded to determine the first orientation pose vector and first confidence level, as well as the second orientation pose vector and second confidence level. Next, all pose vectors corresponding to all keypoints in each frame are stacked in both the channel and time dimensions to obtain the pose tensor corresponding to the sample video. Finally, the pose tensor is encoded and noise-added to obtain the noisy pose latent variables corresponding to the sample video. Thus, by performing pose estimation on the sample video and encoding the results, the horizontal and vertical pose vectors and confidence levels of all keypoints in each frame are determined. By stacking all pose vectors in both the channel and time dimensions to obtain the pose tensor, discrete pose information can be transformed into a structured vector representation. Encoding and noise-adding to the pose tensor to obtain noisy pose latent variables reduces the dimensionality of the pose tensor and decreases redundant data. Furthermore, noise-adding effectively prevents overfitting during model training, providing accurate and reliable data for model training.

[0090] Figure 3 is a flowchart illustrating a training method for an audio-based pose generation model provided in an embodiment of this disclosure.

[0091] As shown in Figure 3, the training method of the audio-based pose generation model may include the following steps: Step 301, process the sample video to determine the audio token sequence corresponding to the sample video and the pose latent variables containing noise corresponding to the sample video.

[0092] In some possible implementations, the process of determining the audio token sequence corresponding to a sample video can be as follows: First, determine the stride of the audio encoder based on the scale of the noisy pose latent variables in the time dimension. Then, based on the stride, encode the sample video to obtain the audio token sequence corresponding to the sample video. By adjusting the stride of the audio encoder to match the scale of the noisy pose latent variables in the time dimension, the length of the generated audio token sequence is made consistent with the scale of the noisy pose latent variables in the time dimension, improving the alignment between pose and music in the time dimension and significantly enhancing the consistency between pose and audio.

[0093] The step size of the audio encoder can also be referred to as the "Hop size," but this disclosure does not limit it.

[0094] It should be noted that there are no restrictions on the type of audio encoder. It can be any audio encoder that supports a controllable hop size, such as Contrastive Language-Audio Pretraining (CLAP), Mel-frequency Spectrogram Transformer Encoder, Mel-Transformer Encoder, etc., as long as it can output an audio token sequence with the same duration as the pose latent variable.

[0095] For example, when determining the audio token sequence corresponding to a sample video, a pre-trained audio encoder can be used. The sample video is audio encoded to obtain an audio token sequence of length L:

[0096] During audio encoding, the step size of the music encoder is adjusted to make the length of the audio token sequence consistent with the duration of the pose latent variable, i.e.: .

[0097] The specific implementation of step 301 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0098] Step 302: Generate the attitude reference latent variable corresponding to the sample video based on the m first attitude latent variables corresponding to the first m frames of the sample video and the second attitude latent variable corresponding to any frame of the sample video.

[0099] Among them, the first m latent variables in the reference latent variables are m first-pose latent variables, and the last Nm latent variables are all second-pose latent variables, where N is the number of images contained in the sample video.

[0100] It should be noted that m is a positive integer less than N, and its value can be set according to actual needs. This disclosure does not limit it.

[0101] Wherein, the first pose latent variable is the latent variable corresponding to the true labeled pose of the first m frames of the sample video.

[0102] The second pose latent variable is the pose latent variable corresponding to any frame of the sample video, which includes the proportion of the person and the screen size.

[0103] In this disclosure, the m first pose latent variables corresponding to the first m frames of the sample video are used as the first m latent variables of the pose reference latent variable corresponding to the sample video, and the last Nm latent variables of the pose reference latent variable are all set as second pose latent variables. This allows the model to learn both the proportion control of human form and the continuity of the boundaries when splicing different video segments during the training process. In other words, it can ensure that the proportion of human form in different video segments remains consistent when the trained model generates long videos, and that the current video segment is seamlessly connected to the end of the previous video segment.

[0104] In other words, in some possible implementations, in order to accommodate long sequence concatenation, this disclosure can replace the reference pose of the first m frames with the true labeled pose with a certain probability during model training: Then, the pose is encoded to obtain the pose reference latent variables corresponding to the sample video. This allows the model to learn both the case where the proportion is controlled solely by the shape of the person and the case where the action is aligned with the end of the previous segment to ensure boundary continuity.

[0105] Step 303: Generate a mask vector corresponding to the pose reference latent variable based on the value of m and the position of any frame image in the sample video.

[0106] The mask vector is used to mark the location of the real reference frame in the attitude reference latent variables.

[0107] In this disclosure, after generating the pose reference latent variables corresponding to the sample video, a mask vector corresponding to the pose reference latent variables can be generated based on the value of m and the position of any frame image in the sample video. Thus, the first m latent variables in the pose reference latent variables can be marked by the mask vector to mark the position of the real reference frame in the reference latent variables, prompting the model to fit the reference pose as closely as possible at these positions.

[0108] Step 304: Input the pose latent variables containing noise, the pose reference latent variables, the mask vector, and the audio token sequence into the initial generation model to obtain the prediction noise of the initial generation output.

[0109] In some possible implementations, before inputting the pose latent variables containing noise, the pose reference latent variables, the mask vector, and the audio token sequence into the initial generative model, a 3D position index can be assigned to each pose latent variable token (spatial position (w, t)) and audio token (temporal position ttt): Pose token: Audio token: This enables time-sharing location indexing.

[0110] Therefore, when using rotational position encoding, the pose and audio token share the same time dimension t, resulting in higher phase alignment between tokens at the same moment in cross-modal attention. This strengthens the binding between motion events and changes in musical rhythm, enabling the generated pose to respond quickly to audio rhythm, musical phrase transitions, and energy changes. This design differs from conventional text-image settings where images use (0, w, t) and text uses (...). The practice of using ,0,0) can significantly improve rhythm alignment and local responsiveness to musical changes.

[0111] In some possible implementations, the shared time dimension can be not only the t dimension in the rotation position encoding, but can also be shared between the pose and audio modalities by explicitly adding a timestamp embedding vector, or it can be designed to have a higher attention bias by explicitly adding the same t token during attention calculation. This disclosure does not limit this to any particular implementation.

[0112] Step 305: Based on the difference between the predicted noise and the actual noise, the initial generation model is corrected to obtain the corrected generation model.

[0113] Step 306: Based on the corrected generative model, return to the step of performing noise prediction until the target generative model is obtained.

[0114] The specific implementation of steps 304 to 306 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0115] In this embodiment, the sample video is first processed to determine the audio token sequence corresponding to the sample video and the pose latent variable containing noise corresponding to the sample video. Then, based on the m first pose latent variables corresponding to the first m frames of the sample video and the second pose latent variable corresponding to any frame of the sample video, a pose reference latent variable corresponding to the sample video is generated. Based on the value of m and the position of any frame of the image in the sample video, a mask vector corresponding to the pose reference latent variable is generated. Then, the pose latent variable containing noise, the pose reference latent variable, the mask vector, and the audio token sequence are input into the initial generation model to obtain the predicted noise of the initial generation output. Based on the difference between the predicted noise and the actual noise, the initial generation model is corrected to obtain the corrected generation model. Finally, based on the corrected generation model, the step of performing noise prediction is returned until the target generation model is obtained. Therefore, by using the latent variables of the real-labeled poses corresponding to the first m frames of the sample video as the first m pose reference latent variables of the sample video, and using the pose latent variables corresponding to any frame of the video as the remaining pose latent variables in the reference latent variables, the model can learn the transition between different video segments during training. Based on the value of m and the position of any frame, a mask vector is generated, allowing the model to learn to control the proportions of the human figure shape during training. Then, the pose latent variables containing noise, the pose reference latent variables, the mask vector, and the audio token sequence are input into the initial generation model to obtain the predicted noise as the output. Based on the difference between the predicted noise and the actual noise, the model is corrected, thereby improving the training effect and reliability of the audio-based pose generation model.

[0116] Figure 4 is a flowchart illustrating a training method for an audio-based pose generation model according to an embodiment of this disclosure.

[0117] As shown in Figure 4, the training method of the audio-based pose generation model may include the following steps: Step 401, process the sample video to determine the audio token sequence corresponding to the sample video and the pose latent variables containing noise corresponding to the sample video.

[0118] Step 402: Determine the pose reference latent variables and the mask vector corresponding to the pose reference latent variables in the sample video.

[0119] The mask vector is used to mark the location of the real reference frame in the attitude reference latent variables.

[0120] The specific implementation of steps 401 to 402 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0121] Step 403: The pose latent variables containing noise, the pose reference latent variables, and the mask vector are concatenated along the channel dimension to obtain the concatenated tensor.

[0122] In this disclosure, after determining the pose reference latent variables and the mask vector corresponding to the pose reference latent variables in the video, the initial generation model can be iteratively trained based on the noisy pose latent variables, the pose reference latent variables, the mask vector, and the audio token sequence. During model training, the noisy pose latent variables, the pose reference latent variables, and the mask vector can first be concatenated along the channel dimension to obtain a concatenated tensor.

[0123] For example, the pose latent variables containing noise, the pose reference latent variables, and the mask vector are concatenated along the channel dimension to obtain the concatenated tensor: ,in, These are latent attitude variables that include noise. For attitude reference hidden variables.

[0124] Step 404: Input the concatenated tensor and audio token sequence into the initial generation model to obtain the prediction noise of the initial generation output.

[0125] In this disclosure, after concatenating the noisy pose latent variables, pose reference latent variables, and mask vector along the channel dimension, the resulting tensor and audio token sequence are input into the initial generation model to obtain the predicted noise of the initial generation output. This enables model training based on multimodal (pose and audio) features, as well as reference pose and mask constraints. This allows the model to learn control over the proportions of the character's shape and the alignment of the pose with the music during training, thus improving the model training quality.

[0126] For example, when the concatenated tensor and the audio token sequence are input into the initial generation model, the predicted noise output by the model is: .

[0127] in, For the initial generation model, It is an audio token sequence.

[0128] Step 405: Based on the difference between the predicted noise and the actual noise, the initial generation model is corrected to obtain the corrected generation model.

[0129] For example, when determining the difference between predicted noise and actual noise, the loss function is based on the standard mean squared error: ,in, Let L be the actual noise, and L be the loss function value, i.e., the difference. It is the square of the L2 norm.

[0130] In some possible implementations, classifier-free guidance can also be used during model training, which improves controllability by discarding music conditions with a certain probability.

[0131] Step 406: Based on the corrected generative model, return to the step of performing noise prediction until the target generative model is obtained.

[0132] The specific implementation of steps 405 to 406 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0133] In this embodiment, the sample video is first processed to determine the audio token sequence and the noisy pose latent variable corresponding to the sample video. Then, the pose reference latent variable and the mask vector corresponding to the pose reference latent variable are determined. The noisy pose latent variable, the pose reference latent variable, and the mask vector are concatenated along the channel dimension to obtain a concatenated tensor. Next, the concatenated tensor and the audio token sequence are input into the initial generation model to obtain the predicted noise of the initial generation output. Then, based on the difference between the predicted noise and the actual noise, the initial generation model is corrected to obtain a corrected generation model. Finally, based on the corrected generation model, the noise prediction step is returned until the target generation model is obtained. Thus, by concatenating and fusing multimodal features along the channel dimension, and training the model based on the fused multimodal features and the audio token sequence, the training effect and quality of the audio-based pose generation model are improved.

[0134] Figure 5 is a flowchart illustrating an audio-based pose generation method according to an embodiment of this disclosure.

[0135] As shown in Figure 5, the audio-based pose generation method may include the following steps: Step 501, receiving a pose sequence generation request, wherein the generation request includes a reference image and an audio sequence.

[0136] The reference images can be used as reference figures for the dancers. There can be one or more reference images. The specific content and number of reference images can be determined according to actual needs. This disclosure does not limit this.

[0137] It should be noted that the pose sequence generation request can be generated and sent by the user. The user can select or upload an audio sequence and one or more reference images on the client. Based on the user-selected audio sequence and reference images, the client generates a pose sequence generation request and sends it to the server.

[0138] Step 502: Perform pose estimation and encoding processing on the reference image to determine the first pose reference latent variable and the first mask vector.

[0139] In this disclosure, after receiving a pose sequence generation request, the server can perform pose estimation on the reference image in the request to obtain a reference pose, then encode the reference pose to obtain a first pose reference latent variable, and construct a first mask vector of the same size for the first pose reference latent variable.

[0140] For example, in some possible implementations, when determining the first pose reference latent variables and the first mask vector, the pose can be estimated from a user-provided reference image, thus resolving the reference image into a reference pose. After that, Convert to a pose vector representation, and copy T frames in the time dimension to obtain a reference pose tensor that only carries the character's proportions and screen scale, without specific actions: The first attitude reference hidden variable is obtained by encoding the reference tensor using VAE: And construct a first mask vector with the same size as the first attitude reference hidden variable: This disclosure does not impose any limitations on this matter.

[0141] Step 503: Perform audio encoding on the audio sequence to obtain the audio token sequence corresponding to the audio sequence.

[0142] It should be noted that the specific implementation of audio encoding of the audio sequence to obtain the corresponding audio token sequence can be referred to the relevant descriptions in the above embodiments of this disclosure, and will not be repeated here.

[0143] Step 504: Input the preset pure noise attitude latent variables, the first attitude reference latent variables, the first mask vector and the audio token sequence into the generation model to obtain the attitude latent variables output by the generation model.

[0144] The generative model is trained using the audio-based pose generation model method described in the above embodiments of this disclosure.

[0145] Among them, the pure noise attitude latent variable can be obtained by initialization with standard Gaussian noise, and this disclosure does not limit it.

[0146] It should be noted that the specific type and structure of the generated model can be referred to the relevant descriptions in the above embodiments of this disclosure, and will not be repeated here.

[0147] Among them, the pose latent variable is the pose latent variable corresponding to the audio sequence output by the generated model.

[0148] In this disclosure, the audio token sequence and pose reference latent variables are determined. The pure noise attitude latent variables are initialized from standard Gaussian noise, and then denoised stepwise according to a preset diffusion step S to obtain the final attitude latent variables. .

[0149] Step 505: Decode the latent attitude variables to obtain the attitude sequence corresponding to the audio sequence.

[0150] In this disclosure, after obtaining the pose latent variables output by the generative model, the pose sequence corresponding to the audio sequence is obtained by decoding the pose latent variables. Thus, through multimodal feature fusion and the constraints of reference variables and masks, the pose sequence generated by the model can be kept consistent and aligned with the audio sequence. At the same time, the proportion of the human figure in the pose sequence is consistent with the reference image provided by the user. Finally, a pose sequence that is temporally coherent, rhythmically follows the music, and has the same proportion of the human figure as the reference image can be obtained.

[0151] Example, for pose latent variables When decoding to obtain the attitude sequence corresponding to the audio sequence, the VAE encoder can first be used to process the attitude latent variables. Decoding is performed to obtain the pose tensor, i.e., the multi-channel pose image, i.e., the pose tensor. Then, the two-dimensional coordinates and confidence scores of each keypoint are recovered using argmax to obtain the final pose sequence corresponding to the audio sequence. .

[0152] Here, argmax is a function that finds the independent variable (or index) that maximizes the objective function or sequence.

[0153] Specifically, for each keypoint, argmax is executed on the corresponding channel to recover the coordinates, and the minimum value of the maximum value of the channel in the x or y direction is used as the confidence level. : , , Compared to two-dimensional heatmaps, this representation reduces the single-frame dimension from O(H×W) to O(H+W), significantly reducing the output dimension and computational cost in long sequence scenarios, while retaining stability similar to classification supervision.

[0154] In some possible implementations, after decoding the pose latent variables to obtain the pose sequence corresponding to the audio sequence, if the pose sequence includes hand keypoints, the hand keypoints are complex and are often magnified during processing. Therefore, when decoding the pose sequence corresponding to the audio sequence, the coordinates of the hand keypoints can first be scaled down along the direction pointing to the wrist to obtain the updated pose sequence. This scaling-down strategy, by periodically scaling the hand back to the original proportions consistent with the body, improves the generation quality of densely jointed areas of the hand without modifying the mesh structure, resulting in a more natural and accurate hand pose.

[0155] In some possible implementations, after obtaining the pose sequence corresponding to the audio sequence, the pose sequence can be optionally lightly filtered (e.g., smoothed joint trajectories) while ensuring compatibility with the set of reference poses, which is not limited in this disclosure.

[0156] In some possible implementations, after obtaining the pose sequence, the pose sequence can be input into an existing pose-driven video generation engine, and the user reference image can be used as an appearance constraint to generate the final dance video.

[0157] In this embodiment of the disclosure, a pose sequence generation request is first received, pose estimation and encoding processing is performed on the reference image to determine the first pose reference latent variable and the first mask vector, then audio encoding is performed on the audio sequence to obtain the audio token sequence corresponding to the audio sequence, then the preset pure noise pose latent variable, the first pose reference latent variable, the first mask vector and the audio token sequence are input into the generation model to obtain the pose latent variable output by the generation model, and finally the pose latent variable is decoded to obtain the pose sequence corresponding to the audio sequence. Therefore, by estimating and encoding the pose of the user-provided reference image, corresponding pose reference latent variables and masks are obtained. The user-provided audio sequence is encoded to obtain the corresponding audio token sequence. These multimodal features and preset pure noise pose latent variables are input into the trained generative model to obtain the pose latent variables output by the model. Thus, the model can focus on the consistency and alignment between pose and audio, as well as the consistency between the proportion of the human figure in the pose and the reference image, based on the constraints of multimodal feature fusion, reference variables, and masks, in the process of generating pose latent variables. This results in generating more accurate and reliable pose latent variables, obtaining a pose sequence that is temporally coherent, rhythmically follows the music, and maintains the same proportion of the human figure as the reference image. This improves the quality of the generated pose sequence and enhances the effect of audio-based pose generation.

[0158] Figure 6 is a flowchart illustrating an audio-based pose generation method according to an embodiment of this disclosure.

[0159] As shown in Figure 6, the audio-based pose generation method may include the following steps: Step 601, receiving a pose sequence generation request, wherein the generation request includes a reference image and an audio sequence.

[0160] Step 602: Perform pose estimation and encoding processing on the reference image to determine the first pose reference latent variable and the first mask vector.

[0161] Step 603: Perform audio encoding on the audio sequence to obtain the audio token sequence corresponding to the audio sequence.

[0162] The specific implementation of steps 601 to 603 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0163] Step 604: If the duration of the audio sequence is greater than the threshold, the audio sequence is divided into L audio subsequences.

[0164] The threshold can be a duration threshold used to determine whether the current audio sequence is a long audio sequence. It can be set according to actual needs, and this disclosure does not limit it.

[0165] The value of L can be set according to actual needs, and this disclosure does not limit it.

[0166] In other words, if the duration of an audio sequence exceeds a threshold, the audio sequence can be divided into L subsequences according to a preset fixed time window. The size of the time window can be set according to actual needs, and this disclosure does not limit it.

[0167] In this disclosure, when the duration of the audio sequence is greater than a threshold, the current audio sequence can be determined to be a long audio sequence. In order to ensure the quality of the pose sequence corresponding to the long audio sequence, the audio sequence can be divided into L audio subsequences.

[0168] Step 605: For the first audio subsequence among the L audio subsequences, input the preset pure noise attitude latent variable, the first attitude reference latent variable, the first mask vector, and the audio token subsequence corresponding to the first audio subsequence into the generation model to obtain the first attitude latent variable corresponding to the first audio subsequence output by the generation model.

[0169] It should be noted that, for the first audio subsequence among the L audio subsequences, the specific implementation of obtaining the pose latent variable corresponding to the audio subsequence using the generative model can be referred to the relevant description in the above embodiments, and will not be elaborated here.

[0170] Since the first audio subsequence has no adjacent preceding audio subsequence, the first pose reference latent variable used when generating the pose latent variable corresponding to the first audio subsequence is generated based on the reference image.

[0171] This ensures that the shape features of the person in the latent pose variables corresponding to the first audio subsequence are consistent with the shape features of the person in the reference image provided by the user, providing accurate and reliable reference data for determining the pose of subsequent audio segments and improving the quality of pose generation in long audio sequence scenarios.

[0172] Step 606: For the i-th audio subsequence among multiple audio subsequences, determine the i-th attitude reference latent variable and the i-th mask vector corresponding to the i-th audio subsequence based on the last m attitude representations in the attitude latent variables corresponding to the (i-1)-th audio subsequence, where i is an integer greater than 1 and less than L.

[0173] The value of m can be set according to actual needs, and this disclosure does not limit it.

[0174] In this disclosure, after dividing the audio sequence into multiple audio subsequences, when determining the latent variables corresponding to each audio subsequence, for the i-th audio subsequence among the multiple audio subsequences, the i-th attitude reference latent variable and the i-th mask vector corresponding to the i-th audio subsequence can be determined based on the last m attitude representations in the attitude latent variables corresponding to the (i-1)-th audio subsequence. Thus, by using the last attitude representation of the previous audio subsequence to determine the reference and mask constraints, a smooth transition of the corresponding attitude between adjacent audio segments can be achieved, reducing attitude breaks and stuttering.

[0175] In some possible implementations, when using the last m pose representations of the previous segment as a reference, an overlapping window + weighted fusion strategy can be used to smooth the overlapping area, or an additional small network can be used to finely correct the inter-segment boundaries. This disclosure does not limit this.

[0176] Step 607: Input the preset pure noise attitude latent variable, the i-th attitude reference latent variable, the i-th mask vector, and the audio token subsequence corresponding to the i-th audio subsequence into the generation model to obtain the i-th attitude latent variable corresponding to the i-th audio subsequence output by the generation model.

[0177] It should be noted that the specific implementation of using the generative model to determine the pose latent variable corresponding to the i-th audio subsequence can be referred to the relevant description in the above embodiments of this disclosure, and will not be repeated here.

[0178] In other words, in the generation of long sequence segments, when generating corresponding pose latent variables for each segment, the first segment uses a shape-only reference pose as a reference, and each subsequent segment uses the last m pose representations of the previous segment's generation result as a reference. Pose-aware reference latent variables are constructed and masks are set, thereby achieving a smooth transition of poses between segments, avoiding obvious gaps during splicing, ensuring the motion coherence of the pose sequence corresponding to the final audio sequence, and thus improving the viewing experience of subsequent pose-driven generated videos.

[0179] Step 608: Decode each latent pose variable to obtain the pose sequence corresponding to the audio sequence.

[0180] In this disclosure, after obtaining the pose latent variables corresponding to each audio subsequence, the pose latent variables can be decoded to obtain the pose subsequence corresponding to each audio subsequence. Then, the pose subsequences are spliced ​​together according to the time dimension to finally obtain the pose sequence corresponding to the audio sequence, thereby ensuring the continuity and smoothness of the actions in the obtained pose sequence.

[0181] It should be noted that the specific implementation of splicing the various pose subsequences can be set according to actual needs. For example, splicing can be performed through inpainting, direct splicing, weighted average splicing of overlapping frames between segments, interpolation splicing, etc. This disclosure does not limit the specific implementation.

[0182] Among them, the inpainting splicing can model the seams of multiple pose subsequences as pose gaps to be filled. Then, using the poses of the preceding and following segments as constraints, a transition frame that conforms to human kinematics and is temporally coherent is generated through the pose completion model, thereby filling the gaps and solving the problem of pose segment blockage or jump at the seams.

[0183] The specific type and structure of the pose completion model can be set according to actual needs. For example, the pose completion model can be a spatiotemporal graph convolutional network, a graph attention network, a generative pose completion model, etc., and this disclosure does not limit it.

[0184] In other words, this disclosure allows for the concatenation of all pose subsequences along the time dimension after obtaining the pose subsequence corresponding to each audio subsequence, in order to obtain the pose sequence corresponding to the audio sequence.

[0185] In some possible implementations, after obtaining the pose latent variables corresponding to each audio subsequence, all pose latent variables can be concatenated along the time dimension, and then the concatenated pose latent variables can be decoded to obtain the pose sequence corresponding to the audio sequence. This disclosure does not limit this.

[0186] It should be noted that after obtaining the pose latent variables corresponding to each audio subsequence, one can choose to first decode each pose latent variable to obtain the corresponding pose subsequence, and then concatenate all the pose subsequences to obtain the pose sequence corresponding to the audio sequence, or one can choose to first concatenate each pose latent variable, and then decode the concatenated latent variable to obtain the pose sequence corresponding to the audio sequence. The specific implementation can be determined according to actual needs, and this disclosure does not limit it.

[0187] In this embodiment, a pose sequence generation request is first received, and pose estimation and encoding processing is performed on the reference image to determine the first pose reference latent variable and the first mask vector. Then, audio encoding is performed on the audio sequence to obtain the audio token sequence corresponding to the audio sequence. If the duration of the audio sequence is greater than a threshold, the audio sequence is divided into L audio subsequences. Then, for the first audio subsequence among the L audio subsequences, the preset pure noise pose latent variable, the first pose reference latent variable, the first mask vector, and the audio token subsequence corresponding to the first audio subsequence are input into the generation model to obtain the output of the generation model. The first attitude latent variable corresponding to an audio subsequence is used. For the i-th audio subsequence among multiple audio subsequences, the i-th attitude reference latent variable and the i-th mask vector corresponding to the i-th audio subsequence are determined based on the last m attitude representations in the attitude latent variable corresponding to the (i-1)-th audio subsequence. The preset pure noise attitude latent variable, the i-th attitude reference latent variable, the i-th mask vector, and the audio token subsequence corresponding to the i-th audio subsequence are input into the generation model to obtain the i-th attitude latent variable corresponding to the i-th audio subsequence output by the generation model. Finally, each attitude latent variable is decoded to obtain the attitude sequence corresponding to the audio sequence. Therefore, after obtaining the reference image and audio sequence provided by the user, if the audio sequence is long, it is divided into multiple audio sub-sequences. For each audio sub-sequence, a corresponding pose latent variable is determined. When determining the pose latent variable for a sub-sequence other than the first one, the reference latent variable and mask for that sub-sequence are determined by obtaining the last multiple pose representations in the pose latent variable of the preceding audio sub-sequence. This allows the model to generate constraints based on the reference latent variable and mask, effectively reducing pose stuttering and abrupt changes between adjacent audio sub-sequences, ensuring pose coherence and smoothness. Then, each pose latent variable is decoded to obtain the pose sequence corresponding to the audio sequence, thereby improving the pose smoothness and coherence of the obtained pose sequence, improving the quality of music-based pose generation, and thus improving the viewing quality of subsequent music-driven generated videos and enhancing the user experience.

[0188] The system architecture of the music-based pose generation method proposed in this disclosure will be illustrated below with reference to Figure 7. Figure 7 is a schematic diagram of the system architecture of the music-based pose generation method proposed in an embodiment of this disclosure. The architecture shown in Figure 7 is merely an example and is not intended to be limiting.

[0189] As shown in Figure 7, the system architecture includes: a data preprocessing and pose extraction module 701, a pose multi-channel representation module 702, an image latent variable encoding module 703, a music encoding and time-sharing position index module 704, a reference pose condition and mask module 705, a pose generation module 706, a decoding and pose reconstruction module 707, and a long sequence segmentation generation and splicing module 708.

[0190] Among them, the data preprocessing and pose extraction module 701: collects dance videos and corresponding music audio from the Internet or its own data sources; uses pose estimation methods to parse the human body key points in each frame into two-dimensional coordinates and confidence scores; segments the training samples according to a fixed time length (such as 256 frames per segment) and performs quality filtering.

[0191] Pose multichannel representation module 702: Converts the pose of each frame into a one-hot sparse vector and stacks them in the time dimension to form a multichannel image pose tensor.

[0192] Image latent variable encoding module 703: Encodes the multi-channel image pose tensor into a low-dimensional latent variable representation through a pre-trained graph VAE, resulting in smaller dimensions and a more regular structure.

[0193] Music Encoding and Time-Shared Location Index Module 704: Uses a pre-trained audio encoder to map audio to a sequence of tokens; by controlling the step size, the length of the audio token sequence is made consistent with the time length of the latent variables, and shared time dimension coordinates are specified for the music and pose latent variables.

[0194] Reference pose condition and mask module 705: Obtains reference pose (single frame or multiple frames) from user input or historical sequence, converts it into one-hot form and encodes it as reference latent variable; constructs a binary mask marking the position of reference frame to prompt the model to strictly align with the reference pose at these time points.

[0195] Pose generation module 706: It adopts a pose generation model to denoise and predict pose latent variables containing noise; the inputs are the current pose latent variables containing noise, the pose reference latent variables, the mask, and the music token sequence; the loss function is the standard ε-predictive loss.

[0196] Decoding and Pose Reconstruction Module 707: Decodes the denoised latent variables into a multi-channel one-hot pose image; recovers the two-dimensional coordinates and confidence of each keypoint through argmax.

[0197] Long sequence segmentation generation and splicing module 708: Divides long audio into segments of fixed length and adopts an overlapping sampling strategy; uses several frames at the end of the previous segment as the reference pose of the current segment to ensure smooth connection of actions at segment boundaries.

[0198] It should be noted that the modules in this system 700 can be deployed on the same server or in a distributed manner, depending on actual needs. This disclosure does not impose any restrictions on this.

[0199] It should be noted that the training method of the music-based pose generation model and the specific content of each module in the system 700 used in the pose generation method proposed in this disclosure can be referred to the relevant descriptions in the above embodiments of this disclosure, and will not be repeated here.

[0200] The system framework principle of the music-based pose generation model training method proposed in this disclosure will be illustrated below with reference to Figure 8. Figure 8 is a schematic diagram of the system framework principle of the music-based pose generation model training method proposed in this disclosure. The system architecture shown in Figure 8 is merely an example and is not intended to be limiting.

[0201] As shown in Figure 8, the system framework includes a one-hot encoder 801, a VAE encoder 802, an audio encoder 803, and a pose generation model 804. The framework shown in Figure 8 is merely an example and is not intended to be limiting.

[0202] Pose estimation is performed on the sample video to obtain the pose of each frame. The pose of each frame is encoded using a one-hot encoder 801 to obtain the corresponding one-hot encoded representation and the pose vectors and confidence scores of each keypoint in each frame in the horizontal and vertical directions. Then, all pose vectors corresponding to all keypoints in each frame are stacked in the channel dimension and time dimension respectively to obtain the multi-channel, image-like pose tensor corresponding to the sample video. Next, these channels are grouped into three-channel groups and each group of three channels is input into the VAE encoder 802 to obtain the compressed pose latent variables. Then, the pose latent variables are noise-added to obtain the pose latent variables containing noise.

[0203] Next, based on the m pose latent variables corresponding to the first m frames of the sample video and the pose latent variables corresponding to any frame of the video, the pose reference latent variables corresponding to the video are generated, and based on the value of m and the position of any frame of the video, a mask vector corresponding to the pose reference latent variables is constructed.

[0204] The audio sequence is extracted from the sample video and encoded using an audio encoder 803 to obtain the corresponding audio token sequence. The sampling step size of the audio encoder 803 is adjusted to match the scale of the noisy pose latent variables in the time dimension, so that the length of the audio token sequence is consistent with the length of the latent variables in the time dimension.

[0205] Furthermore, matching time position indices can be assigned to the latent variables and the audio token sequence to ensure that the j-th column of the latent variables has the same time stamp as the j-th music token. This disclosure does not limit this.

[0206] Then, the pose reference latent variables, mask vector, and pose latent variables containing noise are concatenated along the channel dimension to obtain the concatenated tensor. The audio token sequence and the concatenated tensor are then input into the pose generation model 804 to obtain the predicted noise output by model 804.

[0207] During the training phase, the pose generation model can be corrected based on the difference between the predicted noise and the actual noise until the target pose generation model is obtained.

[0208] During the generation phase, the pose generation model gradually recovers clean pose latent variables by predicting noise and iteratively denoising it. Finally, the VAE encoder 802 reconstructs the denoised latent variables into an intermediate image representation and converts it into the final pose sequence through the argmax operation.

[0209] It should be noted that the specific implementation of each process in the system framework schematic diagram shown in Figure 8 can be referred to the relevant descriptions in the above embodiments of this disclosure, and will not be repeated here.

[0210] The following example, with reference to Figure 9, illustrates the system application flow of the music-based pose generation method proposed in this disclosure. Figure 9 is a schematic diagram of the system application flow of the music-based pose generation method proposed in this disclosure.

[0211] In Figure 9, the system 900 includes two subsystems: a music-driven pose generation subsystem 901 and a pose-driven video generation subsystem 902. The structure of the music-driven pose generation subsystem 901 can be the system architecture shown in Figure 7. No restrictions are imposed here.

[0212] It should be noted that the structure of the attitude-driven video generation subsystem 902 can be configured according to actual needs. For example, the subsystem 902 may include an attitude-driven video generation model, etc., and this disclosure does not limit this.

[0213] As shown in Figure 9, the user first inputs a piece of music. The system then calls the music-driven posture generation subsystem 901 to automatically generate a sequence of dance postures that highly match the rhythm changes. These posture sequences are then further parsed and converted into continuous, natural dance movement trajectories. Finally, the system uses the posture-driven video generation subsystem 902, combined with a reference image provided by the user, to generate a highly realistic dance video that matches the appearance of the reference image and whose movements are completely synchronized with the music.

[0214] Based on this complete process, this disclosure possesses excellent versatility and scalability, and can be applied to multiple practical scenarios. For example: it can enable automatic choreography in virtual idols and virtual hosts, allowing digital characters to present professional dance performances in real time according to music; it can provide real-time backup dance functions for short video platforms and live streaming scenarios, enhancing the interactivity and entertainment value for creators; in the fields of advertising and brand creativity, it can quickly generate dance clips that meet rhythm and style requirements by using character IPs or brand images as references; and in educational and entertainment products, it can be used to automatically create dance demonstration videos, providing users with more intuitive and vivid learning materials.

[0215] These application scenarios fully demonstrate the broad value and practical potential of this disclosure in areas such as creative production, character-driven approaches, and music interaction.

[0216] Specifically, the model or generation method disclosed herein can be used as a core module for music-driven dance generation and embedded into various products or systems. A typical application process is shown below, which is only an example and is not intended to be limiting: 1. User interaction and front-end collection: The user selects or uploads an audio clip and one or more reference images of the dancer on the client; the front-end uploads the music audio and images to the server.

[0217] 2. Backend Processing Flow: Reference Pose Extraction: The server performs pose estimation on the reference person image to obtain the reference pose. The first pose reference latent variable is constructed. Music feature extraction and segmentation: The audio is encoded to obtain an audio token sequence A; the entire audio segment is segmented into several segments according to a fixed time window. Segment pose generation: For the first segment, the pose generation model is run using the first pose reference latent variable to generate the pose sequence for that segment; for subsequent segments, the pose generated at the end of the previous segment (m frames) is used as the pose reference, the reference latent variable and mask are updated, and the pose for that segment is generated again; the above steps are repeated until the entire music segment is covered. Post-processing of pose sequence: The generated pose sequence may optionally be lightly filtered (e.g., smoothing joint trajectories) while ensuring geometric compatibility with the reference pose.

[0218] Steps 1 and 2 are steps in which the music-driven gesture generation subsystem 901 generates the gesture sequence corresponding to the audio sequence.

[0219] 3. Downstream video generation: The generated posture sequence is input into the posture-driven video generation subsystem 902, and the user-provided reference image is used as an appearance constraint to generate the final dance video.

[0220] Because the pose representation and generation process disclosed herein is decoupled from the specific character appearance, the same model can serve multiple different product lines, achieving shared underlying capabilities.

[0221] To implement the above embodiments, this disclosure also proposes a training device for an audio-based pose generation model.

[0222] Figure 10 is a schematic diagram of the structure of the training device for the audio-based pose generation model provided in the embodiments of this disclosure.

[0223] As shown in Figure 10, the training device 1000 for the audio-based pose generation model includes: a first processing module 1001, a determination module 1002, a first input module 1003, a correction module 1004, and a second processing module 1005.

[0224] The first processing module 1001 is used to process the sample video to determine the audio token sequence corresponding to the sample video and the pose latent variable containing noise corresponding to the sample video; the determination module 1002 is used to determine the pose reference latent variable corresponding to the sample video and the mask vector corresponding to the pose reference latent variable, wherein the mask vector is used to mark the position of the real reference frame in the pose reference latent variable; the first input module 1003 is used to input the pose latent variable containing noise, the pose reference latent variable, the mask vector and the audio token sequence into the initial generation model to obtain the prediction noise of the initial generation output; the correction module 1004 is used to correct the initial generation model based on the difference between the prediction noise and the actual noise to obtain the corrected generation model; the second processing module 1005 is used to return to the noise prediction step based on the corrected generation model until the target generation model is obtained.

[0225] In one possible implementation of this disclosure, the first processing module 1001 is specifically used for: performing pose estimation on the sample video to determine the coordinates of each key point in each frame of the sample video; encoding the coordinates of each key point to determine the first orientation pose vector and first confidence level, and the second orientation pose vector and second confidence level corresponding to each key point; stacking all pose vectors corresponding to all key points in each frame of the image in the channel dimension and the time dimension respectively to obtain the pose tensor corresponding to the sample video; and encoding and adding noise to the pose tensor to obtain the pose latent variable containing noise corresponding to the sample video.

[0226] In one possible implementation of this disclosure, the first processing module 1001 is specifically used to: when the image contains hand key points, magnify the hand key points along the direction away from the wrist, with the wrist as the center, to obtain the updated coordinates of the hand key points; and encode the updated coordinates of the hand key points.

[0227] In one possible implementation of this disclosure, the first processing module 1001 is specifically used to: determine the step size of the audio encoder based on the scale of the noisy pose latent variables in the time dimension; and based on the step size, perform audio encoding on the sample video to obtain the audio token sequence corresponding to the sample video.

[0228] In one possible implementation of this disclosure, the determining module 1002 is specifically used to: generate a pose reference latent variable corresponding to the sample video based on the m first pose latent variables corresponding to the first m frames of the sample video and the second pose latent variable corresponding to any frame of the sample video, wherein the first m latent variables in the reference latent variable are m first pose latent variables, and the last Nm latent variables are all second pose latent variables, where N is the number of images contained in the sample video; and generate a mask vector corresponding to the pose reference latent variable based on the value of m and the position of any frame of the image in the sample video.

[0229] In one possible implementation of this disclosure, the first input module 1003 is specifically used to: concatenate the noisy pose latent variables, pose reference latent variables, and mask vector in the channel dimension to obtain a concatenated tensor; and input the concatenated tensor and the audio token sequence into the initial generation model to obtain the prediction noise of the initial generation output.

[0230] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0231] In this embodiment, the sample video is first processed to determine the audio token sequence corresponding to the sample video and the pose latent variable containing noise corresponding to the sample video. Then, the pose reference latent variable corresponding to the sample video and the mask vector corresponding to the pose reference latent variable are determined. The pose latent variable containing noise, the pose reference latent variable, the mask vector and the audio token sequence are input into the initial generation model to obtain the predicted noise of the initial generation output. Then, based on the difference between the predicted noise and the actual noise, the initial generation model is corrected to obtain the corrected generation model. Finally, based on the corrected generation model, the step of performing noise prediction is returned until the target generation model is obtained. Therefore, by iteratively training the initial generative model using audio token sequences based on sample videos, noisy pose latent variables, pose reference latent variables, and their corresponding mask vectors, the model training based on multimodal feature fusion and constraints of reference latent variables and masks is achieved. This effectively solves the problems of audio and pose asynchrony and pose ratio inconsistency with reference pose in the generated dance videos. Furthermore, based on the difference between the predicted noise and the actual noise in the model output, the model is continuously corrected to obtain the target generative model, thereby improving the stability and accuracy of the model training process and enhancing the performance of the target generative model.

[0232] Figure 11 is a schematic diagram of the structure of the audio-based posture generation device provided in the embodiments of this disclosure.

[0233] As shown in Figure 11, the audio-based attitude generation device 1100 includes: a receiving module 1101, a third processing module 1102, an encoding module 1103, a second input module 1104, and a decoding module 1105.

[0234] The receiving module 1101 is used to receive a pose sequence generation request, wherein the generation request includes a reference image and an audio sequence; the third processing module 1102 is used to perform pose estimation and encoding processing on the reference image to determine the first pose reference latent variable and the first mask vector; the encoding module 1103 is used to perform audio encoding on the audio sequence to obtain the audio token sequence corresponding to the audio sequence; the second input module 1104 is used to input the preset pure noise pose latent variable, the first pose reference latent variable, the first mask vector and the audio token sequence into the generation model to obtain the pose latent variable output by the generation model, wherein the generation model is trained based on any one of the methods described in claims 1-6; the decoding module 1105 is used to decode the pose latent variable to obtain the pose sequence corresponding to the audio sequence.

[0235] In one possible implementation of this disclosure, the second input module 1104 is specifically used to: divide the audio sequence into L audio sub-sequences when the duration of the audio sequence is greater than a threshold; for the first audio sub-sequence among the L audio sub-sequences, input the preset pure noise attitude latent variable, the first attitude reference latent variable, the first mask vector and the audio token sub-sequence corresponding to the first audio sub-sequence into the generation model to obtain the first attitude latent variable corresponding to the first audio sub-sequence output by the generation model.

[0236] In one possible implementation of this disclosure, the second input module 1104 is further configured to: for the i-th audio subsequence among multiple audio subsequences, determine the i-th attitude reference latent variable and the i-th mask vector corresponding to the i-th audio subsequence based on the last m attitude representations in the attitude latent variable corresponding to the (i-1)-th audio subsequence, where i is an integer greater than 1 and less than L; input the preset pure noise attitude latent variable, the i-th attitude reference latent variable, the i-th mask vector, and the audio token subsequence corresponding to the i-th audio subsequence into the generation model to obtain the i-th attitude latent variable corresponding to the i-th audio subsequence output by the generation model.

[0237] In one possible implementation of this disclosure, the decoding module 1105 is further configured to: reduce the coordinates of the hand key points along the direction pointing to the wrist when the pose sequence contains hand key points, so as to obtain the pose sequence updated with hand key points.

[0238] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0239] In this embodiment of the disclosure, a pose sequence generation request is first received, pose estimation and encoding processing is performed on the reference image to determine the first pose reference latent variable and the first mask vector, then audio encoding is performed on the audio sequence to obtain the audio token sequence corresponding to the audio sequence, then the preset pure noise pose latent variable, the first pose reference latent variable, the first mask vector and the audio token sequence are input into the generation model to obtain the pose latent variable output by the generation model, and finally the pose latent variable is decoded to obtain the pose sequence corresponding to the audio sequence. Therefore, by estimating and encoding the pose of the user-provided reference image, corresponding pose reference latent variables and masks are obtained. The user-provided audio sequence is encoded to obtain the corresponding audio token sequence. These multimodal features and preset pure noise pose latent variables are then input into the trained generative model to obtain the pose latent variables output by the model. This allows the model to focus on the consistency and alignment between pose and audio during the generation of pose latent variables based on the fusion of multimodal features and the constraints of reference variables and masks. The proportion of the human figure in the pose remains consistent with the reference image, generating more accurate and reliable pose latent variables. Then, by decoding the pose latent variables, the pose sequence corresponding to the audio sequence is obtained, thus obtaining a pose sequence that is temporally coherent, rhythmically follows the music, and maintains the same proportion of the human figure as the reference image. This improves the quality of the generated pose sequence and enhances the effect of audio-based pose generation.

[0240] Figure 12 illustrates a schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0241] As shown in Figure 12, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0242] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0243] The computing unit 1201 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as training methods for audio-based pose generation models and audio-based pose generation methods. For example, in some embodiments, the training methods for audio-based pose generation models and audio-based pose generation methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by computing unit 1201, one or more steps of the training method for the audio-based pose generation model and one or more steps of the audio-based pose generation method described above can be performed. Alternatively, in other embodiments, computing unit 1201 can be configured to execute the training method for the audio-based pose generation model and the audio-based pose generation method by any other suitable means (e.g., by means of firmware).

[0244] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0245] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a general-purpose computer, a special-purpose computer, or other programmable processor or controller, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0246] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0247] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0248] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0249] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0250] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0251] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this disclosure, the words "if" and "suppose" as used may be interpreted as "when," "when," "in response to determination," or "in the circumstances."

[0252] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for an audio-based pose generation model, characterized in that, include: The sample video is processed to determine the audio token sequence corresponding to the sample video and the pose latent variable containing noise corresponding to the sample video. Determine the pose reference latent variable corresponding to the sample video and the mask vector corresponding to the pose reference latent variable, wherein the mask vector is used to mark the position of the real reference frame in the pose reference latent variable; input the noisy pose latent variable, the pose reference latent variable, the mask vector and the audio token sequence into the initial generation model to obtain the prediction noise of the initial generation output; Based on the difference between the predicted noise and the actual noise, the initial generation model is corrected to obtain the corrected generation model; based on the corrected generation model, the step of performing noise prediction is returned until the target generation model is obtained.

2. The method as described in claim 1, characterized in that, The process of determining the noisy pose latent variables corresponding to the sample video includes: performing pose estimation on the sample video to determine the coordinates of each keypoint in each frame of the sample video; encoding the coordinates of each keypoint to determine the first orientation pose vector and first confidence level, and the second orientation pose vector and second confidence level corresponding to each keypoint; stacking all pose vectors corresponding to all keypoints in each frame of the image in both the channel dimension and the time dimension to obtain the pose tensor corresponding to the sample video; and encoding and adding noise to the pose tensor to obtain the noisy pose latent variables corresponding to the sample video.

3. The method as described in claim 2, characterized in that, Encoding the coordinates of each key point includes: when the image contains hand key points, magnifying the hand key points away from the wrist, centered on the wrist, to obtain the updated coordinates of the hand key points; and encoding the updated coordinates of the hand key points.

4. The method as described in claim 2, characterized in that, The process of determining the audio token sequence corresponding to the sample video includes: determining the step size of the audio encoder based on the scale of the noisy pose latent variable in the time dimension; and encoding the sample video into audio based on the step size to obtain the audio token sequence corresponding to the sample video.

5. The method according to any one of claims 1-4, characterized in that, The step of determining the pose reference latent variable corresponding to the sample video and the mask vector corresponding to the pose reference latent variable includes: generating the pose reference latent variable corresponding to the sample video based on the m first pose latent variables corresponding to the first m frames of the sample video and the second pose latent variable corresponding to any frame of the sample video, wherein the first m latent variables of the reference latent variable are the m first pose latent variables, and the last Nm latent variables are all second pose latent variables, where N is the number of images contained in the sample video; and generating the mask vector corresponding to the pose reference latent variable based on the value of m and the position of any frame of the image in the sample video.

6. The method as described in claim 5, characterized in that, The step of inputting the noisy pose latent variables, the pose reference latent variables, the mask vector, and the audio token sequence into the initial generation model to obtain the predicted noise of the initial generation output includes: concatenating the noisy pose latent variables, the pose reference latent variables, and the mask vector along the channel dimension to obtain a concatenated tensor; and inputting the concatenated tensor and the audio token sequence into the initial generation model to obtain the predicted noise of the initial generation output.

7. An audio-based pose generation method, characterized in that, include: A pose sequence generation request is received, wherein the generation request includes a reference image and an audio sequence; pose estimation and encoding processing is performed on the reference image to determine a first pose reference latent variable and a first mask vector; audio encoding is performed on the audio sequence to obtain an audio token sequence corresponding to the audio sequence; preset pure noise pose latent variables, the first pose reference latent variables, the first mask vector, and the audio token sequence are input into a generation model to obtain the pose latent variables output by the generation model, wherein the generation model is trained based on the method described in any one of claims 1-6; the pose latent variables are decoded to obtain the pose sequence corresponding to the audio sequence.

8. The method as described in claim 7, characterized in that, The step of inputting a preset pure noise attitude latent variable, a first attitude reference latent variable, a first mask vector, and an audio token sequence into a generation model to obtain the attitude latent variable output by the generation model includes: if the duration of the audio sequence is greater than a threshold, dividing the audio sequence into L audio sub-sequences; for the first audio sub-sequence among the L audio sub-sequences, inputting the preset pure noise attitude latent variable, the first attitude reference latent variable, the first mask vector, and the audio token sub-sequence corresponding to the first audio sub-sequence into the generation model to obtain the first attitude latent variable corresponding to the first audio sub-sequence output by the generation model.

9. The method as described in claim 8, characterized in that, The method further includes: for the i-th audio subsequence among the plurality of audio subsequences, determining the i-th attitude reference latent variable and the i-th mask vector corresponding to the i-th audio subsequence based on the last m attitude representations in the attitude latent variable corresponding to the (i-1)-th audio subsequence, where i is an integer greater than 1 and less than L; inputting the preset pure noise attitude latent variable, the i-th attitude reference latent variable, the i-th mask vector, and the audio token subsequence corresponding to the i-th audio subsequence into the generation model to obtain the i-th attitude latent variable corresponding to the i-th audio subsequence output by the generation model.

10. The method according to any one of claims 7-9, characterized in that, After decoding the latent pose variables to obtain the pose sequence corresponding to the audio sequence, the method further includes: if the pose sequence contains hand key points, reducing the coordinates of the hand key points along the direction pointing to the wrist to obtain the pose sequence updated with the hand key points.

11. A training device for an audio-based pose generation model, characterized in that, The apparatus includes: a first processing module for processing sample videos to determine an audio token sequence corresponding to the sample videos and a noisy pose latent variable corresponding to the sample videos; a determining module for determining a pose reference latent variable corresponding to the sample videos and a mask vector corresponding to the pose reference latent variable, wherein the mask vector is used to mark the position of the real reference frame in the pose reference latent variable; a first input module for inputting the noisy pose latent variable, the pose reference latent variable, the mask vector, and the audio token sequence into an initial generation model to obtain the predicted noise of the initial generation output; a correction module for correcting the initial generation model based on the difference between the predicted noise and the actual noise to obtain a corrected generation model; and a second processing module for returning to the noise prediction step based on the corrected generation model until a target generation model is obtained.

12. An audio-based posture generation device, characterized in that, The apparatus includes: a receiving module for receiving a pose sequence generation request, wherein the generation request includes a reference image and an audio sequence; a third processing module for performing pose estimation and encoding processing on the reference image to determine a first pose reference latent variable and a first mask vector; an encoding module for performing audio encoding on the audio sequence to obtain an audio token sequence corresponding to the audio sequence; a second input module for inputting a preset pure noise pose latent variable, the first pose reference latent variable, the first mask vector, and the audio token sequence into a generation model to obtain the pose latent variable output by the generation model, wherein the generation model is trained based on the method described in any one of claims 1-6; and a decoding module for decoding the pose latent variable to obtain the pose sequence corresponding to the audio sequence.

13. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the training method of the audio-based pose generation model according to any one of claims 1-6, or the audio-based pose generation method according to any one of claims 7-10.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the training method of the audio-based pose generation model according to any one of claims 1-6, or the audio-based pose generation method according to any one of claims 7-10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the audio-based pose generation model according to any one of claims 1-6, or the audio-based pose generation method according to any one of claims 7-10.

Images