A dance generation method based on residual perceptual quantization and residual mask modeling

Abstract

The application discloses a multifunctional dance generation method based on residual perception quantization and residual mask modeling, and belongs to the technical field of artificial intelligence. The dance is generated based on a MORE framework, and the method comprises the following steps: introducing a mask modeling method to generate dance actions through a mask modeling mechanism; a residual perception quantization (RAQ) module based on the MORE framework encodes the dance actions into multi-layer discrete motion features, layer by layer captures the incremental details and complexity of the actions, and obtains the detailed information of the actions; the MORE framework uses a residual mask modeling (RBMM) model and a bidirectional transformer model to predict and generate the sequence of dance actions layer by layer under the condition of music, and generates dance actions matched with the music through the RBMM model. Through the combination of the hierarchical mask strategy and the music features, the method improves the hierarchy and the degree of detail of the generation process, and optimizes the matching degree of the generated dance actions and the music features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a dance generation method based on residual perception quantization and residual mask modeling. BACKGROUND

[0002] Dance is a performing art that uses the body to perform various elegant or difficult movements, usually accompanied by music, and is an artistic form that uses rhythmic movements as the main means of expression. There is an important link in dance creation, choreography, which is the arrangement and design of dance according to the given music melody, conveying the dancer's thoughts and expressions through body movements and details. Traditional choreography usually relies on the experience and creativity of professional choreographers, and has a certain subjectivity, and is a complex and labor-intensive task.

[0003] With the development of technology, the choreography process can be combined with computer vision and artificial intelligence technology, and a complete dance movement can be generated according to the music by matching the corresponding dance sequence through the computer model and the music. In the early stage, a retrieval method is usually used to generate dance, which divides dance movements into several fixed-length segments and matches and splices them according to the structure of the music to generate a complete dance sequence. With the development of deep learning technology, direct generation methods have gradually become mainstream, which can generate dance movements from scratch by training models. Diffusion models have been widely used in recent years, and with their gradual denoising generation process, they can generate high-quality and coherent dance movement sequences.

[0004] Although existing dance generation technology has made some progress in generating dance movements that match music, there are still some defects: many retrieval-based dance generation methods rely on pre-defined dance segments, and cannot flexibly cope with the diversity and complexity of different music, resulting in a lack of sufficient generalization ability of the generated dance movements in the face of diverse music styles and rhythms, and difficulty in capturing the subtle changes of music; while direct generation methods can generate high-quality dance movements, they still face challenges in ensuring the consistency and smoothness of movements over a long period of time, especially when dealing with more complex music, the generated dance movements are prone to be out of sync and broken, thereby affecting the naturalness and aesthetics of the overall dance. Current methods face the difficulty of how to effectively capture the overall action while maintaining the details of the dance when learning and representing high-quality features of dance movements. Traditional quantization methods may cause loss or blurring of dance movement features, and cannot fully reflect the rich expression of dance movements. SUMMARY

[0005] The application aims at providing a dance generation method based on residual perceptual quantization and residual mask modeling to solve the above problems.

[0006] The technical scheme adopted by the application is as follows:

[0007] The dance generation method based on residual perceptual quantization and residual mask modeling generates dance based on the MORE framework, and comprises the following steps: S1: introducing a mask modeling method, generating dance actions through a mask modeling mechanism; S2: based on the residual perceptual quantization RAQ module of the MORE framework, encoding the dance actions into multi-layer discrete motion features, capturing the incremental details and complexity of the actions layer by layer, obtaining the detail information of the actions, using vector quantization to generate basic motion features in the initial layer, and then calculating and storing residual features to enhance the complexity and details of the actions layer by layer; S3: the MORE framework uses the residual mask modeling RBMM module to predict and generate the sequence of dance actions layer by layer under the condition of music by using a bidirectional transformer model, and generates dance actions matching the music through the residual mask modeling module RBMM.

[0008] Further, the MORE framework comprises a residual perceptual quantization RAQ module and a residual-based mask modeling RBMM module; wherein the RAQ module tokenizes the dance actions into a multi-layer discrete feature sequence, retaining the semantic information in the dance actions; and the RBMM module is used to predict the randomly masked action features.

[0009] Further, the step S2 comprises: S201: dividing the dance actions into a dance action sequence d through the residual perceptual quantization RAQ module; S202: encoding the dance action sequence d into a latent vector z through a one-dimensional convolutional encoder ; S203: quantizing the latent vector z ; , and projecting the quantized code sequence back to the dance action space, and using a one-dimensional decoder to reconstruct the dance actions .

[0010] Further, the quantization process in S203 is: through the vector quantization variational autoencoder VQ-VAE, the dance action latent vector z is expressed as an ordered code latent sequence, forming a quantization layer; and the residual quantization RQ module recursively approximates the residual error by calculating the code sequence in each quantization layer. In the multi-layer quantization process, the motion details are effectively preserved, and the accuracy and realism of the dance motion generation are significantly improved, as follows:

[0011]

[0012] wherein the above process starts from the 0th residual ;

[0013] After passing through the residual quantization RQ module, the final representation of the latent sequence is the sum of all quantized sequences, and the final output latent sequence is input into the decoder , which is then used to reconstruct the dance motion.

[0014] Further, the training of the vector quantization variational autoencoder VQ-VAE is performed by the dance motion reconstruction loss, and its loss function is as follows:

[0015]

[0016] wherein denotes the stop gradient operation, the hyperparameter is the weighting factor of embedding constraint, is the identification number, is the code sequence, is the residual, d is the dance motion sequence, is the decoded reconstructed dance motion sequence, and the loss function is optimized by the straight-through gradient estimator.

[0017] Further, the step S3 comprises:

[0018] S301: Mark the residual dance motion latent sequence in the RAQ module as sequence;

[0019] S302: Pre-calculate the features of the music;

[0020] S303: Fuse the dance motion sequence with the music feature embedding and provide it as input to the multi-layer bidirectional mask transformer architecture;

[0021] S304: Generate the dance motion labels for the base layer and the residual layer respectively through the mask transformer in the residual mask modeling RBMM;

[0022] It should be noted that the use of these transformers allows fine generation of dance motion labels that follow the structure of the quantization process layer

[0023] S305: The dance motion labels are decoded and restored to motion sequences by the decoder of the RAQ-VAE, thereby generating dance motions that match the music.

[0024] Furthermore, the multi-layer bidirectional masking transformer architecture is used to implement a random masking process, in which a specified proportion of sequence elements are masked by replacing the original label with a specified [MASK] label. Subsequently, this [MASK] label serves as a placeholder indicating input distortion, prompting the model to infer and reconstruct the original masked label.

[0025] It should be noted that the musical characteristics are obtained according to standard protocols.

[0026] Furthermore, the mask transformer in the mask modeling RBMM includes: a basic transformer FT and a subsequent transformer ST;

[0027] The basic transformer (FT) is used to represent the dance motion markers of the basic layer. ,when When a subset of is masked, the resulting sequence is represented as The fundamental transformer FT uses musical features v and partially masked sequences. The labeling of the prediction mask; the optimization of the basic transformer FT involves minimizing the negative log-likelihood function associated with the prediction target, as shown in the following formula:

[0028]

[0029] in, For a given musical feature v and a partially masked sequence Predicting labels in the case of The probability of;

[0030] The subsequent transformer ST is used to represent the dance motion markers of the residual layer. quantization layer During the training phase, a quantization layer is randomly selected. During the learning process, all the labels from the preceding layers undergo an embedding process, and then they are aggregated to form a composite label embedding input; this label embedding is combined with the musical feature v and the residual quantization layer indicator. These are used together as multiple inputs to ST, and then ST is trained to simultaneously predict the Lth layer. The label. The training process can be represented as:

[0031]

[0032] in Indicates the number of potential vectors. For a given musical feature v and a partially masked sequence And the predicted label in the case of residual layer label L The probability of.

[0033] Further, the mask transformer can simultaneously and in parallel decode multiple dance action features, and comprehensively consider the information of the front and rear action features, and automatically fill in the dance vacancy by inserting the mask feature in the generated dance segment.

[0034] Further, the results of the mask transformer are guided by the CFG predictor without a classifier, and the final output dance label probability logits The calculation formula is:

[0035]

[0036] Wherein, And conditional and unconditional logits, is the guide ratio.

[0037] In summary, due to the adoption of the above technical solutions, the beneficial effects of the present application are:

[0038] The present application introduces a generated mask modeling, generates dance actions through the mask modeling mechanism, so that the generated dance can maintain smoothness and naturalness under various music conditions; based on the residual perception quantization module and the residual mask modeling module of the MORE model, the dance action sequence is hierarchically encoded through the recursive residual quantization strategy, and the dance actions of the basic layer and the residual layer are predicted by using the bidirectional transformer network. This method combines hierarchical mask strategy and music features to improve the hierarchical and detailed degree of the generation process, and optimizes the matching degree of the generated dance action and music features. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is the method flowchart of the present application;

[0040] Figure 2 is the method flowchart of the present application;

[0041] Figure 3 is the overall framework diagram of the present application;

[0042] Figure 4 is the generation result diagram under different conditions based on the MORE framework of the present application. DETAILED DESCRIPTION

[0043] The present application will be described in detail below with reference to the accompanying drawings.

[0044] In order to make the purpose, technical scheme and advantages of the present application more clear and explicit, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0045] The existing dance generation method lacks sufficient generalization ability when facing diversified music styles and rhythms, and it is difficult to capture the subtle changes of music, so the matching degree of the generated dance action with the music is not high, and the continuity of the action is poor.

[0046] As Figures 1-3 shown, one embodiment of the present application proposes a dance generation method based on residual perceptual quantization and residual mask modeling, which improves the flexibility and consistency of dance action generation through the proposed dance generation framework MORE.

[0047] The method comprises the following steps: S1: introducing a mask modeling method, generating dance actions through a mask modeling mechanism; realizing the full function generation of dance, and the generated dance can still maintain smoothness and naturalness under various music conditions; S2: based on the residual perceptual quantization RAQ module of the MORE framework, the dance action is encoded into multi-layer discrete motion features, the incremental details and complexity of the action are captured layer by layer, the detail information of the action is obtained, and the inherent rich semantic information of the action is ensured to be retained; the initial layer uses vector quantization to generate basic motion features, and the subsequent layers calculate and store residual features to enhance the complexity and details of the action layer by layer; S3: the MORE framework uses the residual mask modeling RBMM module to predict and generate the sequence of dance actions layer by layer under the condition of music by using a bidirectional transformer model, and the sequence information of the action is generated into dance actions matched with the music through the residual mask modeling module RBMM.

[0048] The MORE framework comprises a residual perceptual quantization RAQ module and a residual-based mask modeling RBMM module; wherein the RAQ module labels the dance action into a multi-layer discrete feature sequence, and retains the semantic information in the dance action; the RBMM module is used to predict the randomly masked action features.

[0049] Because the quantization process inevitably leads to the reduction of information, a residual perceptual quantization module is introduced, and the step S2 comprises:

[0050] S201: the dance action is divided into a dance action sequence d through the residual perceptual quantization RAQ module;

[0051] S202: the dance action sequence d is converted into a latent vector through a one-dimensional convolutional encoder .

[0052] S203: the latent vector is quantized , and the quantized code sequence is projected back to the dance action space, and a one-dimensional decoder is used to reconstruct the dance action .

[0053] where the quantization process is:

[0054] The dance action latent vector is quantized by the vector-quantized variational autoencoder VQ-VAE is represented as an ordered sequence of code latent forms

[0055] The residual quantization RQ module recursively approximates the residual by computing the code sequence at each quantization layer, as follows:

[0056]

[0057] where the above process starts from the 0th residual ;

[0058] After the residual quantization RQ module, the final representation of the latent sequence is the sum of all quantized sequences, and the final output latent sequence is input into the decoder , which is then used to reconstruct the dance action.

[0059] The training of the vector-quantized variational autoencoder VQ-VAE is performed by the dance action reconstruction loss, whose loss function is as follows:

[0060]

[0061] where denotes the stop-gradient operation, the hyperparameter is the weighting factor of the embedding constraint, is the identification number, is the code sequence, is the residual, d is the dance action sequence, is the decoded reconstructed dance action sequence, and the loss function is optimized by the straight-through gradient estimator.

[0062] The step S3 comprises:

[0063] S301: Mark the residual dance action latent sequence in the RAQ module as a sequence;

[0064] S302: Pre-compute the features of the music;

[0065] S303: Fuse the dance action sequence with the music feature embedding and provide it as input to the multi-layer bidirectional masked transformer architecture;

[0066] In the RAQ module, the residual dance action is marked First, the output of the encoder is passed to a residual vector quantizer to obtain a sequence of dance action labels, which is then combined with the music feature embedding as input to a multi-layer bidirectional masked transformer architecture trained to predict the masked labels. The multi-layer bidirectional masked transformer architecture is used to implement a random masking process in which a specified proportion of the sequence elements are masked by replacing the original labels with a specified [MASK] label, which then serves as a placeholder indicating input distortion, prompting the model to infer and reconstruct the original labels that were masked.

[0067] S304: The masked transformer in the RBMM generates dance action labels for the base layer and the residual layer, respectively, through residual masking modeling;

[0068] S305: The dance action labels are decoded by the decoder of the RAQ-VAE and restored to action sequences, thereby generating dance actions that match the music.

[0069] To further implement hierarchical residual quantization, the masked transformer in the RBMM employs two different masked transformers: a base transformer FT and a subsequent transformer ST, which are used to generate dance action labels for the base layer and the residual layer, respectively;

[0070] The base transformer FT is used to represent the dance action labels for the base layer When a subset of the labels is masked, the resulting sequence is represented as The base transformer FT predicts the masked labels by using the music feature v and the partially masked sequence The base transformer FT aims to accurately predict the masked labels by using the music feature v and the partially masked sequence The base transformer FT aims to accurately predict the masked labels by using the music feature v and the partially masked sequence The specific process is as follows:

[0071] To synthesize the base layer label sequence in N iterations, first start from a blank sequence with all labels hidden, in each iteration, starting from the hidden label sequence, the base transformer FT calculates the probability of various labels appearing in the hidden position, and selects the action label according to this probability; then, the least likely label is hidden again, and the remaining labels are used for subsequent iterations; this updated label sequence is then used to predict the sequence for the next iteration, and this process continues until the Nth iteration is completed.

[0072] The optimization of the base transformer FT involves minimizing the negative log-likelihood function related to the prediction target, which is as follows:

[0073]

[0074] where, to predict the probability of the label given the music feature v and the partially masked sequence .

[0075] After the base layer labels are completely generated, the system will systematically predict the label sequence of the remaining quantized layers; in the final stage, all labels are decoded and restored to the action sequence by the decoder of the RAQ-VAE.

[0076] The subsequent transformer ST is used to represent the dance action label of the residual layer , where the quantized layer ; in the training stage, a quantized layer is randomly selected for learning, and the labels of all previous layers are subjected to an embedding process and then aggregated to form a composite label embedding input; this label embedding is jointly used as a multi-aspect input of the ST with the music feature v and the residual quantized layer indicator , and then the ST is trained to simultaneously predict the label of the Lth layer . The training process can be represented as:

[0077]

[0078] where represents the number of latent vectors, to predict the probability of the label given the music feature v and the partially masked sequence and the residual layer label L.

[0079] The mask transformer can simultaneously and in parallel decode multiple dance action features, and comprehensively consider the information of the previous and subsequent action features, automatically fill in the dance gaps by inserting mask features in the generated dance segments, and ensure the fluency and consistency of the generated actions and existing elements.

[0080] In order to detect the matching degree of the dance action generated by the method and the music, the result of the mask transformer is predicted by the classifier-free guidance CFG, and the probability logits of the final output dance label is calculated as follows:

[0081]

[0082] where, and are conditional and unconditional logits, respectively, is the guidance ratio.

[0083] As Figure 4As shown, another embodiment of the present application describes the generation results of the MORE framework under different conditions. Among them, figure (a) is randomly generated, only music is provided, no additional restrictions are provided, and the generated dance action sequence can be generated; figure (b) is a subsequent frame generation, music and a previous dance action sequence are provided, and subsequent dance actions need to be generated; figure (c) is a front frame generation, a dance action of a rear frame is provided, and a dance action of a front frame needs to be generated; and figure (d) is a middle frame generation, a dance of a front and a rear is provided, and a dance action of a middle frame needs to be generated.

[0084] By Figure 4 As can be seen, the method of the embodiment can provide relatively coherent dance actions according to different needs of the user, so that the dance generation method is very flexible to apply; according to the different provided music and dance actions, a complete set of dance actions that meet the song and dance mood can be generated accordingly, so the method has strong generalization ability and can adapt to diversified music styles and rhythms.

[0085] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A dance generation method based on residual perceptual quantization and residual mask modeling, the method generates dance based on the MORE framework, characterized in that, The method comprises the following steps: S1: introducing a mask modeling method, generating dance actions through a mask modeling mechanism; S2: based on a residual-aware quantization RAQ module of the MORE framework, encoding the dance actions into multi-layer discrete motion features, capturing the incremental details and complexity of the actions layer by layer, obtaining the detailed information of the actions, using vector quantization to generate basic motion features in the initial layer, and calculating and storing residual features in the subsequent layers to enhance the complexity and details of the actions layer by layer; S3: the MORE framework uses a residual mask modeling RBMM module to predict and generate the sequence of dance actions layer by layer under the condition of music by using a bidirectional transformer model, and generates dance actions matching the music through the residual mask modeling module RBMM; The MORE framework comprises a residual-aware quantization RAQ module and a residual-based mask modeling RBMM module; The RAQ module labels the dance actions into a multi-layer discrete feature sequence, retaining the semantic information in the dance actions; and the RBMM module is used to predict the randomly masked action features; S201: dividing the dance actions into a dance action sequence d through a residual-aware quantization RAQ module; S202: encode the dance motion sequence d through a one-dimensional convolutional encoder transforming into latent vectors ; S203: project the latent vector quantization , and the quantized code sequence is projected back to the dance motion space, and a one-dimensional decoder is used reconstruct the dance motion ; The step S3 comprises: S301: label the residual dance motion latent sequence in the RAQ module as sequence; S302: pre-calculating the features of the music; S303: mapping the dance action sequence with music feature embeddings and provided as input to a multi-layer bidirectional masked transformer architecture; S304: generating dance action labels of the basic layer and the residual layer through the mask transformer in the residual mask modeling RBMM; S305: decoding and restoring the dance action labels into an action sequence through the decoder of the RAQ-VAE, thereby generating dance actions matching the music.

2. The dance generation method based on residual perceptual quantization and residual mask modeling according to claim 1, characterized in that, The quantization process in S203 is: dancing action latent vectors by vector quantization variational autoencoder VQ-VAE is represented as an ordered sequence of code latents, forming quantization layers; The residual quantization RQ module recursively approximates the residual by computing a code sequence at each quantization level as follows:​ wherein the above process starts from the 0th residual ; After passing through the residual quantization RQ module, the final representation of the latent sequence is the sum of all quantized sequences, and the final output latent sequence is input to the decoder where it is subsequently used to reconstruct the dance motion.

3. The dance generation method based on residual perceptual quantization and residual mask modeling according to claim 2, characterized in that, The training of the vector quantization variational autoencoder VQ-VAE is performed through a dance action reconstruction loss, and the loss function is as follows: wherein, denotes a stop-gradient operation, hyperparameters is a weighting factor for embedding constraints, is an identification number, is a code sequence, is a residual, d is a dance motion sequence, is a decoded reconstructed dance motion sequence, a loss function is optimized by a straight-through gradient estimator.

4. The dance generation method based on residual perceptual quantization and residual mask modeling according to claim 1, characterized in that, The multi-layer bidirectional mask transformer architecture is used to implement a random masking process, in which a specified proportion of sequence elements is masked by replacing the original label with a specified [MASK] label, and subsequently, this [MASK] label serves as a placeholder indicating input distortion, prompting the model to infer and reconstruct the original masked true label.

5. The dance generation method based on residual perceptual quantization and residual mask modeling according to claim 1, characterized in that, The mask transformer in the residual mask modeling RBMM comprises a basic transformer FT and a subsequent transformer ST; The base transformer FT is used to represent the dance motion markers of the base layer When The resulting sequence is represented as The base transformer FT predicts the masked markers by the music features v and the partially masked sequence The optimization of the base transformer FT involves minimizing a negative log-likelihood function related to the prediction target, formulated as follows: wherein, is the probability of predicting the label given the music feature v and the partially masked sequence of features. Subsequent transformer ST is used to represent dance motion labels of residual layers where the quantized layers ; at the training phase, one quantized layer is randomly selected to learn, all previous layers' labels are passed through an embedding process, then aggregated to form a composite label embedding input; this label embedding is jointly with music features v and residual quantized layer indicator as multi-faceted input to ST, which is then trained to simultaneously predict the label of the Lth layer , the training process can be represented as: wherein denotes the number of latent vectors, is the probability of predicting the label given the music feature v and the sequence of partial masks and the residual layer label L.

6. The dance generation method based on residual perceptual quantization and residual mask modeling according to claim 5, characterized in that, The mask transformer can simultaneously and in parallel decode multiple dance action features, and comprehensively consider the information of the front and rear action features, automatically fill in the dance gaps by inserting mask features in the generated dance segments.

7. The dance generation method based on residual perceptual quantization and residual mask modeling according to claim 6, characterized in that, The results of the no-classifier guided CFG prediction mask transformer are passed through a final output dance token probability logits The formula is: wherein, and are conditional and unconditional logits, respectively, is a steering proportion.

Images