Multifunctional dance generation method based on residual perception quantization and residual mask modeling

Abstract

The invention discloses a multifunctional dance generation method based on residual perception quantization and residual mask modeling, and belongs to the technical field of artificial intelligence, dance is generated based on an MORE framework, and the method comprises the following steps: introducing a mask modeling method, and generating a dance action through a mask modeling mechanism; a residual error perception quantization RAQ module based on an MORE framework codes dance motions into multilayer discrete motion features, captures increment details and complexity of the motions layer by layer, and obtains detail information of the motions; the MORE framework models an RBMM model through a residual mask, predicts and generates a sequence of dance actions layer by layer under a music condition by using a bidirectional transformer model, and generates dance actions matched with music from sequence information of the actions through the RBMM model. According to the method, through the combination of the hierarchical mask strategy and the music features, the hierarchy and fineness of the generation process are improved, and the matching degree of the generated dance movement and the music features is optimized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multifunctional dance generation method based on residual perceptual quantization and residual mask modeling. Background Art

[0002] Dance is a performing art that uses the body to perform various graceful or difficult movements, usually accompanied by music, and uses rhythmic movements as the main means of expression. Choreography is a very important part of dance creation, which is to design the dance according to the given music melody, and convey the dancer's thoughts and expressions through body movements and details. Traditional choreography usually relies on the experience and creativity of professional choreographers, has a certain degree of subjectivity, and is a complex and labor-intensive task.

[0003] With the development of science and technology, the choreography process can be combined with computer vision and artificial intelligence technology, and the corresponding dance sequence can be matched with the computer model and music, so as to generate a complete set of dance movements according to the music. In the early days, the retrieval method was usually used to generate dance, which decomposed the dance movements into several fixed-length segments, and matched and spliced ​​them according to the structure of the music to generate a complete dance sequence. With the development of deep learning technology, the direct generation method has gradually become the mainstream, generating dance movements from scratch by training models. Diffusion models have also been widely used in recent years. With their step-by-step denoising generation process, they can generate high-quality and coherent dance movement sequences.

[0004] Although existing dance generation technology has made some progress in generating dance movements that match music, there are still some defects: many retrieval-based dance generation methods rely on pre-defined dance clips and cannot flexibly cope with the diversity and complexity of different music, resulting in the lack of sufficient generalization ability of the generated dance movements when facing diverse music styles and rhythms, and it is difficult to capture the subtle changes of music; while the direct generation method can generate high-quality dance movements, it still faces challenges in ensuring the consistency and fluency of movements over a long period of time, especially when dealing with more complex music. The generated dance movements are prone to incoordination and breakage, which affects the naturalness and viewing of the overall dance. When learning and representing high-quality features of dance movements, current methods face the difficulty of how to effectively capture the overall picture of the movements while maintaining the details of the dance. Traditional quantization methods may cause the loss or blurring of dance movement features and fail to fully reflect the rich expression of dance movements. Summary of the invention

[0005] The purpose of the present invention is to provide a multifunctional dance generation method based on residual perceptual quantization and residual mask modeling to address the above-mentioned problems, in order to improve the problems of insufficient flexibility and generalization ability and poor timing consistency of dance generated by existing dance generation methods.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A multifunctional dance generation method based on residual-aware quantization and residual mask modeling, the method generates dance based on the MORE framework, and the method comprises the following steps: S1: introducing a mask modeling method, and generating dance movements through a mask modeling mechanism; S2: based on the residual-aware quantization RAQ module of the MORE framework, encoding dance movements into multi-layer discrete motion features, capturing incremental details and complexity of the movements layer by layer, and obtaining detailed information of the movements, the initial layer uses vector quantization to generate basic motion features, and the subsequent layers calculate and store residual features to enhance the complexity and details of the movements layer by layer; S3: the MORE framework uses a residual mask modeling RBMM model, and uses a bidirectional transformer model to predict and generate a sequence of dance movements layer by layer under music conditions, and the sequence information of the movements is passed through the residual mask modeling module RBMM to generate dance movements matching the music.

[0008] Furthermore, the MORE framework includes a residual-aware quantization (RAQ) module and a residual-based mask modeling (RBMM) module; wherein the RAQ module tokenizes dance movements into multi-layer discrete feature sequences to retain semantic information in the dance movements; and the RBMM module is used to predict randomly masked movement features.

[0009] Further, the step S2 includes: S201: dividing the dance action into a dance action sequence d through a residual perceptual quantization RAQ module; S202: converting the dance action sequence d into a latent vector through a one-dimensional convolutional encoder ε S203: The latent vector The quantized Q(·) is then projected back into the dance motion space and decoded using a one-dimensional decoder Reconstructing dance moves

[0010] Furthermore, the quantization process in S203 is: the dance action potential vector is converted into It is represented as L+1 ordered code potential sequences, forming L+1 quantization layers; the residual quantization RQ module calculates the code sequence at each quantization layer To recursively approximate the residual The movement details are effectively preserved in the multi-layer quantization process, which significantly improves the accuracy and authenticity of dance movement generation. The formula is as follows:

[0011] z l =Q(r l ), r l+1 =r l -z l .

[0012] The above process starts from the 0th residual start;

[0013] After passing through the residual quantization RQ module, the final representation of the latent sequence is the sum of all quantized sequences, and the final output latent sequence is input to the decoder , which was then used to reconstruct the dance movements.

[0014] Furthermore, the training of the vector quantized variational autoencoder VQ-VAE is performed by dance motion reconstruction loss, and its loss function is as follows:

[0015]

[0016] Where sg[·] indicates stopping the gradient operation, the hyperparameter β is the weighting factor of the embedded constraint, i is an identification number, is the code sequence, is the residual, d is the dance action sequence, To decode the reconstructed dance motion sequence, the loss function is optimized through a straight-through gradient estimator.

[0017] Furthermore, the step S3 includes:

[0018] S301: Label the residual dance action potential sequence in the RAQ module as {t 0 , t 1 ,…,t L}sequence;

[0019] S302: Pre-calculate the characteristics of music;

[0020] S303: The dance action {t 0 , t 1 ,…,t L The sequence is fused with the music feature embedding and fed as input to a multi-layer bidirectional masked transformer architecture;

[0021] S304: Generate dance motion markers of the base layer and the residual layer respectively through the mask transformer in the residual mask modeling RBMM;

[0022] It is important to note that the use of these transformers allows for the fine-grained generation of dance movement markers that follow the structural hierarchy of the quantization process.

[0023] S305: The dance action tags are decoded by the RAQ-VAE decoder and restored to action sequences, thereby generating dance actions that match the music.

[0024] Furthermore, the multi-layer bidirectional mask transformer architecture is used to implement a random masking process, in which a specified proportion of sequence elements are masked by replacing the original tags with a specified [MASK] tag, which then acts as a placeholder indicating the input distortion, prompting the model to infer and reconstruct the true tags that were originally masked.

[0025] It should be noted that the music features were obtained following the standard protocol.

[0026] Further, the mask transformer in the mask modeling RBMM includes: a basic transformer FT and a subsequent transformer ST;

[0027] The basis transformer FT is used to represent the dance movement tags t of the base layer 0 , when t 0 When the subset of is masked, the resulting sequence is represented as The basis transformer FT passes through the music feature v and the partially masked sequence Predict the masked markers; the optimization of the basis transformer FT involves minimizing the negative log-likelihood function associated with the predicted target, as follows:

[0028]

[0029] in, is the sequence of given music features v and partially masked Prediction mark probability;

[0030] The subsequent transformer ST is used to represent the dance action label t of the residual layer l , where the quantization layer l∈[1,L]; in the training phase, a quantization layer l is randomly selected for learning, and the tags of all previous layers are embedded and then aggregated to form a composite tag embedding input; this tag embedding is used together with the music feature v and the residual quantization layer indicator l as the multi-faceted input of ST, and then ST is trained to simultaneously predict the tag of the L-th layer l-th. The training process can be expressed as:

[0031]

[0032] Where K represents the number of latent vectors, is the sequence of given music features v and partially masked And the predicted label in the case of residual layer label L probability.

[0033] Furthermore, the mask transformer can decode multiple dance movement features in parallel and simultaneously, and comprehensively consider the information of the previous and next movement features, and automatically fill in the dance gaps by inserting mask features into the generated dance clips.

[0034] Furthermore, the result of the mask transformer is predicted by guiding the CFG without a classifier, and the final output dance tag probability logits∈ g The calculation formula is:

[0035] ∈ g =(1+η)·∈ c -η·∈ u ,

[0036] Among them, ∈ c and ∈ u are the conditional and unconditional logits, respectively, and η is the bootstrap proportion.

[0037] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0038] The present invention introduces generative mask modeling and generates dance movements through mask modeling mechanism, so that the generated dance can remain smooth and natural under various music conditions; based on the residual perceptual quantization module and residual mask modeling module of the MORE model, the dance movement sequence is hierarchically encoded through a recursive residual quantization strategy, and the dance movements of the base layer and the residual layer are predicted using a bidirectional transformer network. This method improves the hierarchy and meticulousness of the generation process through the combination of hierarchical masking strategies and music features, and optimizes the matching degree between the generated dance movements and music features. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the overall flow chart of the method of the present invention;

[0040] Figure 2 It is a partial flow chart of the method of the present invention;

[0041] Figure 3 It is the overall framework diagram of the present invention;

[0042] Figure 4 This is a schematic diagram of the generation results of the present invention based on the MORE framework under different conditions. DETAILED DESCRIPTION

[0043] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] Because existing dance generation methods lack sufficient generalization ability when faced with diverse music styles and rhythms, and find it difficult to capture the subtle changes in music, the generated dance movements do not match the music well and the continuity of the movements is poor.

[0046] like Figure 1-3 As shown, an embodiment of the present invention proposes a multifunctional dance generation method based on residual perceptual quantization and residual mask modeling, and improves the flexibility and consistency of dance movement generation through the proposed dance generation framework MORE.

[0047] The method comprises the following steps: S1: introducing a mask modeling method, and generating dance movements through a mask modeling mechanism; achieving full-function generation of dance, and the generated dance can remain smooth and natural under various music conditions; S2: based on the residual-aware quantization (RAQ) module of the MORE framework, encoding dance movements into multi-layer discrete motion features, capturing incremental details and complexity of the movements layer by layer, obtaining detailed information of the movements, and ensuring that the rich semantic information inherent in the movements is retained; the initial layer uses vector quantization to generate basic motion features, and the subsequent layers calculate and store residual features to enhance the complexity and details of the movements layer by layer; S3: the MORE framework uses a residual mask modeling (RBMM) model, and uses a bidirectional transformer model to predict and generate a sequence of dance movements layer by layer under music conditions, and generates dance movements matching the music through the residual mask modeling module (RBMM) using the sequence information of the movements.

[0048] The MORE framework includes a residual-aware quantization (RAQ) module and a residual-based mask modeling (RBMM) module. The RAQ module tokenizes dance movements into multi-layer discrete feature sequences to retain semantic information in the dance movements. The RBMM module is used to predict randomly masked movement features.

[0049] Since the quantization process inevitably leads to a reduction in information, a residual-aware quantization module is introduced. Step S2 includes:

[0050] S201: Divide the dance movements into dance movement sequences d through the residual perceptual quantization RAQ module;

[0051] S202: Convert the dance action sequence d into a latent vector through a one-dimensional convolutional encoder ε

[0052] S203: The latent vector The quantized Q(·) is then projected back into the dance motion space and decoded using a one-dimensional decoder Reconstructing dance moves

[0053] The quantization process is:

[0054] The dance action latent vector is transformed into Represented as L+1 ordered code potential sequences, forming L+1 quantization layers;

[0055] The residual quantization RQ module calculates the code sequence at each quantization layer To recursively approximate the residual The formula is as follows:

[0056] z l =Q(r l ), r l+1 =r l -z l .

[0057] The above process starts from the 0th residual start;

[0058] After passing through the residual quantization RQ module, the final representation of the latent sequence is the sum of all quantized sequences, and the final output latent sequence is input to the decoder , which was then used to reconstruct the dance movements.

[0059] The training of the vector quantized variational autoencoder VQ-VAE is performed through dance motion reconstruction loss, and its loss function is as follows:

[0060]

[0061] Where sg[·] indicates stopping the gradient operation, the hyperparameter β is the weighting factor of the embedded constraint, i is an identification number, is the code sequence, is the residual, d is the dance action sequence, To decode the reconstructed dance motion sequence, the loss function is optimized through a straight-through gradient estimator.

[0062] The step S3 comprises:

[0063] S301: Label the residual dance action potential sequence in the RAQ module as {t 0 , t 1 ,…,t L}sequence;

[0064] S302: Pre-calculate the characteristics of music;

[0065] S303: The dance action {t 0 , t 1 ,…,t L The sequence is fused with the music feature embedding and fed as input to a multi-layer bidirectional masked transformer architecture;

[0066] In the RAQ module, the residual dance action label {t 0 , t 1 ,…,t L}First, the output of the encoder is passed to a residual vector quantizer. Subsequently, the dance movement token sequence is combined with the music feature embedding and provided as input to a multi-layer bidirectional mask transformer architecture, which is trained to predict randomly masked tokens. The multi-layer bidirectional mask transformer architecture is used to implement a random masking process, in which a specified proportion of sequence elements are masked by replacing the original token with a specified [MASK] token. Subsequently, this [MASK] token acts as a placeholder indicating the input distortion, prompting the model to infer and reconstruct the true token that was originally masked.

[0067] S304: Generate dance motion markers of the base layer and the residual layer respectively through the mask transformer in the residual mask modeling RBMM;

[0068] S305: The dance action tags are decoded by the RAQ-VAE decoder and restored to action sequences, thereby generating dance actions that match the music.

[0069] In order to further realize hierarchical residual quantization, the mask transformer in the mask modeling RBMM adopts two different mask transformers: a basic transformer FT and a subsequent transformer ST, which are used to generate dance motion markers of the base layer and the residual layer respectively;

[0070] The basis transformer FT is used to represent the dance movement tags t of the base layer 0 , when t 0 When the subset of is masked, the resulting sequence is represented as The basis transformer FT passes through the music feature v and the partially masked sequence Predict the masked tokens; the main goal of the base transformer FT is to predict the masked tokens by leveraging the musical features v and the partially masked sequences Accurately predict masked markers; the specific process is:

[0071] To synthesize the base layer tag sequence in N iterations, we first start with a blank sequence where all tags are hidden. In each iteration, starting from the hidden tag sequence, the base transformer FT calculates the probability of various tags appearing in the hidden position and selects the action tag based on this probability; then, the least likely tag is hidden again, and the remaining tags are used for subsequent iterations; this updated tag sequence is then used to predict the sequence for the next iteration, and this process continues until the Nth iteration is completed.

[0072] The optimization of the basis transformer FT involves minimizing the negative log-likelihood function associated with the predicted target, as follows:

[0073]

[0074] in, is the sequence of given music features v and partially masked Prediction mark probability;

[0075] After the base layer markers are fully generated, the system methodically predicts the marker sequences of the remaining quantized layers; in the final stage, all markers are decoded by the RAQ-VAE decoder and restored to action sequences.

[0076] The subsequent transformer ST is used to represent the dance action label t of the residual layer l , where the quantization layer l∈[1,L]; in the training phase, a quantization layer l is randomly selected for learning, and the tags of all previous layers are embedded and then aggregated to form a composite tag embedding input; this tag embedding is used together with the music feature v and the residual quantization layer indicator l as the multi-faceted input of ST, and then ST is trained to simultaneously predict the tag of the L-th layer l-th. The training process can be expressed as:

[0077]

[0078] Where K represents the number of latent vectors, is the sequence of given music features v and partially masked And the predicted label in the case of residual layer label L probability.

[0079] The mask transformer can decode multiple dance movement features in parallel and simultaneously, and comprehensively consider the information of the previous and next movement features. By inserting mask features into the generated dance clips, it can automatically fill in the dance gaps, thereby ensuring the fluency and consistency of the generated movements and existing elements.

[0080] In order to detect the matching degree between the dance movements generated by this method and the music, the result of the mask transformer is predicted by guiding CFG without a classifier, and the final output dance tag probability logits∈ g The calculation formula is:

[0081] ∈ g =(1+η)·∈ c -η·∈ u ,

[0082] Among them, ∈ c and ∈ u are the conditional and unconditional logits, respectively, and η is the bootstrap proportion.

[0083] like Figure 4 As shown, another embodiment of the present invention describes the generation results of the MORE framework under different conditions. Among them, Figure (a) is random generation, only music is provided, and no restrictions are added to generate a dance action sequence; Figure (b) is subsequent frame generation, music and the previous dance action sequence are provided, and subsequent dance actions need to be generated; Figure (c) is the front frame generation, the dance action of the latter frame is provided, and the dance action of the front frame needs to be generated; Figure (d) is the middle frame generation, the dance of the front and back two sections is provided, and the dance action of the middle frame needs to be generated.

[0084] Depend on Figure 4 It can be seen that the method of this embodiment can provide relatively coherent dance movements according to the different needs of users, making the application of the dance generation method very flexible; according to the different music and dance movements provided, a whole set of dance movements that conform to the mood of the song and dance can be generated accordingly. Therefore, this method has a strong generalization ability and can adapt to a variety of music styles and rhythms.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multifunctional dance generation method based on residual perceptual quantization and residual mask modeling, which generates dance based on the MORE framework, characterized in that: The method comprises the following steps: S1: Introducing the mask modeling method to generate dance movements through the mask modeling mechanism; S2: The residual-aware quantization (RAQ) module based on the MORE framework encodes dance movements into multiple layers of discrete motion features, capturing incremental details and complexity of the movements layer by layer, and obtaining detailed information of the movements. The initial layer uses vector quantization to generate basic motion features, and the subsequent layers calculate and store residual features to enhance the complexity and details of the movements layer by layer. S3: The MORE framework uses the residual mask modeling RBMM model, and uses the bidirectional transformer model to predict and generate a sequence of dance movements layer by layer under the condition of music. The sequence information of the movements is passed through the residual mask modeling module RBMM to generate dance movements that match the music.

2. The multifunctional dance generation method based on residual perceptual quantization and residual mask modeling according to claim 1, characterized in that: The MORE framework includes a residual-aware quantization (RAQ) module and a residual-based mask modeling (RBMM) module; Among them, the RAQ module tokenizes dance movements into multi-layer discrete feature sequences to retain the semantic information in the dance movements; the RBMM module is used to predict randomly masked movement features.

3. The multifunctional dance generation method based on residual perceptual quantization and residual mask modeling according to claim 1, characterized in that: The step S2 comprises: S201: Divide the dance movements into dance movement sequences d through the residual perceptual quantization RAQ module; S202: Convert the dance action sequence d into a latent vector through a one-dimensional convolutional encoder ε S203: The latent vector The quantized Q(·) is then projected back into the dance motion space and decoded using a one-dimensional decoder Reconstructing dance moves 4. The multifunctional dance generation method based on residual perceptual quantization and residual mask modeling according to claim 3 is characterized in that: The quantization process in S203 is: The dance action latent vector is transformed into Represented as L+1 ordered code potential sequences, forming L+1 quantization layers; The residual quantization RQ module calculates the code sequence at each quantization layer To recursively approximate the residual The formula is as follows: z l =Q(r l ),r l+1 =r l -z l . The above process starts from the 0th residual start; After passing through the residual quantization RQ module, the final representation of the latent sequence is the sum of all quantized sequences, and the final output latent sequence is input to the decoder , which was then used to reconstruct the dance movements.

5. The multifunctional dance generation method based on residual perceptual quantization and residual mask modeling according to claim 4, characterized in that: The training of the vector quantized variational autoencoder VQ-VAE is performed through dance motion reconstruction loss, and its loss function is as follows: Where sg[·] indicates stopping the gradient operation, the hyperparameter β is the weighting factor of the embedded constraint, i is an identification number, is the code sequence, is the residual, d is the dance action sequence, To decode the reconstructed dance motion sequence, the loss function is optimized through a straight-through gradient estimator.

6. A multifunctional dance generation method based on residual perceptual quantization and residual mask modeling according to any one of claims 1 to 5, characterized in that: The step S3 comprises: S301: Label the residual dance action potential sequence in the RAQ module as {t 0 , t 1 ,…,t L }sequence; S302: Pre-calculate the characteristics of music; S303: The dance action {t 0 , t 1 ,…,t L The sequence is fused with the music feature embedding and fed as input to a multi-layer bidirectional masked transformer architecture; S304: Generate dance motion markers of the base layer and the residual layer respectively through the mask transformer in the residual mask modeling RBMM; S305: The dance action tags are decoded by the RAQ-VAE decoder and restored to action sequences, thereby generating dance actions that match the music.

7. The multifunctional dance generation method based on residual perceptual quantization and residual mask modeling according to claim 6, characterized in that: The multi-layer bidirectional mask transformer architecture is used to implement a random masking process, in which a specified proportion of sequence elements are masked by replacing the original token with a specified [MASK] token, which then acts as a placeholder indicating the input distortion, prompting the model to infer and reconstruct the true token that was originally masked.

8. The multifunctional dance generation method based on residual perceptual quantization and residual mask modeling according to claim 6, characterized in that: The mask transformer in the mask modeling RBMM includes: a basic transformer FT and a subsequent transformer ST; The basis transformer FT is used to represent the dance movement tags t of the base layer 0 , when t 0 When the subset of is masked, the resulting sequence is represented as The basis transformer FT passes through the music feature v and the partially masked sequence Predict the masked markers; the optimization of the basis transformer FT involves minimizing the negative log-likelihood function associated with the predicted target, as follows: in, is the sequence of given music features v and partially masked Prediction mark probability; The subsequent transformer ST is used to represent the dance action label t of the residual layer l , where the quantization layer l∈[1,L]; in the training phase, a quantization layer l is randomly selected for learning, and the tags of all previous layers are embedded and then aggregated to form a composite tag embedding input; this tag embedding is used together with the music feature v and the residual quantization layer indicator l as the multi-faceted input of ST. Then, ST is trained to simultaneously predict the tag of the L-th layer. The training process can be expressed as: Where K represents the number of latent vectors, is the sequence of given music features v and partially masked And the predicted label in the case of residual layer label L probability.

9. The multifunctional dance generation method based on residual perceptual quantization and residual mask modeling according to claim 8, characterized in that: The mask transformer can decode multiple dance movement features in parallel and simultaneously, and comprehensively consider the information of the previous and next movement features, and automatically fill in the dance gaps by inserting mask features into the generated dance clips.

10. The multifunctional dance generation method based on residual perceptual quantization and residual mask modeling according to claim 8, characterized in that: The result of the mask transformer is predicted by guiding CFG without a classifier, and the final output dance label probability logits∈ g The calculation formula is: ∈ g =(1+η)·∈ c -η·∈ u , Among them, ∈ c and ∈ u are the conditional and unconditional logits, respectively, and η is the bootstrap proportion.

Images