Music-driven dance generation method and device based on rhythm perception feature representation of gating enhancement

Abstract

The invention discloses a music-driven dance generation method and equipment based on gating enhanced rhythm perception feature representation, and belongs to the technical field of music-driven dance generation. In order to solve the problem that dance generated by an existing generation method is poor in expressive force and rhythm coherence, the method comprises the following steps of: encoding motion of an upper body and motion of a lower body by using VQ-VAE to obtain an upper body code and a lower body code; performing channel dimension alignment on the music features, the upper body codes and the lower body codes by using independent linear layers to obtain respective corresponding embedded features, splicing the embedded features, obtaining rhythm enhancement features in combination with phase features representing music rhythms, and processing the rhythm enhancement features by using a time gating causal attention module; and generating respective probabilities of each action in an upper body code book and a lower body code book in a current prediction time frame by combining a parallel Mama motion modeling module, respectively selecting feature codes of the upper body action and the lower body action with the maximum probabilities in the code books, and decoding and outputting the actions by utilizing a VQ-VAE decoder.

Description

Technical Field

[0001] This invention belongs to the field of music-driven dance generation technology, specifically a music-driven dance generation method and device. Background Technology

[0002] Music-driven dance generation technology aims to generate dance movements that are highly coordinated with the rhythm and style of input music, achieving synchronization and consistency between dance postures and music. This technology can automatically synthesize natural, realistic, and expressive 3D dance movements, thereby driving 3D character models to perform animated routines according to music. It has wide applications in virtual humans, games, virtual reality, and film production. With the increasing demands for human-computer interaction, digital entertainment, and content creation, music-driven dance generation, as a cutting-edge interdisciplinary technology integrating audio understanding and motion modeling, has become a research hotspot in both academia and industry.

[0003] In recent years, generative models have demonstrated significant success across various tasks, and these generative methods have also been applied to music-driven dance generation. Many generative techniques have emerged, such as the Bailando method, which introduces a cross-conditional causal attention mechanism to align music features with corresponding dance movements, significantly improving the quality of generated dance movements. The Enhancing-Bailando method employs a pre-trained audio encoder (MERT) to obtain richer music feature representations. The Beat-It method explicitly introduces beat synchronization through a beat distance predictor.

[0004] Currently, cross-conditional causal attention primarily captures the local correspondence between music and movement, failing to fully utilize the global rhythmic characteristics of the music, thus limiting the expressiveness and rhythmic coherence of generated dances. Explicit beat synchronization introduces beat synchronization through a beat predictor, solving the rhythm alignment problem, but imposes strict control constraints, greatly limiting the expressiveness and diversity of generated dances.

[0005] However, current music-driven dance generation methods have shortcomings in long sequence motion modeling, accurate synchronization of rhythm and movement, and generation of natural and diverse dances. Furthermore, the generated dance movements suffer from drifting and sliding issues. Summary of the Invention

[0006] The invention aims to address the problem that existing methods for generating dances suffer from poor expressiveness and rhythmic coherence.

[0007] A music-driven dance generation method based on gated enhanced rhythm-aware feature representation includes the following steps: Step S1: Using two pre-trained motion VQ-VAEs, encode the upper body and lower body motions respectively to obtain the upper body code. and lower body coding Phase features representing musical rhythm are obtained based on musical characteristics. Three independent linear layers were used to encode musical features and upper body characteristics, respectively. and lower body coding Channel dimension alignment is performed to obtain music embedding features. Upper body movement embedded features Lower body movement embedding features And splice together the features ;based on and Obtain rhythm enhancement features ; Step S2, rhythm enhancement feature The data is fed into the Time-Gated Causal Attention (TGCA) module for processing. TGCA utilizes cross-causal attention and gating units to obtain global rhythmic features. : Step S3 The data is fed into the parallel Mamba motion modeling module, i.e., the PMMM module, for processing. Re-separate along the time dimension into global rhythm enhancement upper body motion embedding features Global rhythm enhancement and lower body motion embedding features and global rhythm enhancement music embedding features The data is fed into three parallel Mamba blocks for processing to obtain upper and lower body motion features. , With musical characteristics ;Will , and Features are obtained by piecing them together ; feature The code is then fed into the TGCA module for processing, and the upper and lower body codebooks are generated based on the processing results of the TGCA module. The probability of each action in the current prediction time frame ; based on , respectively in The feature codes of the upper and lower body movements with the highest probability are selected as the predicted action coding sequence. , ; Encoding sequence , The action is output after decoding by the VQ-VAE decoder.

[0008] Furthermore, the musical features are obtained by processing the original music using the librosa library.

[0009] Furthermore, phase features representing musical rhythm are obtained based on musical characteristics. The process includes the following steps: Short-time Fourier transform is used to convert musical features into a time-frequency graph, and the phase angle is extracted. Furthermore, a center-clipping strategy is applied to selectively focus on the phase region with the highest information content to obtain phase features. To obtain the final phase features representing musical rhythm, a linear layer, a batch normalization layer, and an activation function are used to process the phase features. .

[0010] Furthermore, the application-centric pruning strategy selectively focuses on the phase regions with the highest information content, and the process of obtaining phase features includes: Regarding the phase angle ,exist Cut out a length of [length] from the middle of the direction. phase angle That is, phase characteristics; This represents the number of time frames after STFT conversion. for The channel dimension, Indicates the length of time. This represents the time downsampling rate.

[0011] Furthermore, based on and Obtain rhythm enhancement features The process includes: Take 3 identical ones Features are obtained by piecing them together and features Fusion: , Enhance the rhythmic features of the output.

[0012] Furthermore, the processing steps of the Temporally Gated Causal Attention Module (TGCA) include: First, it is processed through an RMSnorm normalization layer to obtain... ; Then, a time gating mechanism is applied: Then, based on cross-conditional causal attention Obtain features ; This indicates element-wise multiplication; Finally, global rhythm features are obtained based on RMSnorm and multilayer perceptron. .

[0013] Furthermore, in the Mamba block, the input first passes through an RMSnorm normalization layer and then through a Mamba layer, and after residual connections, intermediate features are obtained; then the intermediate features pass through an RMSnorm normalization layer and then through a GateMlp layer, and finally after residual connections, output features are obtained.

[0014] Furthermore, upper and lower body codebooks are generated based on the processing results of the TGCA module. The probability of each action in the current prediction time frame The process includes: This will be achieved through the continuous autoregressive Transformer layer in TGCA. The result is denoted as Then, the upper and lower body codebooks are generated through a linear layer and a softmax layer. The probability of each action in the current prediction time frame ,Right now:

[0015] in, , This indicates the codebook The probability of each upper body movement in the current prediction time frame. This indicates the codebook The probability of each lower body movement in the current prediction time frame.

[0016] Furthermore, the model comprising the VQ-VAE encoder, the TGCA module, units containing the PMMM module, the TGCA module, and Linear and softmax layers, and the VQ-VAE decoder is denoted as MotionGPT. The loss function used by MotionGPT during training includes the motion probabilities for both the upper and lower body. Cross-entropy loss ;

[0017] in, The label value corresponding to the known action; The probability of upper body and lower body movements; Indicates the duration of time.

[0018] A music-driven dance generation device based on gated augmented rhythm-aware feature representation, the device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the aforementioned music-driven dance generation method based on gated augmented rhythm-aware feature representation.

[0019] The beneficial effects of this invention are: 1. By explicitly separating the rhythmic features and semantic features of music through phase-based rhythm extraction (PRE), the rhythmic sensitivity in the dance generation process is enhanced. The rhythmic vector and semantic vector output by PRE are fed into the temporally gated causal attention (TGCA) module. The gating mechanism is used to highlight the global rhythmic structure in cross-conditional causal attention, thereby reducing the contextual relationship between music rhythm and dance movements. At the spatiotemporal modeling level, the upper body and lower body sequences are modeled separately using the parallel Mamba motion modeling module to capture more granular limb dynamics, enhancing the expressiveness and coherence of dance movements, and ensuring that the dance postures are highly synchronized with the music rhythm and are expressive.

[0020] 2. This invention enhances the rhythm-aware feature representation and motion modeling of the music-driven dance generation model, ensuring accurate beat-dance synchronization and generating more rhythmic, natural, and diverse dance movements. Attached Figure Description

[0021] Figure 1 This is a flowchart of the present invention.

[0022] Figure 2 Network structure diagram of a music-driven dance generation method based on gating-enhanced rhythm-aware feature representation.

[0023] Figure 3 This method provides a quantitative comparison of experimental results.

[0024] Figure 4 A visual representation of the results on the AIST++ dataset. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] See appendix Figure 1 This embodiment provides a music-driven dance generation method based on gated enhanced rhythm-aware feature representation, including the following steps: Step S1: Construct two pre-trained motion VQ-VAEs, and encode the motion of the upper body and lower body respectively to obtain upper body code and lower body code; Short-time Fourier Transform (STFT) is used to perform STFT on the input music features to convert the music signal into a time-frequency diagram and extract the phase angle. A center clipping strategy is applied to selectively focus on the phase region with the most information. The output phase features are processed through a linear layer, a batch normalization layer, and a ReLU activation function to obtain the phase features representing the music rhythm. Based on music features, upper body encoding, and lower body encoding, music embedding features, upper body motion embedding features, and lower body motion embedding features are obtained. These features are then concatenated and fused with phase features to obtain rhythm-enhanced features.

[0027] This embodiment uses an embedding feature fusion module to extract phase features, perform center clipping, and enhance rhythm features to obtain music embedding features, which are then fused with upper and lower body movement embedding features. See Appendix. Figure 2 The specific steps are as follows: S101, based on a dance sequence Obtain upper body motion embedding features and lower body motion embedding features Characteristics of primitive music The following steps are required to obtain: See appendix Figure 2 Given a dance sequence ,in It refers to the length of time. It refers to the number of joints; using two pre-trained motion VQ-VAEs, the movements of the upper and lower body are encoded separately. , The two pre-trained actions VQ-VAE are... and Their codebooks are respectively and ;

[0028] in, , , , Indicates the length of time. The time downsampling rate; S102. Perform STFT on the input music features to extract the phase angle:

[0029]

[0030] in, The complex spectrum obtained by STFT. A function to calculate the phase angle of a complex numerical input. From The phase angle extracted from , Represents the field of complex numbers. This represents the number of time frames after STFT conversion. for Channel dimension; For music features, music feature m is the audio feature obtained by processing the original music using the librosa library. It is the channel dimension of musical characteristics. , The time downsampling rate; Because of the non-stationary nature of music and the sliding window method of STFT, sudden rhythmic changes and complex rhythmic patterns can be effectively captured.

[0031] S103. A center-clipping strategy is used to selectively focus on the phase region with the most information:

[0032] in, , The time lengths are equal.

[0033] The center clipping strategy is based on the phase angle. ,exist Cut out a length of [length] from the middle of the direction. phase angle Extracted phase angle Essentially, this reflects the time shift and periodic structure of musical rhythm: low-frequency phase changes correspond to a wider range of rhythmic patterns, while high-frequency changes capture finer rhythmic details. This strategy selectively focuses on the phase regions with the most information.

[0034] S104. By processing the phase features through linear transformation, the phase features representing the musical rhythm are obtained:

[0035] in To represent the phase characteristics of musical rhythm.

[0036] S105. Use three independent linear layers to process the music features respectively. Upper body coding and lower body coding Channel dimension alignment is performed to obtain embedded features: ,

[0037] in, , , These represent music embedding features, upper body motion embedding features, and lower body motion embedding features, respectively. It is the channel dimension of the feature.

[0038] Will , and Features are obtained by piecing them together To further enhance the rhythmic characteristics, three identical [rhythms] were [used]. along Features are obtained by concatenating the dimensions and directions. .

[0039] S106, Phase and Features Fusion: , Enhance the rhythmic features of the output.

[0040] Step S2: Construct a Temporally Gated Causal Attention (TGCA) module to pass the rhythm enhancement features through the TGCA module to obtain global rhythm features. : Rhythm enhancement features First, it is processed through an RMSnorm normalization layer to obtain... ; Normalized Temporal causal attention computation is performed, which includes a temporal gating mechanism. Temporal gating consists of a linear layer and a SiLU activation function. Normalized And the processing results after time gating, through a continuous autoregressive Transformer layer deal with, Including cross-conditional causal attention ( The mechanism consists of a linear layer, an RMSnorm normalization layer, and an MLP layer. The normalized layer... Perform cross-conditional causal attention ( ) calculations to capture causal relationships between different time steps.

[0041] The processed result, after passing through a linear layer, is compared with the original input rhythm enhancement feature. The elements are added together; then the result is passed through an RMSnorm normalization layer and an MLP layer to output the final result.

[0042] Cross-causal attention mechanism (represented as) A key issue exists: the next predicted token lacks a direct control signal from historical tokens, making it impossible to establish stable global beat attention. This misalignment leads to accumulated errors. This invention proposes the TGCA module to address this problem.

[0043] See appendix Figure 2 The global rhythmic feature attention is enhanced by using a Time-Gated Causal Attention (TGCA) module, incorporating historical information to guide the prediction of the next action token. The specific steps are as follows: Normalization results of rhythm enhancement feature X via Temporal deal with:

[0044] in, For linear layer processing, For activation functions; Normalization results of rhythm enhancement feature X Simultaneously through cross-conditional causal attention ( The module outputs... The cross-conditional causal attention is the cross-conditional causal attention in "Bailando: 3D dance generation via Actor-Critic GPT with Choreographic Memory".

[0045] The global rhythm awareness enhancement feature is obtained by multiplying the outputs of the cross-conditional causal attention and the gating unit by element-wise and then passing them through a linear layer. :

[0046] in, This indicates element-wise multiplication.

[0047] Then After RMSNorm mean square normalization and a multilayer perceptron (MLP), the residual connections finally output global rhythm features. :

[0048] Enhanced global rhythm awareness For the modeling of generated dance movements, dance is a complex spatiotemporal sequence that encodes trajectories, movements, and speeds over time, requiring the capture of local transitions and global rhythmic changes. Since the upper and lower body exhibit different movement patterns and speeds, they also need to be processed separately to maintain coherence and diversity.

[0049] Step S3: Construct the PMMM module and TGCA module, and predict the action-encoded sequence based on the output of the TGCA module: Parallel Mamba Motion Modeling (PMMM) module: First, it analyzes global rhythmic features. Process it, Re-separate along the time dimension into global rhythm enhancement upper body motion embedding features Global rhythm enhancement and lower body motion embedding features and global rhythm enhancement music embedding features ; then , , Motion modeling in the state space is performed using three parallel Mamba blocks: upper body, lower body, and music. Features are input into the Mamba block, passed through an RMSnorm normalization layer, then another Mamba layer, and finally through residual connections to obtain intermediate features. These intermediate features are then passed through another RMSnorm normalization layer, a GateMlp layer, and finally through residual connections to obtain the output features. The Mamba block selectively preserves motion features and performs spatiotemporal sequence modeling for the upper and lower body dance sequences and the music sequence, respectively, outputting upper and lower body motion features. , With musical characteristics The processing of the PMMM module can be expressed in the following formula:

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] in, This indicates enhanced upper body movement characteristics of gating. This indicates enhanced lower body movement characteristics due to gating. This represents the gating-enhanced music features; GateMlp is the MLP network in "Pay Attention to MLPs".

[0056] See appendix Figure 2In the PMMM module, GateMlp, introduced after the Mamba block, selectively retains necessary motion features through a gating network while filtering out less relevant information, thereby enhancing the coherence and expressiveness of the generated dance sequence.

[0057] Then , and Features are obtained by piecing them together This serves as the input to another TGCA module, which is set up as an additional configuration. It should be noted that: Both are based on time length Yes, but they have a time step difference, that is... This is the set of music features obtained from time step 1 to t. These are the sets of upper and lower body motion codes obtained from time step 0 to t-1, respectively. TGCA module: The structure is the same as the TGCA module in step 2, for... Processing is performed to further enhance the rhythmic characteristics of the movements.

[0058] To enable the upper and lower body to exhibit different movement patterns and speeds, they are processed separately to maintain coherence and diversity. The parallel Mamba motion modeling architecture models music, upper body and lower body motion separately, while using state space modeling to ensure smooth and diverse motion generation.

[0059] The gating mechanism dynamically adjusts the frequency, speed, and amplitude of the movements in response to the music beat, ensuring that the dance sequence remains synchronized with the music. Three parallel Mamba architectures model the upper and lower body dance sequences respectively, outputting motion and music features. These features are then further enhanced rhythmically by the TGCA module, ensuring that the generated dance movements are expressive, diverse, and temporally coherent.

[0060] This will be achieved through the continuous autoregressive Transformer layer in TGCA. The result is denoted as Then, the upper and lower body codebooks are generated through a linear layer and a softmax layer. The probability of each action in the current prediction time frame ,Right now:

[0061] in, , This indicates the codebook The probability of each upper body movement in the current prediction time frame. , For codebook Size, This indicates the codebook The probability of each lower body movement in the current prediction time frame. , For codebook Size.

[0062] based on , respectively in The feature codes of the upper and lower body movements with the highest probability are selected as the predicted action coding sequence. , Discard musical feature sequences; Encoded sequence , The action is output after decoding by the VQ-VAE decoder.

[0063] See appendix Figure 2 The final dance generation task is to start from the codebook. The most probable future action is selected based on the music and previous actions, and at each time step t, the probability of the action code is estimated using MotionGPT. At each time step t, gated causal attention in TGCA further enhances the action features with rhythmic features, while an internal transformer estimates the action code. ( or The probability of the action code is used to predict the upper body movement by selecting the action code with the highest probability. Predicted lower body movements This invention actually generates and outputs actions through a VQ-VAE autoencoder, based on... Determine the predicted action encoding sequence , The process is actually based on The corresponding action code is selected from the codebook. and The algorithm indexes an integer to a specific action, performs cross-conditional prediction to maintain coherence, and finally generates dance movements through an action decoder, thus achieving separate modeling of the upper and lower body:

[0064]

[0065] in, , The codebooks are located on the upper and lower body, respectively. and Action coding in the middle.

[0066] Cross-conditional prediction refers to the process during model training where, when predicting upper and lower body movements at the current time step, the predictions are simultaneously influenced by historical music, historical upper body movements, and historical lower body movements. In other words, the MotionGPT model at each time step not only predicts information from one modality but also utilizes information from another modality to enhance prediction accuracy.

[0067] Step S4: The model including the VQ-VAE encoder, TGCA module, unit containing PMMM module, TGCA module and Linear and softmax layers, and VQ-VAE decoder is denoted as MotionGPT, and MotionGPT is trained.

[0068] Supervised training optimizes the model, using the action probabilities for the upper and lower body at each time step t. Cross-entropy loss:

[0069] in, The label value corresponds to the known action. In this autoregressive prediction, the model at step t sees the historical information of the music from time 1 to time t and the action from time 0 to time t-1, therefore its output prediction... This represents a "guess about the next action". During training, it's crucial to ensure that the prediction and label are aligned, so the actual action at time t+1 is used. Come and Calculate the loss. If incorrectly used... That's equivalent to letting the model not know Predicting under certain circumstances This is impossible to achieve during the inference phase, so training and inference must be consistent, and predictions always correspond to actions at the next moment.

[0070] Based on the trained and complete prediction model, in the actual dance generation process, i.e. driving dance generation, MotionGPT predicts motion codes based on the initial motion codes (an integer is generated randomly at the beginning) and the entire music, and the VQ-VAE decoder generates new dances.

[0071] See appendix Figure 3 , 4 The model was trained using the publicly available AIST++ dataset to obtain a weighted model, and the quality, diversity, and beat-dance alignment of the dance movements generated by the trained model were evaluated.

[0072] The model was trained using the publicly available AIST++ dataset to obtain a weighted model. AIST++ contains 991 high-quality 3D motion sequences, recorded at 60 frames per second in Skinned Multi-Person Linear (SMPL) format. Of these, 951 sequences were designated for training, and 40 were reserved for evaluation. This invention generates 40 dance sequences on the test set of the AIST++ dataset and samples these generated dance sequences with a sampling length of 20 seconds for performance evaluation.

[0073] This invention trains the model on an NVIDIA A100. VQ-VAE training uses a 512-byte codebook with a batch size of 32, employing the Adam optimizer for 400 training epochs at a learning rate of 3e-5. MotionGPT training also uses 400 epochs with a batch size of 30, an initial learning rate of 3e-4, and a decay rate of 0.1 after 200 epochs. To enhance the quality of generated dance movements, reference audio and motion frames are discarded with a probability of 0.1 during training.

[0074] See appendix Figure 3 The quantitative evaluation metrics followed previous research, quantitatively assessing the generated samples from three key aspects: quality, diversity, and beat-dance alignment. For dance quality, calculations were based on kinematic characteristics. and geometric feature-based Regarding dance diversity, we use indicators. and , representing the distance of kinematic features and the distance of geometric features, respectively. For beat-dance alignment, the beat alignment score (BAS) is used, which quantifies the average distance between the musical beat and its nearest dance beat.

[0075] To verify the superiority of our method (Danceba), we conducted comparative experiments with state-of-the-art methods and real-world results, calculating the aforementioned metrics. Underlined values ​​indicate the best performance among existing methods. Blue indicates results exceeding the current best method. The comparison results are shown in Table 1. Our method achieves state-of-the-art performance in... , , This method significantly outperforms existing methods in terms of BAS (Body Assay) and achieves the best results. Furthermore, this method... It ranks third in terms of indicators.

[0076] By explicitly separating the rhythmic and semantic features of music through phase-based rhythm extraction (PRE), the rhythmic sensitivity in the dance generation process is enhanced. The rhythmic and semantic vectors output by PRE are fed into a temporally gated causal attention (TGCA) module, utilizing a gating mechanism to highlight the global rhythmic structure in cross-conditional causal attention, thus simplifying the contextual relationship between musical rhythm and dance movements. At the spatiotemporal modeling level, the parallel Mamba motion modeling (PMMM) module is used to model the upper and lower body sequences separately to capture finer-grained limb dynamics. This enhances the expressiveness and coherence of the dance movements, ensuring that the dance postures are highly synchronized with the musical rhythm and are highly expressive.

[0077] This invention can be used in virtual human performance and content creation platforms: applied to performances of digital virtual idols, virtual anchors, etc., to achieve synchronized music and dance performances, enhancing the expressiveness of virtual characters and the audience's interactive experience. It can also be applied to dance teaching and education: used to automatically generate instructional dance steps based on different music, assisting beginners in getting started.

[0078] In some embodiments, a music-driven dance generation device based on gated enhanced rhythm-aware feature representation is also provided. The device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement a music-driven dance generation method based on gated enhanced rhythm-aware feature representation.

[0079] It should be understood that the device described in this embodiment includes, but is not limited to, devices including processors and memory, and may also include other devices corresponding to units or modules with information acquisition, information interaction, and control functions, such as signal acquisition devices. The device includes, but is not limited to, PCs, workstations, and mobile devices.

[0080] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A music-driven dance generation method based on gated enhancement rhythm-aware feature representation, characterized in that: Includes the following steps: Step S1: Using two pre-trained motion VQ-VAEs, encode the upper body and lower body motions respectively to obtain the upper body code. and lower body coding Phase features representing musical rhythm are obtained based on musical characteristics. Three independent linear layers were used to encode musical features and upper body characteristics, respectively. and lower body coding Channel dimension alignment is performed to obtain music embedding features. Upper body movement embedded features Lower body movement embedding features And splice together the features ;based on and Obtain rhythm enhancement features ; Step S2, rhythm enhancement feature The data is fed into the Time-Gated Causal Attention (TGCA) module for processing. TGCA utilizes cross-causal attention and gating units to obtain global rhythmic features. : Step S3 The data is fed into the parallel Mamba motion modeling module, i.e., the PMMM module, for processing. Re-separate along the time dimension into global rhythm enhancement upper body motion embedding features Global rhythm enhancement and lower body motion embedding features and global rhythm enhancement music embedding features The data is fed into three parallel Mamba blocks for processing to obtain upper and lower body motion features. , With musical characteristics ;Will , and Features are obtained by piecing them together ; feature The code is then fed into the TGCA module for processing, and the upper and lower body codebooks are generated based on the processing results of the TGCA module. The probability of each action in the current prediction time frame ; based on , respectively in The feature codes of the upper and lower body movements with the highest probability are selected as the predicted action coding sequence. , ; Encoding sequence , The action is output after decoding by the VQ-VAE decoder.

2. The music-driven dance generation method based on gated enhancement rhythm-aware feature representation according to claim 1, characterized in that: Musical features are obtained by processing the original music using the librosa library.

3. The music-driven dance generation method based on gated enhancement rhythm-aware feature representation according to claim 1, characterized in that: Phase features representing musical rhythm are obtained based on musical characteristics. The process includes the following steps: Short-time Fourier transform is used to convert musical features into a time-frequency graph, and the phase angle is extracted. Furthermore, a center-clipping strategy is applied to selectively focus on the phase region with the highest information content to obtain phase features. To obtain the final phase features representing musical rhythm, a linear layer, a batch normalization layer, and an activation function are used to process the phase features. .

4. The music-driven dance generation method based on gated enhancement rhythm-aware feature representation according to claim 3, characterized in that: The application-centric cropping strategy selectively focuses on the phase region with the highest information content. The process of obtaining phase features includes: Regarding the phase angle ,exist Cut out a length of [length] from the middle of the direction. phase angle That is, phase characteristics; This represents the number of time frames after STFT conversion. for The channel dimension, Indicates the length of time. This represents the time downsampling rate.

5. A music-driven dance generation method based on gating-enhanced rhythm-aware feature representation according to any one of claims 1 to 4, characterized in that: based on and Obtain rhythm enhancement features The process includes: Take 3 identical ones Features are obtained by piecing them together and features Fusion: , Enhance the rhythmic features of the output.

6. The music-driven dance generation method based on gated enhancement rhythm-aware feature representation according to claim 5, characterized in that: The processing steps of the Time-Gated Causal Attention Module (TGCA) include: First, it is processed through an RMSnorm normalization layer to obtain... ; Then, a time gating mechanism is applied: Then, based on cross-conditional causal attention Obtain features ; This indicates element-wise multiplication; Finally, global rhythm features are obtained based on RMSnorm and multilayer perceptron. .

7. The music-driven dance generation method based on gated enhancement rhythm-aware feature representation according to claim 6, characterized in that: In the Mamba block, the input first passes through an RMSOrm normalization layer and then through a Mamba layer, and after residual connections, intermediate features are obtained. The intermediate features are then passed through an RMSnorm normalization layer, a GateMlp layer, and finally through residual connections to obtain the output features.

8. The music-driven dance generation method based on gated enhancement rhythm-aware feature representation according to claim 7, characterized in that: Generate upper and lower body codebooks based on the processing results of the TGCA module. The probability of each action in the current prediction time frame The process includes: This will be achieved through the continuous autoregressive Transformer layer in TGCA. The result is denoted as Then, the upper and lower body codebooks are generated through a linear layer and a softmax layer. The probability of each action in the current prediction time frame ,Right now: ,in, , This indicates the codebook The probability of each upper body movement in the current prediction time frame. This indicates the codebook The probability of each lower body movement in the current prediction time frame.

9. The music-driven dance generation method based on gated enhancement rhythm-aware feature representation according to claim 8, characterized in that: The model comprising a VQ-VAE encoder, a TGCA module, units containing PMMM, TGCA modules, and Linear and softmax layers, and a VQ-VAE decoder is denoted as MotionGPT. The loss function used by MotionGPT during training includes the motion probabilities for both the upper and lower body. Cross-entropy loss ; ,in, The label value corresponding to the known action; The probability of upper body and lower body movements; Indicates the length of time.

10. A music-driven dance generation device based on gated enhancement rhythm-aware feature representation, characterized in that: The device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement a music-driven dance generation method based on gating-enhanced rhythm-aware feature representation as described in any one of claims 1 to 9.

Images