A dance motion generation method and system based on hierarchical motion representation learning

Abstract

The application discloses a dance action generation method and system based on hierarchical motion representation learning in the technical field of human action generation and multi-modal generation. The method comprises the following steps: extracting global motion features by using a basic layer global motion feature extraction branch according to a reference action sequence; generating modulation scaling parameters and offset parameters by using an adaptive modulation layer according to the global motion features; generating an initial action sequence by using a dance action diffusion generation module according to a target music condition, a diffusion time step, a noise tensor, the modulation scaling parameters and the offset parameters; generating a to-be-recovered action sequence by using a mask processing module according to the reference action sequence and the initial action sequence; generating recovery level motion features by using a recovery layer motion feature extraction branch according to the global motion features and the to-be-recovered action sequence; and generating a target dance action sequence by using the dance action diffusion generation module according to the recovery level motion features, the target music condition, the diffusion time step and the noise tensor.

Description

Technical Field

[0001] This invention relates to the field of human motion generation and multimodal generation technology, and in particular to a method and system for generating dance movements based on hierarchical motion representation learning. Background Technology

[0002] Dance is an art form closely related to musical rhythm, body posture, and continuous movement expression. Traditional choreography typically relies on professional choreographers to design movements based on musical melodies, beats, and emotions, a complex and labor-intensive process. With the development of applications such as digital humans, film and television animation, game characters, augmented reality / virtual reality, and human-computer interaction, music-driven dance generation technology, which can automatically generate dance movements based on music, has received widespread attention. This technology aims to transform musical input into dynamic dance movement sequences, significantly reducing the creative burden on choreographers.

[0003] Existing music-driven dance generation methods typically rely on music features for motion prediction or diffusion denoising, such as generation schemes based on recurrent neural network autoencoders, dilated convolutional networks, generative adversarial networks, variational autoencoders, cross-modal transformer networks, and optimal transport or diffusion models. While these methods can achieve motion results synchronized with the music beat to some extent, the lack of direct modeling of the user's intentions often results in overly generalized dance patterns, failing to express the user's desired specific movement style, posture preferences, motion contours, or continuous motion features. For the same music clip, different users may expect different interpretations of the movements, and methods relying solely on music cannot provide this flexibility.

[0004] To improve the controllability of generation, some existing methods introduce dance category labels, style labels, text cues, starting poses, sparse keyframes, or local pose constraints as additional conditions. However, style labels and text cues typically only provide coarse-grained category semantic control, making it difficult to specify fine-grained motion patterns in a particular reference dance performance. Starting poses or sparse keyframes mainly constrain local poses at a few time points, failing to fully describe the continuous motion dynamics, global motion tendencies, and local transition details in the reference motion sequence. Furthermore, motion-editing-based methods achieve local control by applying pose constraints at specified time points, but these methods focus on satisfying local constraints at specified locations, failing to model the overall motion characteristics of the reference motion or consider how the reference motion adapts to different target music. Moreover, if the reference motion sequence is directly used as a strong constraint input, the global motion features and local frame-level details of the reference motion have different scopes of influence. Using a single feature representation for fusion can easily lead to insufficient global controllability, unnatural local transitions, or decreased music synchronization. Therefore, how to extract and differentially fuse motion features at different levels in the reference motion sequence is a key challenge for achieving controllable generation of reference motions.

[0005] Diffusion models generate motion sequences through progressive denoising and possess strong multimodal conditional fusion capabilities. Existing pre-trained music-driven motion diffusion generation models have learned the basic mapping relationship between music and dance movements; however, these models are typically not designed to support user-provided reference motion sequences as continuous motion control signals. Furthermore, training a new generation model from scratch is not only computationally expensive but may also result in the loss of prior knowledge about the music-motion mapping in existing models. Therefore, how to retain the pre-trained music-motion mapping capabilities while introducing hierarchical motion features of reference motion sequences, so that the generated results simultaneously satisfy music synchronization and controllable reference movements, has become a pressing technical problem to be solved.

[0006] Therefore, there is an urgent need for a new music-driven dance motion generation scheme with controllable reference motion. This scheme would extract sequence-level global motion features and frame-level local motion details from the reference motion sequence and fuse them with the target music conditions in a way that adapts to a pre-trained diffusion generation model. This would generate natural, synchronized dance motion sequences that are controlled by the reference motion under different music and reference motion combinations. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems of existing music-driven dance generation methods, which mainly rely on music conditions, lack fine-grained control over reference motion sequences, and have difficulty maintaining continuous motion features and music synchronization simultaneously due to sparse posture constraints. This invention provides a dance motion generation method and system based on hierarchical motion representation learning, which can take into account both music rhythm synchronization and reference motion control, thereby improving the fine-grained controllability and naturalness of dance motion generation.

[0008] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0009] In a first aspect, the present invention provides a method for generating dance movements based on hierarchical motion representation learning, comprising:

[0010] The obtained reference motion sequence, target music conditions, diffusion time step, and noise tensor are input into the pre-trained dance motion generation model for two-stage inference to obtain the target dance motion sequence.

[0011] The two-stage reasoning includes:

[0012] Based on the reference action sequence, global motion features are extracted using the base layer global motion feature extraction branch of the motion representation extraction module;

[0013] Based on the global motion characteristics, modulation scaling parameters and offset parameters are generated using an adaptive modulation layer;

[0014] Based on the target music conditions, diffusion time step, noise tensor, modulation scaling parameters, and offset parameters, the initial motion sequence is obtained by performing basic inference using the dance motion diffusion generation module.

[0015] Based on the reference action sequence and the initial action sequence, the masking module is used to generate the action sequence to be recovered.

[0016] Based on the global motion features and the sequence of actions to be recovered, the recovery-level motion features are generated using the recovery layer motion feature extraction branch of the motion representation extraction module;

[0017] Based on the recovery-level motion characteristics, target music conditions, diffusion time step, and noise tensor, the target dance motion sequence is obtained by using the dance motion diffusion generation module for refined reasoning.

[0018] Optionally, the target music conditions are obtained by extracting features from the target music segment using a music feature extractor;

[0019] The music feature extractor is either a feature extractor based on handcrafted acoustic features or a pre-trained music encoder based on neural networks.

[0020] The reference motion sequence is a motion example provided to the user, including motion capture data, existing dance motion databases, 3D human pose estimation results, manually edited skeletal animations, virtual character motion assets, or other data that can be converted into human skeletal motion.

[0021] The diffusion time step is a preset value;

[0022] The noise tensor is Gaussian noise.

[0023] Optionally, the step of extracting global motion features based on the reference action sequence using the base layer global motion feature extraction branch of the motion representation extraction module includes:

[0024] The reference action sequence is input into the first linear embedding layer to obtain the reference action embedding sequence;

[0025] The reference action embedding sequence is concatenated with the learnable global marker to obtain the base layer input sequence;

[0026] The base layer input sequence is input into the transformer encoder, and the output of the transformer encoder is linearly projected to obtain the global motion features; wherein, the transformer encoder includes multiple self-attention heads.

[0027] Optionally, the step of generating recovery-level motion features using the recovery layer motion feature extraction branch of the motion representation extraction module based on the global motion features and the sequence of actions to be recovered includes:

[0028] The sequence of actions to be recovered is input into the second linear embedding layer, and position encoding is added to obtain the embedded sequence of actions to be recovered;

[0029] The embedded sequence of the action to be recovered and the global motion features are input into the recovery layer transformer, and the output of the transformer encoder is linearly projected to obtain the recovery level motion features; wherein, the recovery layer transformer includes a multilayer perceptron.

[0030] Optionally, the dance motion diffusion generation module includes an input processing module, a multi-layer decoder block, and an output processing module;

[0031] The initial motion sequence is obtained by performing basic inference using the dance motion diffusion generation module based on the target music conditions, diffusion time step, noise tensor, modulation scaling parameters, and offset parameters, including:

[0032] The target music conditions, diffusion time steps, and noise tensors are input into the input processing module, which generates motion hidden feature sequences, time step embedding vectors, and music condition features, and injects the time step embedding vectors into each network layer of the pre-trained dance motion generation model.

[0033] The motion concealment feature sequence, music condition features, modulation scaling parameters, and offset parameters are input into the multi-layer decoder block for layer-by-layer processing. The multi-layer decoder block injects music condition features into the motion concealment feature sequence through a cross-attention mechanism, and modulates the output of each layer using modulation scaling parameters and offset parameters. The decoded features of the last layer after modulation are used as the final decoded features.

[0034] The output processing module is used to map the final decoded features into an initial action sequence.

[0035] Optionally, the input processing module includes an action input embedding module, a time step embedding module, and a music encoder;

[0036] The action input embedding module is used to map the noise tensor into an action sequence with a uniform hidden dimension, and add position encoding to obtain an action hidden feature sequence containing time frame position information.

[0037] The time step embedding module is used to encode the diffusion time step to obtain a time step embedding vector.

[0038] The music encoder is used to map the target music conditions into music condition features;

[0039] The processing procedure for each layer of the multi-layer decoder block includes:

[0040] The input action features of this layer are used as queries, and the music condition features are used as keys and values. Cross-attention calculation is performed to obtain music condition-enhanced action features.

[0041] The motion features enhanced by the music conditions are sequentially scaled and offset using the modulation scaling parameters and offset parameters to obtain the modulated decoded features.

[0042] The modulated decoded features are used as the input action features for the next layer of decoder blocks;

[0043] The input action features of the first layer decoder block are the action hidden feature sequence, and the input action features of each subsequent layer decoder block are the decoded features modulated by the previous layer.

[0044] Optionally, the step of refining the reasoning based on the restored motion features, target music conditions, diffusion time step, and noise tensor using the dance motion diffusion generation module to obtain the target dance motion sequence includes:

[0045] The target music conditions, diffusion time steps, and noise tensors are input into the input processing module, which generates motion hidden feature sequences, time step embedding vectors, and music condition features, and injects the time step embedding vectors into each network layer of the pre-trained dance motion generation model.

[0046] The motion concealment feature sequence, music condition features, and relativity level motion features are input into the multilayer decoder block for layer-by-layer processing. The multilayer decoder block injects music condition features into the motion concealment feature sequence through a cross-attention mechanism, and uses relativity level motion features to add the output of each layer element by element. The decoded features after adding the last layer element by element are used as the final decoded features.

[0047] The output processing module is used to map the final decoded features into a target dance movement sequence.

[0048] Optionally, the step of generating the action sequence to be recovered using the masking module based on the reference action sequence and the initial action sequence includes:

[0049] Calculate the acceleration of each joint in the reference motion sequence;

[0050] Based on a preset local window, the location of the local maximum value of the acceleration of each joint is used as a keyframe.

[0051] Calculate the frame-level similarity between the keyframe and each frame in the initial action sequence;

[0052] An adaptive mask is constructed based on a preset similarity threshold and the frame-level similarity. If the frame-level similarity of a frame is less than the similarity threshold, the mask value of that frame is set to 0, indicating that the frame needs to be masked; otherwise, the mask value of that frame is set to 1, indicating that the frame needs to be retained.

[0053] The adaptive mask is multiplied element-wise with the initial action sequence to obtain the action sequence to be recovered;

[0054] And / or,

[0055] The adaptive modulation layer comprises a first fully connected layer, a SiLU activation function, and a second fully connected layer connected in sequence.

[0056] Optionally, the training process of the dance motion generation model includes:

[0057] The dance motion generation model is iteratively trained based on a pre-built training set. When a preset training stopping condition is reached, the model parameters that minimize the model loss are loaded into the pre-built dance motion generation model to obtain a trained dance motion generation model. The training set includes music conditions, ground truth dance motion sequences, reference motion sequences randomly sampled from the ground truth dance motion sequences, noisy motion samples obtained by adding noise to the ground truth dance motion sequences, and a preset diffusion time step.

[0058] A single iteration of training includes:

[0059] Based on the reference action sequence, global motion features are extracted using the base layer global motion feature extraction branch of the motion representation extraction module;

[0060] Based on the global motion characteristics, modulation scaling parameters and offset parameters are generated using an adaptive modulation layer;

[0061] Based on the music conditions, noise motion samples, diffusion time steps, modulation scaling parameters, and offset parameters, the initial motion sequence is obtained by performing basic inference using the dance motion diffusion generation module.

[0062] Based on the true value dance movement sequence, the mask processing module performs a random mask based on cosine scheduling in the time frame dimension to obtain the movement sequence to be recovered.

[0063] Based on the global motion features and the sequence of actions to be recovered, the recovery-level motion features are generated using the recovery layer motion feature extraction branch of the motion representation extraction module;

[0064] Based on the recovery-level motion characteristics, music conditions, diffusion time steps, and noise motion samples, the dance motion diffusion generation module is used to perform refined reasoning to obtain the target dance motion sequence.

[0065] The model loss is obtained by weighted fusion based on the error between the initial action sequence and the ground truth dance action sequence, and the error between the target dance action sequence and the ground truth dance action sequence.

[0066] The model parameters of the dance motion generation model are updated based on the model loss.

[0067] Secondly, the present invention provides a dance movement generation system based on hierarchical motion representation learning, comprising:

[0068] The two-stage inference module is used to input the obtained reference motion sequence, target music conditions, diffusion time step, and noise tensor into the pre-trained dance motion generation model for two-stage inference to obtain the target dance motion sequence.

[0069] The base layer global motion feature module is used to extract global motion features based on the reference action sequence;

[0070] An adaptive modulation module is used to generate modulation scaling parameters and offset parameters based on the global motion features;

[0071] The basic reasoning module is used to perform basic reasoning based on the target music conditions, diffusion time step, noise tensor, modulation scaling parameters and offset parameters, using the dance motion diffusion generation module to obtain the initial motion sequence.

[0072] The mask processing module is used to generate the action sequence to be recovered based on the reference action sequence and the initial action sequence;

[0073] The recovery layer motion feature extraction module is used to generate recovery-level motion features based on the global motion features and the sequence of actions to be recovered.

[0074] The refinement inference module is used to perform refinement inference based on the recovery-level motion features, target music conditions, diffusion time steps, and noise tensors, using the dance motion diffusion generation module to obtain the target dance motion sequence.

[0075] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0076] 1. By providing example-level control signals through reference motion sequences, users can control the motion features of generated dances by giving a reference motion. Compared with solutions that rely solely on music conditions, style tags, or sparse keyframes, this provides more granular motion control capabilities.

[0077] 2. A motion representation extraction module consisting of a base layer and a recovery layer is adopted to model the global motion features and local motion details in the reference action separately, avoiding the problem that a single action representation cannot take into account the overall motion tendency and the natural transition of the frame. The base layer captures the sequence-level global motion features through a transformer encoder, and the recovery layer learns the frame-level local temporal dependencies through a mask recovery mechanism. The two layers are trained together to ensure the coherence of the motion representation.

[0078] 3. Global motion features are injected into the dance motion diffusion generation module through adaptive modulation, and the recovery-level motion features are supplemented to the decoder block output through feature-level fusion. This allows the generation process to retain the music synchronization capability of the pre-trained model while incorporating control information from the reference motion sequence. This hierarchical fusion strategy adopts differentiated fusion methods based on the different scopes of global and local features, effectively avoiding the problems of insufficient controllability or decreased synchronization capability caused by a single fusion method.

[0079] 4. It can be combined with a low-rank adaptation module to efficiently adapt parameters of the pre-trained model. Only a few new parameters need to be updated to adapt to the reference action conditions, reducing training and deployment costs. Through a two-stage inference and adaptive mask refinement mechanism, low similarity frames with large deviations from the reference action keyframes are refined by a recovery layer, thereby improving the local naturalness, temporal continuity and music synchronization quality of the generated action. It can be widely used in fields such as digital human dance generation, game character animation generation, film and television animation production, virtual anchors, augmented reality / virtual reality content production, dance-assisted creation and intelligent choreography systems. Attached Figure Description

[0080] Figure 1A flowchart of a dance movement generation method based on hierarchical motion representation learning according to an embodiment of the present invention;

[0081] Figure 2 This is a schematic diagram of the hierarchical motion representation extraction module provided in an embodiment of the present invention;

[0082] Figure 3 This is a schematic diagram of the structure of a dance motion diffusion generation module provided in an embodiment of the present invention;

[0083] Figure 4 This is a schematic diagram of a two-stage inference and adaptive masking process provided according to an embodiment of the present invention. Detailed Implementation

[0084] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0085] It should be noted that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0086] Example 1

[0087] This invention discloses a method for generating dance movements based on hierarchical motion representation learning, with reference to... Figure 1 As shown, the specific steps include the following:

[0088] The obtained reference motion sequence, target music conditions, diffusion time step, and noise tensor are input into the pre-trained dance motion generation model for two-stage inference to obtain the target dance motion sequence.

[0089] refer to Figure 4 As shown, the two-stage reasoning includes:

[0090] S1, based on the reference action sequence, extract global motion features using the basic layer global motion feature extraction branch of the motion representation extraction module;

[0091] S2, Based on the global motion characteristics, the modulation scaling parameters and offset parameters are generated using the adaptive modulation layer;

[0092] S3. Based on the target music conditions, diffusion time step, noise tensor, modulation scaling parameters and offset parameters, the initial motion sequence is obtained by performing basic reasoning using the dance motion diffusion generation module.

[0093] S4, Based on the reference action sequence and the initial action sequence, the mask processing module generates the action sequence to be recovered;

[0094] S5, based on the global motion features and the sequence of actions to be recovered, the recovery-level motion features are generated using the recovery layer motion feature extraction branch of the motion representation extraction module;

[0095] S6. Based on the recovery-level motion characteristics, target music conditions, diffusion time step, and noise tensor, the dance motion diffusion generation module is used to perform refined reasoning to obtain the target dance motion sequence.

[0096] The target music condition in this embodiment can be obtained by extracting music features from the input music segment. The music feature extractor can be a feature extractor based on handcrafted acoustic features, such as beat, rhythm, and timbre features extracted based on the librosa library, or a pre-trained music encoder based on neural networks, such as acoustic embedding features extracted based on the Jukebox model. The target music condition can have the same or aligned time length as the generated action sequence, which is used to provide a basis for music beat and rhythm synchronization during the diffusion generation process.

[0097] The reference motion sequence provides users with motion examples, which can come from motion capture data, existing dance motion databases, 3D human pose estimation results, manually edited skeletal animations, virtual character motion assets, or other data that can be converted into human skeletal motion. The reference motion sequence is not required to have the same length as the target music, nor is it required to belong to the same dance style. That is, cross-style combinations between the reference motion sequence and the target music are allowed, and the generated result should adapt to the rhythm of the target music while maintaining the motion characteristics of the reference motion.

[0098] This embodiment considers that the motion features in the reference motion sequence include at least two different levels: First, global motion features spanning the entire sequence, such as overall motion tendency, body part usage preference, motion amplitude, motion contour, and posture distribution. This level is a sequence-level attribute and requires consistent guidance. Second, local frame-level motion details, such as local transitions, subtle changes near key postures, short-term continuity, and local naturalness. This level is a frame-level attribute and requires temporal coherence. Because the two levels have different scopes, this embodiment adopts the following... Figure 2 The hierarchical motion representation extraction modules shown are modeled separately, and the two levels are jointly trained to ensure the coherence of the motion representation.

[0099] In step S1, the step of extracting global motion features based on the reference action sequence using the base layer global motion feature extraction branch of the motion representation extraction module includes:

[0100] S1.1, The reference action sequence is input into the first linear embedding layer to obtain the reference action embedding sequence;

[0101] S1.2, the reference action embedding sequence is concatenated with the learnable global label to obtain the base layer input sequence; the learnable global label is used to aggregate the global information of the entire reference action sequence during the transformer encoding process, and its corresponding output is the source of global motion features;

[0102] S1.3, the base layer input sequence is input into the transformer encoder. The transformer encoder models the long-range dependency between different time frames through a multi-head self-attention mechanism, thereby capturing the overall motion pattern in the reference action sequence. After processing by several transformer layers, the output corresponding to the learnable global label is extracted, and the global motion features are obtained through linear layer projection. The global motion features characterize the sequence-level motion features of the reference action sequence, including overall motion tendency, posture preference, and motion contour.

[0103] In step S2, the adaptive modulation layer includes a first fully connected layer, a SiLU activation function, and a second fully connected layer connected in sequence.

[0104] In step S3, refer to Figure 3 As shown, the dance motion diffusion generation module includes an input processing module, a multi-layer decoder block, and an output processing module. The dance motion diffusion generation module can be a diffusion denoising network built based on a transformer, a temporal convolutional network, or a U-Net structure. During the pre-training phase, this module has learned the basic mapping relationship from music conditions to dance motion sequences, including the alignment ability between music beats and motion rhythms. The multi-layer decoder block uses LoRA (Low-Rank Adaptation) to adjust the parameters of the decoder block. In the figure, Mod represents the modulo operation. The output processing module includes fully connected layers.

[0105] The initial motion sequence is obtained by performing basic inference using the dance motion diffusion generation module based on the target music conditions, diffusion time step, noise tensor, modulation scaling parameters, and offset parameters, including:

[0106] S3.1, The target music conditions, diffusion time step, and noise tensor are input into the input processing module. The input processing module generates a motion hidden feature sequence, a time step embedding vector, and music condition features, and injects the time step embedding vector into each network layer of the pre-trained dance motion generation model. The input processing module includes a motion input embedding module, a time step embedding module, and a music encoder.

[0107] The action input embedding module is used to map the noise tensor into an action sequence with a uniform hidden dimension and add position encoding to obtain an action hidden feature sequence containing time frame position information.

[0108] The time step embedding module is used to encode the diffusion time step to obtain a time step embedding vector.

[0109] The music encoder is used to map the target music conditions into music condition features;

[0110] S3.2, the action concealment feature sequence, music conditional features, modulation scaling parameters, and offset parameters are input into the multi-layer decoder block for layer-by-layer processing. The multi-layer decoder block injects music conditional features into the action concealment feature sequence through a cross-attention mechanism, and modulates the output of each layer using the modulation scaling parameters and offset parameters. The decoded features of the last layer after modulation are used as the final decoded features. The processing procedure of each layer of the multi-layer decoder block includes:

[0111] The input action features of this layer are used as queries, and the music condition features are used as keys and values. Cross-attention calculation is performed to obtain music condition-enhanced action features.

[0112] The motion features enhanced by the music conditions are sequentially scaled and offset using the modulation scaling parameters and offset parameters to obtain the modulated decoded features.

[0113] The modulated decoded features are used as the input action features for the next layer of decoder blocks;

[0114] The input action features of the first layer decoder block are the action hidden feature sequence, and the input action features of each subsequent layer decoder block are the decoded features modulated by the previous layer.

[0115] S3.3, The output processing module is used to map the final decoded features into an initial action sequence.

[0116] Step S4, which involves generating the action sequence to be recovered using the masking module based on the reference action sequence and the initial action sequence, includes:

[0117] S4.1 Calculate the acceleration of each joint in the reference action sequence;

[0118] S4.2, based on a preset local window, the position of the local maximum value of the acceleration of each joint is taken as a key frame; the key frame is used to characterize the significant posture and action transition in the reference action sequence. In other embodiments, the key frame can also be determined based on the change in action speed, the magnitude of posture change, or user specification.

[0119] S4.3 Calculate the frame-level similarity between the keyframe and each frame in the initial action sequence; the frame-level similarity can be determined based on Euclidean distance, joint angle distance, skeleton space distance, kinematic feature distance or a combination thereof; for each frame, the minimum distance or maximum similarity between it and all keyframes can be used as the reference matching score for that frame;

[0120] S4.4, Construct an adaptive mask based on the preset similarity threshold and the frame-level similarity. If the frame-level similarity of a certain frame is less than the similarity threshold, the mask value of that frame is set to 0, indicating that the frame needs to be masked; otherwise, the mask value of that frame is set to 1, indicating that the frame needs to be retained.

[0121] S4.5, Multiply the adaptive mask element-wise with the initial action sequence to obtain the action sequence to be recovered.

[0122] In step S5, based on the global motion features and the sequence of actions to be recovered, the recovery-level motion features are generated using the recovery layer motion feature extraction branch of the motion representation extraction module, including:

[0123] The sequence of actions to be recovered is input into the second linear embedding layer, and position encoding is added to obtain the embedded sequence of actions to be recovered;

[0124] The embedded sequence of the action to be recovered and the global motion features are input into the recovery layer transformer, and the output of the transformer encoder is linearly projected to obtain the recovery level motion features; wherein, the recovery layer transformer includes a multilayer perceptron.

[0125] In step S6, the refinement reasoning based on the recovery-level motion characteristics, target music conditions, diffusion time step, and noise tensor, using the dance motion diffusion generation module, yields the target dance motion sequence, including:

[0126] S6.1, The target music conditions, diffusion time steps, and noise tensors are input into the input processing module. The input processing module generates motion hidden feature sequences, time step embedding vectors, and music condition features, and injects the time step embedding vectors into each network layer of the pre-trained dance motion generation model.

[0127] S6.2, The motion hiding feature sequence, music condition features and restoration level motion features are input into the multi-layer decoder block for layer-by-layer processing. The multi-layer decoder block injects music condition features into the motion hiding feature sequence through a cross-attention mechanism, and uses restoration level motion features to add the output of each layer element by element. The decoding features after adding the last layer element by element are used as the final decoding features.

[0128] S6.3, The output processing module is used to map the final decoded features into a target dance movement sequence.

[0129] In this embodiment, the training process of the dance motion generation model includes:

[0130] The dance motion generation model is iteratively trained based on a pre-built training set. When a preset training stopping condition is reached, the model parameters that minimize the model loss are loaded into the pre-built dance motion generation model to obtain a trained dance motion generation model. The training set includes music conditions, ground truth dance motion sequences, reference motion sequences randomly sampled from the ground truth dance motion sequences, noisy motion samples obtained by adding noise to the ground truth dance motion sequences, and a preset diffusion time step.

[0131] A single iteration of training includes:

[0132] Based on the reference action sequence, global motion features are extracted using the base layer global motion feature extraction branch of the motion representation extraction module;

[0133] Based on the global motion characteristics, modulation scaling parameters and offset parameters are generated using an adaptive modulation layer;

[0134] Based on the music conditions, noise motion samples, diffusion time steps, modulation scaling parameters, and offset parameters, the initial motion sequence is obtained by performing basic inference using the dance motion diffusion generation module.

[0135] Based on the true value dance movement sequence, the mask processing module performs a random mask based on cosine scheduling in the time frame dimension to obtain the movement sequence to be recovered.

[0136] Based on the global motion features and the sequence of actions to be recovered, the recovery-level motion features are generated using the recovery layer motion feature extraction branch of the motion representation extraction module;

[0137] Based on the recovery-level motion characteristics, music conditions, diffusion time steps, and noise motion samples, the dance motion diffusion generation module is used to perform refined reasoning to obtain the target dance motion sequence.

[0138] Based on the error between the initial motion sequence and the ground truth dance motion sequence, and the error between the target dance motion sequence and the ground truth dance motion sequence, a model loss is obtained through weighted fusion; the model loss may be mean squared error loss, absolute error loss, velocity loss, acceleration loss, foot contact loss, or a combination thereof.

[0139] The model parameters of the dance motion generation model are updated based on the model loss.

[0140] In this embodiment, reference action conditions are randomly discarded during training, and a learnable empty condition embedding is used to represent the state without reference action conditions. This supports guided generation during the inference phase, allowing users to adjust the influence of reference actions on the generated results through guided weights. Specifically, during the inference phase, the final prediction result is a weighted fusion of the prediction result with reference action conditions and the prediction result without reference action conditions, where the guided weights control the degree of influence of reference action conditions.

[0141] Example 2

[0142] Based on the same inventive concept as Embodiment 1, this embodiment of the invention discloses a dance movement generation system based on hierarchical motion representation learning, comprising:

[0143] The two-stage inference module is used to input the obtained reference motion sequence, target music conditions, diffusion time step, and noise tensor into the pre-trained dance motion generation model for two-stage inference to obtain the target dance motion sequence.

[0144] The base layer global motion feature module is used to extract global motion features based on the reference action sequence;

[0145] An adaptive modulation module is used to generate modulation scaling parameters and offset parameters based on the global motion features;

[0146] The basic reasoning module is used to perform basic reasoning based on the target music conditions, diffusion time step, noise tensor, modulation scaling parameters and offset parameters, using the dance motion diffusion generation module to obtain the initial motion sequence.

[0147] The mask processing module is used to generate the action sequence to be recovered based on the reference action sequence and the initial action sequence;

[0148] The recovery layer motion feature extraction module is used to generate recovery-level motion features based on the global motion features and the sequence of actions to be recovered.

[0149] The refinement inference module is used to perform refinement inference based on the recovery-level motion features, target music conditions, diffusion time steps, and noise tensors, using the dance motion diffusion generation module to obtain the target dance motion sequence.

[0150] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for generating dance movements based on hierarchical motion representation learning, characterized in that, include: The obtained reference motion sequence, target music conditions, diffusion time step, and noise tensor are input into the pre-trained dance motion generation model for two-stage inference to obtain the target dance motion sequence. The two-stage reasoning includes: Based on the reference action sequence, global motion features are extracted using the base layer global motion feature extraction branch of the motion representation extraction module; Based on the global motion characteristics, modulation scaling parameters and offset parameters are generated using an adaptive modulation layer; Based on the target music conditions, diffusion time step, noise tensor, modulation scaling parameters, and offset parameters, the initial motion sequence is obtained by performing basic inference using the dance motion diffusion generation module. Based on the reference action sequence and the initial action sequence, the masking module is used to generate the action sequence to be recovered. Based on the global motion features and the sequence of actions to be recovered, the recovery-level motion features are generated using the recovery layer motion feature extraction branch of the motion representation extraction module; Based on the recovery-level motion characteristics, target music conditions, diffusion time step, and noise tensor, the target dance motion sequence is obtained by using the dance motion diffusion generation module for refined reasoning.

2. The dance movement generation method based on hierarchical motion representation learning according to claim 1, characterized in that, The target music conditions are obtained by extracting features from the target music segment using a music feature extractor; The music feature extractor is either a feature extractor based on handcrafted acoustic features or a pre-trained music encoder based on neural networks. The reference motion sequence is a motion example provided to the user, including motion capture data, existing dance motion databases, 3D human pose estimation results, manually edited skeletal animations, virtual character motion assets, or other data that can be converted into human skeletal motion. The diffusion time step is a preset value; The noise tensor is Gaussian noise.

3. The dance movement generation method based on hierarchical motion representation learning according to claim 1, characterized in that, The step of extracting global motion features based on the reference action sequence using the base layer global motion feature extraction branch of the motion representation extraction module includes: The reference action sequence is input into the first linear embedding layer to obtain the reference action embedding sequence; The reference action embedding sequence is concatenated with the learnable global marker to obtain the base layer input sequence; The base layer input sequence is input into the transformer encoder, and the output of the transformer encoder is linearly projected to obtain the global motion features; wherein, the transformer encoder includes multiple self-attention heads.

4. The dance movement generation method based on hierarchical motion representation learning according to claim 1, characterized in that, The step of generating recovery-level motion features using the recovery layer motion feature extraction branch of the motion representation extraction module based on the global motion features and the sequence of actions to be recovered includes: The sequence of actions to be recovered is input into the second linear embedding layer, and position encoding is added to obtain the embedded sequence of actions to be recovered; The embedded sequence of the action to be recovered and the global motion features are input into the recovery layer transformer, and the output of the transformer encoder is linearly projected to obtain the recovery level motion features; wherein, the recovery layer transformer includes a multilayer perceptron.

5. The dance movement generation method based on hierarchical motion representation learning according to claim 1, characterized in that, The dance motion diffusion generation module includes an input processing module, a multi-layer decoder block, and an output processing module. The initial motion sequence is obtained by performing basic inference using the dance motion diffusion generation module based on the target music conditions, diffusion time step, noise tensor, modulation scaling parameters, and offset parameters, including: The target music conditions, diffusion time steps, and noise tensors are input into the input processing module, which generates motion hidden feature sequences, time step embedding vectors, and music condition features, and injects the time step embedding vectors into each network layer of the pre-trained dance motion generation model. The motion concealment feature sequence, music condition features, modulation scaling parameters, and offset parameters are input into the multi-layer decoder block for layer-by-layer processing. The multi-layer decoder block injects music condition features into the motion concealment feature sequence through a cross-attention mechanism, and modulates the output of each layer using modulation scaling parameters and offset parameters. The decoded features of the last layer after modulation are used as the final decoded features. The output processing module is used to map the final decoded features into an initial action sequence.

6. The dance movement generation method based on hierarchical motion representation learning according to claim 5, characterized in that, The input processing module includes an action input embedding module, a time step embedding module, and a music encoder; The action input embedding module is used to map the noise tensor into an action sequence with a uniform hidden dimension, and add position encoding to obtain an action hidden feature sequence containing time frame position information. The time step embedding module is used to encode the diffusion time step to obtain a time step embedding vector. The music encoder is used to map the target music conditions into music condition features; The processing procedure for each layer of the multi-layer decoder block includes: The input action features of this layer are used as queries, and the music condition features are used as keys and values. Cross-attention calculation is performed to obtain music condition-enhanced action features. The motion features enhanced by the music conditions are sequentially scaled and offset using the modulation scaling parameters and offset parameters to obtain the modulated decoded features. The modulated decoded features are used as the input action features for the next layer of decoder blocks; The input action features of the first layer decoder block are the action hidden feature sequence, and the input action features of each subsequent layer decoder block are the decoded features modulated by the previous layer.

7. The dance movement generation method based on hierarchical motion representation learning according to claim 1, characterized in that, The process involves refining the reasoning based on the restored motion characteristics, target music conditions, diffusion time step, and noise tensor using a dance motion diffusion generation module to obtain the target dance motion sequence, including: The target music conditions, diffusion time steps, and noise tensors are input into the input processing module, which generates motion hidden feature sequences, time step embedding vectors, and music condition features, and injects the time step embedding vectors into each network layer of the pre-trained dance motion generation model. The motion concealment feature sequence, music condition features, and relativity level motion features are input into the multilayer decoder block for layer-by-layer processing. The multilayer decoder block injects music condition features into the motion concealment feature sequence through a cross-attention mechanism, and uses relativity level motion features to add the output of each layer element by element. The decoded features after adding the last layer element by element are used as the final decoded features. The output processing module is used to map the final decoded features into a target dance movement sequence.

8. The dance movement generation method based on hierarchical motion representation learning according to claim 1, characterized in that, The step of generating the action sequence to be recovered using the masking module based on the reference action sequence and the initial action sequence includes: Calculate the acceleration of each joint in the reference motion sequence; Based on a preset local window, the location of the local maximum value of the acceleration of each joint is used as a keyframe. Calculate the frame-level similarity between the keyframe and each frame in the initial action sequence; An adaptive mask is constructed based on a preset similarity threshold and the frame-level similarity. If the frame-level similarity of a frame is less than the similarity threshold, the mask value of that frame is set to 0, indicating that the frame needs to be masked; otherwise, the mask value of that frame is set to 1, indicating that the frame needs to be retained. The adaptive mask is multiplied element-wise with the initial action sequence to obtain the action sequence to be recovered; And / or, The adaptive modulation layer comprises a first fully connected layer, a SiLU activation function, and a second fully connected layer connected in sequence.

9. The dance movement generation method based on hierarchical motion representation learning according to claim 1, characterized in that, The training process of the dance motion generation model includes: The dance motion generation model is iteratively trained based on a pre-built training set. When a preset training stopping condition is reached, the model parameters that minimize the model loss are loaded into the pre-built dance motion generation model to obtain a trained dance motion generation model. The training set includes music conditions, ground truth dance motion sequences, reference motion sequences randomly sampled from the ground truth dance motion sequences, noisy motion samples obtained by adding noise to the ground truth dance motion sequences, and a preset diffusion time step. A single iteration of training includes: Based on the reference action sequence, global motion features are extracted using the base layer global motion feature extraction branch of the motion representation extraction module; Based on the global motion characteristics, modulation scaling parameters and offset parameters are generated using an adaptive modulation layer; Based on the music conditions, noise motion samples, diffusion time steps, modulation scaling parameters, and offset parameters, the initial motion sequence is obtained by performing basic inference using the dance motion diffusion generation module. Based on the true value dance movement sequence, the mask processing module performs a random mask based on cosine scheduling in the time frame dimension to obtain the movement sequence to be recovered. Based on the global motion features and the sequence of actions to be recovered, the recovery-level motion features are generated using the recovery layer motion feature extraction branch of the motion representation extraction module; Based on the recovery-level motion characteristics, music conditions, diffusion time steps, and noise motion samples, the dance motion diffusion generation module is used to perform refined reasoning to obtain the target dance motion sequence. The model loss is obtained by weighted fusion based on the error between the initial action sequence and the ground truth dance action sequence, and the error between the target dance action sequence and the ground truth dance action sequence. The model parameters of the dance motion generation model are updated based on the model loss.

10. A dance movement generation system based on hierarchical motion representation learning, characterized in that, include: The two-stage inference module is used to input the obtained reference motion sequence, target music conditions, diffusion time step, and noise tensor into the pre-trained dance motion generation model for two-stage inference to obtain the target dance motion sequence. The base layer global motion feature module is used to extract global motion features based on the reference action sequence; An adaptive modulation module is used to generate modulation scaling parameters and offset parameters based on the global motion features; The basic reasoning module is used to perform basic reasoning based on the target music conditions, diffusion time step, noise tensor, modulation scaling parameters and offset parameters, using the dance motion diffusion generation module to obtain the initial motion sequence. The mask processing module is used to generate the action sequence to be recovered based on the reference action sequence and the initial action sequence; The recovery layer motion feature extraction module is used to generate recovery-level motion features based on the global motion features and the sequence of actions to be recovered. The refinement inference module is used to perform refinement inference based on the recovery-level motion features, target music conditions, diffusion time steps, and noise tensors, using the dance motion diffusion generation module to obtain the target dance motion sequence.

Images