Dance generation model training method and dance generation method

Abstract

The embodiment of the application relates to the technical field of dance generation, and provides a dance generation model training method and a dance generation method, a deep-coupled neural network architecture is constructed, a dynamics constraint module based on a space-time graph neural network is innovatively introduced in a latent space of a variational autoencoder, specific dynamics bias is explicitly predicted and applied to an action intention according to a specified clothing type, dance action sequences conforming to physical laws and having specific clothing dynamics are forced to be generated by the network output, the generated dance action sequences can not only match music rhythm, but also can truly present physical dynamics of the specified clothing, and the authenticity and expressiveness of dance generation under complex clothing are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of dance generation technology, and in particular to a training method for a dance generation model and a dance generation method. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning, the fields of computer animation and digital content generation have undergone tremendous changes. Among them, cross-modal music-to-dance generation, as a challenging research topic, shows broad application prospects in future dance teaching, virtual idols, game entertainment, animation production, and metaverse human-computer interaction. Its core goal is to automatically generate a sequence of natural and realistic human dance movements that are highly matched in rhythm, cadence, and emotion based on the input music audio signal.

[0003] Early dance generation methods largely relied on rule-based systems or retrieval-based techniques. These methods typically required building massive motion databases, resulting in extremely limited diversity and creativity in the generated movements, and difficulty in handling complex musical variations. In recent years, data-driven methods based on deep neural networks have gradually become the mainstream approach for dance generation. Researchers have utilized Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTMs), and more advanced generative models such as Transformer architectures, Generative Adversarial Networks (GANs), and Variational Autoencoders (VAEs) to attempt to establish complex mapping relationships between musical features and human motion features. These existing techniques have made significant progress in capturing the alignment of musical rhythm and movement beats, enabling the generation of relatively fluid and artistically appealing dance segments.

[0004] However, most existing mainstream dance generation methods are based on the assumption of a "universal human body." Specifically, these methods typically employ simplified assumptions during model training and generation, treating the human body as a standard skeletal structure unconstrained by external physics, or assuming the character wears tight-fitting clothing that has virtually no impact on movement. While this simplification reduces modeling difficulty, it severely ignores the significant physical effects of complex clothing on human movement in the real world.

[0005] In practical applications, especially when dealing with costumes with significant physical characteristics, such as traditional Hanfu (e.g., wide-sleeved robes), ethnic costumes (e.g., Tibetan robes), elaborate formal wear (e.g., large skirts), or heavy armor, the mass, inertia, damping, and structural limitations of the clothing itself exert considerable constraints on the dancer's movement trajectory, range of motion, and the perceived force of their movements. For example, when spinning in a heavy skirt, greater rotational inertia needs to be overcome, resulting in a noticeable lag in the initiation and stopping of movements; when wearing wide-sleeved robes, rapid arm movements are constrained by air resistance and fabric tension.

[0006] Existing technologies lack the ability to perceive and model the specific dynamic constraints of clothing, resulting in dance movements that often exhibit a sense of "weightlessness" or "floating." This manifests as severe physical inconsistencies when applying general motion data to virtual character models wearing complex clothing, such as arms effortlessly passing through wide sleeves (clipping), heavy skirts failing to move or lag appropriately with the body, and overall movements lacking the necessary power and realism. This significantly limits the effectiveness of existing dance generation technologies in high-quality animation production and immersive virtual experiences.

[0007] Therefore, how to develop a new dance generation method that can not only maintain a high degree of harmony between movement and music, but also explicitly perceive and simulate the complex physical constraints and dynamic effects of specific clothing types on human movement, thereby generating dance sequences with high realism and dynamic expressiveness of specific clothing, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] The present invention aims to solve at least one of the problems existing in the prior art, and to provide a training method for a dance generation model and a dance generation method.

[0009] One aspect of the present invention provides a method for training a dance generation model, the method comprising the following steps: Step S1: Collect human skeleton motion data under multiple target clothing categories and standardize them; extract music and audio features synchronized with the human skeleton motion data; construct time-series training sample pairs using sliding window technology to obtain a multimodal dance dataset containing clothing attributes. Step S2: Construct a conditional sequence variational autoencoder that couples a skeletal graph attention encoder, a spatiotemporal clothing constraint graph network, an audio context encoder, and a skeletal graph attention decoder. Step S3: Construct a composite loss function that includes multidimensional motion reconstruction loss, KL divergence regularization loss, and clothing dynamics consistency contrast loss; Step S4: Through two stages, namely basic motion space preheating and joint dynamics fine-tuning, the conditional sequence variational autoencoder is trained using the multimodal dance dataset and the composite loss function to ensure that the conditional sequence variational autoencoder can correctly learn the dynamic constraints and obtain a trained dance generation model.

[0010] Optionally, the time-series training sample pair includes a historical action sequence window under the target clothing category, an audio context window covering the specified area before and after the current prediction frame, and a target action output window for the current prediction frame.

[0011] Optionally, the skeletal graph attention encoder is used to extract a latent mean vector of the underlying action intent from the input historical action sequence window, without constraint on the target clothing category. Sum of logarithmic variance vector ; The spatiotemporal clothing constraint graph network uses historical action sequence windows and their corresponding target clothing category embedding vectors. As input, output the potential dynamic offset vector. ; Step S2 involves fusing the skeletal graph attention encoder with the spatiotemporal clothing constraint graph network to perform latent spatial constraint fusion, and then using the latent distribution mean vector constrained by the target clothing category. Represented as: ; Step S2 further obtains the latent variable vector through reparameterized sampling according to the following formula. : ; in, Let be the log-variance vector constrained by the target clothing category and , For exponential operations, For Hadamard product operations, This is a noise vector obtained by sampling from a standard normal distribution.

[0012] Optionally, the spatiotemporal clothing constraint graph network performs spatiotemporal feature transformation through several alternating clothing modulation graph convolutional layers and dilated temporal convolutional layers; The feature update formula for the convolutional layer of the clothing modulation map is: ; in, For the first The node feature matrix of the layer, for Activation function This is the basic adjacency matrix of the human skeleton. For the reason A specific adjacency adjustment mask generated by a multilayer perceptron. For the first The input feature matrix of the layer, For the first Layer-learnable weight matrix; The dilated temporal convolutional layer employs a causal one-dimensional convolutional kernel; The spatiotemporal clothing constraint graph network further performs global pooling and fully connected mapping on the output of the last layer of the dilated temporal convolutional layer, ultimately outputting a latent dynamic offset vector. .

[0013] Optionally, the audio context encoder is used to extract high-level music feature vectors of the input audio context window using a five-layer one-dimensional deep convolutional neural network. .

[0014] Optionally, the skeleton map attention decoder will spliced ​​vector As input, it is first mapped back to the initial spatial feature dimension through a fully connected layer, and then deconstructed and reconstructed layer by layer through a skeleton graph attention layer, finally outputting the predicted action pose matrix for the next frame.

[0015] Optionally, the composite loss function is expressed as: ; in, For the total loss function, For multi-dimensional motion reconstruction loss, The loss is the KL divergence regularization loss. For the contrast loss of clothing dynamics consistency, The weights are the KL divergence regularization loss weights. Weighting of the loss for clothing dynamic consistency; Multidimensional motion reconstruction loss Represented as: ; in, Represents the L1 norm; Represents the L2 norm; This represents the actual pose matrix for the next frame. The predicted pose matrix for the next frame; This represents the positive kinematics function, used to convert the motion posture matrix into the corresponding three-dimensional space coordinates of the joints; Represents the actual joint angular velocity. This indicates the predicted joint angular velocity; The loss weights for the action items, The loss weights for the position terms. The loss weight for the velocity term; KL divergence regularization loss Represented as: ; in, For the dimension of latent variables; Indexing for latent variable dimensions; for The first in One portion, for The first in One component; Clothing dynamics consistency contrast loss Represented as: ; in, The preset distance boundary threshold, This represents the potential dynamic offset vector when the target clothing category is correct. This represents the potential dynamic offset vector when the target clothing category is incorrect. Indicates taking 0 and The larger of the values.

[0016] Optionally, step S4 includes: In the first stage, basic motion space preheating is performed, the parameters of the spatiotemporal clothing constraint graph network and the embedding vectors of all target clothing categories are frozen, and the output of the spatiotemporal clothing constraint graph network is forced to zero. Only the skeletal graph attention encoder, the audio context encoder and the skeletal graph attention decoder are trained. The KL divergence regularization loss weight is linearly increased from 0 to 0.01, so that the conditional sequence variational autoencoder learns the basic alignment relationship between human motion manifold and musical rhythm. In the second stage, joint dynamic fine-tuning is performed, all parameters of the conditional sequence variational autoencoder are unfrozen, and joint optimization is performed using the complete composite loss function. The upper limit of the KL divergence regularization loss weight is increased to 0.1, enabling the conditional sequence variational autoencoder to learn dynamic constraints.

[0017] Another aspect of the present invention provides a dance generation method, the dance generation method comprising: Receives the input full-length music audio sequence and the user-specified target clothing category; Using a trained dance generation model, a complete dance sequence is generated frame-by-frame through autoregression based on the input full-length music audio sequence and the target clothing category specified by the user; wherein, the trained dance generation model is trained using the training method of the dance generation model described above.

[0018] Optionally, the step of using a trained dance generation model to generate a complete dance sequence frame-by-frame through autoregression, based on the input full-length music audio sequence and the user-specified target clothing category, includes: Load all network parameters of the pre-trained dance generation model and extract the embedding vector of the target clothing category specified by the user. Initialize a historical action buffer queue filled with standard standing posture data; For each time step : The audio context window corresponding to the current moment is extracted from the full-length music audio sequence. The extracted audio context window is then input into the audio context encoder in the trained dance generation model to obtain the music features at the current moment. ; The current historical action sequence in the historical action buffer queue is input into the skeletal map attention encoder of the trained dance generation model to calculate the mean vector of the basic action intent at the current moment. Simultaneously, the embedding vectors of the most recent frames of actions in the historical action buffer queue and the user-specified target clothing category are also input. Input the spatiotemporal clothing constraint graph network from the trained dance generation model to obtain the dynamic offset vector of the target clothing category specified by the user at the current time. The sum of the mean vector of the basic action intent at the current moment and the dynamic offset vector of the target clothing category specified by the user at the current moment is used as the final latent mean vector constrained by the target clothing category. This final latent mean vector is then directly used as the latent variable vector at the current moment through deterministic sampling. ; Will spliced ​​vector Input the skeletal attention decoder into the trained dance generation model to obtain the predicted action pose for the next frame. Add the predicted action pose for the next frame to the end of the historical action buffer queue and delete the earliest frame data in the historical action buffer queue to complete the update of the historical action buffer queue.

[0019] Compared to existing technologies, this invention introduces an innovative spatiotemporal clothing constraint graph network into the latent space of a variational autoencoder. This allows for the explicit modeling and application of nonlinear physical constraints and inertial effects of complex clothing on human motion. Through the spatiotemporal clothing constraint graph network, particularly its clothing modulation graph convolutional layers and dilated temporal convolutional layers, the model can accurately predict changes in joint linkage and motion lag effects caused by different target clothing categories. By combining a two-stage learning training strategy with clothing dynamics consistency contrastive loss, the basic motion intention is effectively separated from specific clothing dynamics characteristics. The dance sequence generated using the trained dance generation model not only matches the musical rhythm but also realistically presents the physical dynamics of the specified clothing, significantly improving the realism and expressiveness of dance generation under complex clothing. Attached Figure Description

[0020] One or more embodiments are illustrated by way of example with the corresponding pictures in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0021] Figure 1 A flowchart illustrating a training method for a dance generation model according to an embodiment of the present invention; Figure 2 A schematic diagram of the overall structure of a conditional sequence variational autoencoder provided in another embodiment of the present invention; Figure 3 A schematic diagram of the structure of a spatiotemporal clothing constraint graph network provided for another embodiment of the present invention; Figure 4 An autoregressive generation inference flowchart for dance movements under a specific target clothing category, provided as another embodiment of the present invention; Figure 5 This is a schematic diagram comparing the effects of generating dance movements under different clothing categories, as provided in another embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the various embodiments of the present invention to facilitate a better understanding of the invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and with various variations and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0023] One embodiment of the present invention provides a training method for a dance generation model. This method constructs a deeply coupled neural network architecture and innovatively introduces a dynamic constraint module based on a spatiotemporal graph neural network into the latent space of a variational autoencoder (VAE). According to the specified clothing type, it explicitly predicts and applies specific dynamic deviations to the movement intention, forcing the generation network to output a dance sequence that conforms to physical laws and has specific clothing dynamism. This solves the technical defects of existing general dance generation models that cannot perceive and simulate the physical constraints and inertial effects of complex clothing (such as long sleeves, large skirts, heavy armor, etc.) on human movement.

[0024] like Figure 1 As shown, the training method for the dance generation model provided in this embodiment includes the following steps S1 to S4.

[0025] Step S1: Construct a multimodal dance dataset containing clothing attributes: Collect human skeletal motion data under various target clothing categories and standardize them, extract music and audio features synchronized with the human skeletal motion data, and use the sliding window technique to construct time-series training sample pairs to obtain a multimodal dance dataset containing clothing attributes.

[0026] Specifically, step S1 mainly involves constructing a high-precision, multimodal, standardized dataset for training deep neural network models to ensure that the models can learn the relationships between actions, music, and clothing.

[0027] For example, step S1 may include steps 1.1 to 1.4.

[0028] Step 1.1: Human skeletal movement data acquisition and standardization: A simplified human skeletal topology diagram is created, which includes the elements involved in dance movements. Key joint nodes (such as pelvis, spine, shoulder, elbow, wrist, hip, knee, ankle, etc.) and their connections are analyzed. Optical motion capture data of professional dancers performing in various predetermined target costume categories is collected as human skeletal motion data for the corresponding target costume categories. The collected human skeletal motion data is then solved and converted into a local rotation matrix representation of the skeletal joints, where the current moment... Human skeletal motion data Represented as a dimension The flattened matrix serves as the motion pose matrix for the current frame, containing... Key nodes at the current moment The dimension is The rotation matrix. For example, the sampling rate of human skeletal motion data can be set to 60Hz.

[0029] Step 1.2: Music audio feature extraction: Regarding human skeletal motion data Synchronized music audio was sampled at a rate of 44.1 kHz. A Short-Time Fourier Transform (STFT) was performed using a Hanning window with a window length of 2048 points and a step length of 1024 points. The first 20 dimensions of Mel-Frequency Cepstral Coefficients (MFCCs) and 12 dimensions of chroma map features were extracted from the transform result. The extracted features were then concatenated to obtain the corresponding 32-dimensional audio feature vector. This allows for the extraction of music audio features.

[0030] Step 1.3: Embedding Clothing Attribute Parameters: Define the number of categories in the target clothing category as follows: In this embodiment, let This includes eight types of clothing with significant physical differences, such as tight-fitting sportswear, Hanfu-style wide-sleeved shirts, Tibetan robes, heavy formal dresses, metallic armor, casual wear, ballet tutus, and heavy down jackets. For each target clothing category... Initialize a dimension as Learnable embedding vectors In the subsequent model training process, The system will automatically learn and encode the potential physical properties of the target clothing category (such as uneven mass distribution, degree of joint movement restriction, damping coefficient, etc.).

[0031] Step 1.4: Time Series Data Slicing and Window Construction: A sliding window technique is used to segment the continuous long sequence data, including the action pose matrices obtained in step 1.1 and the audio feature vectors obtained in step 1.2, into fixed-length temporal training sample pairs. Each temporal training sample pair includes a historical action sequence window under the target clothing category, an audio context window covering the specified regions before and after the current prediction frame, and a target action output window for the current prediction frame. The historical action sequence window serves as the action sequence input window, and its window length... Set to 64 frames to provide historical motion context, specifically including a motion pose matrix at 64 time points. The window length of the audio context window covering the specified regions before and after the current prediction frame. The frame size is set to 128 frames, including audio feature vectors covering a total of 128 time points before and after the current prediction time, to provide sufficient musical rhythm information. The target action output window of the current prediction frame is used as the target output window, and its window length is... Set to 1 frame, including the action pose matrix for the next moment of the current prediction frame, with the aim of enabling the model to predict the action in the next frame.

[0032] Step S2: Construct a generative network architecture that integrates spatiotemporal clothing dynamics constraints: Construct a conditional sequence variational autoencoder that is coupled together with a skeletal graph attention encoder, a spatiotemporal clothing constraint graph network, an audio context encoder, and a skeletal graph attention decoder.

[0033] Specifically, combined Figure 2 The generative network constructed in this implementation is a complex Conditional Sequence VAE, whose core latent variables have a dimensionality of [missing information]. Set as It is mainly composed of four deep sub-networks coupled together: skeletal graph attention encoder Spacetime Clothing Constraint Diagram Network Audio context encoder and skeleton graph attention decoder Among them, the spatiotemporal clothing constraint graph network This belongs to the core innovation module. Step S2 mainly includes the following steps 2.1 to 2.4.

[0034] Step 2.1: Construct a skeletal graph attention encoder : Skeletal graph attention encoders are used to extract the latent mean vector of underlying action intent from an input window of historical action sequences, without constraints on the target clothing category. Sum of logarithmic variance vector .

[0035] Specifically, the skeletal graph attention encoder Designed to extract historical action sequence window from input Extracting pure human motion intention features without specific clothing constraints. Skeletal graph attention encoder. The main body consists of three stacked Skeletal Graph Attention Layers (SGAT). Each SGAT layer uses the human skeleton topology established in step 1.1 as the base graph structure, dynamically calculating the weights between adjacent joints through a multi-head self-attention mechanism to aggregate spatial features. The spatiotemporal features output by the last SGAT layer are then subjected to global average pooling in the time dimension and mapped through two parallel fully connected layers (Linear Layers) to output parameters representing the latent multidimensional Gaussian distribution of the basic action intent: the mean vector. Sum of logarithmic variance vector .

[0036] Step 2.2: Constructing a spatiotemporal clothing constraint graph network : Specifically, the spatiotemporal clothing constraint graph network is the core innovative module of this invention, used to accurately simulate the dynamic correction effect of specific clothing on human movement intentions. The spatiotemporal clothing constraint graph network is a deterministic prediction network, utilizing historical action sequence windows, particularly historical action sequence windows... The most recent 32 frames of historical action sequence and the corresponding embedding vector of the target clothing category As input, output the potential dynamic offset vector. In order to obtain constraints on clothing dynamics.

[0037] The spatiotemporal clothing constraint graph network comprises several alternating convolutional layers of clothing modulation graphs and dilated temporal convolutional layers, which perform spatiotemporal feature transformation. For example, it combines... Figure 3 The spatiotemporal clothing constraint graph network can include three stacked hierarchical blocks, namely hierarchical block 1, hierarchical block 2, and hierarchical block 3. Each hierarchical block includes a clothing modulation graph convolutional layer and a dilated temporal convolutional layer.

[0038] Garment-Modulated GCNs aim to simulate how clothing alters the transmission of forces between joints. The Garment-Modulated GCN utilizes the embedding vector of the target garment category as input. An adjacency matrix adjustment mask specific to the target clothing category is generated using a multilayer perceptron (MLP) with two hidden layers. The adjacency matrix adjustment mask is related to the basic adjacency matrix of the human skeleton. The Hadamard product (element-wise product) operation is performed to dynamically change the strength weights of information transfer between joints in graph convolution operations, simulating the enhancing (e.g., stiff fabric connection) or inhibiting (e.g., loose fabric cushioning) effects of different clothing on joint linkage. The basic adjacency matrix of the human skeleton is used as a reference. The structure is constructed using the human skeletal topology map established in step 1.1, including... There are 1 element, each representing the connection between the joint in its row and the joint in its column. If the joint in the row and the joint in the column of an element are connected, the value of that element is 1; otherwise, the value of that element is 0.

[0039] The feature update formula for the convolutional layer of the clothing modulation image is: ; in, For the first The node feature matrix of the layer, for Activation function This is the basic adjacency matrix of the human skeleton. For the reason A specific adjacency adjustment mask generated by a multilayer perceptron. For the first The input feature matrix of the layer, For the first A layer-learnable weight matrix.

[0040] Dilated Temporal Convolutional Layer (TCN): Following each garment modulation map convolutional layer, a dilated temporal convolutional layer is added to handle temporal dependencies. The dilated temporal convolutional layer uses a causal 1D convolution kernel. The dilation rate of the dilated temporal convolutional layers in block 1, block 2, and block 3 is set to 1, 2, and 4 respectively, to exponentially expand the temporal receptive field, thereby effectively capturing the long-term lag effect caused by the inertia of the garment in the historical action sequence.

[0041] The spatiotemporal clothing constraint graph network also performs global pooling and fully connected mapping on the output of the last dilated temporal convolutional layer, ultimately outputting a dimension of Potential dynamic offset vector .

[0042] Step 2.3: Fusion and reparameterization of latent spatial constraints: The skeletal graph attention encoder and the spatiotemporal clothing constraint graph network are fused in latent space constraints. Specifically, the basic motion intent obtained from the skeletal graph attention encoder in step 2.1 and the clothing dynamic constraints obtained from the spatiotemporal clothing constraint graph network in step 2.2 are linearly fused in the latent space, resulting in the final latent distribution mean vector constrained by the target clothing category. Represented as: .

[0043] The final log-variance vector constrained by the target clothing category We directly use the log-variance vector of the basic action intention obtained in step 2.1, that is... .

[0044] Subsequently, according to the following formula, the latent variable vector is sampled using the reparameterization trick. : ; in, Let be the log-variance vector constrained by the target clothing category and , For exponential operations, For Hadamard product operations, To obtain from the standard normal distribution The noise vector obtained by sampling.

[0045] Step 2.4: Construct the audio context encoder With skeleton graph attention decoder : Audio Context Encoder Audio context window used to extract input using a five-layer one-dimensional deep convolutional neural network High-level music feature vectors .

[0046] Skeletal Graph Attention Decoder Structural and Skeletal Attention Encoder The structure is mirror-symmetric. Skeletal map attention decoder Will spliced ​​vector As input, the data is first mapped back to the initial spatial feature dimension through a fully connected layer. Then, it undergoes layer-by-layer deconstruction and spatial structure reconstruction through a three-layer skeleton graph attention layer, ultimately outputting the predicted action pose matrix for the next frame. .

[0047] Step S3: Define the composite loss function: Construct a composite loss function that includes multidimensional motion reconstruction loss, KL divergence regularization loss, and clothing dynamics consistency contrast loss.

[0048] Specifically, in order to collaboratively train the various network modules in the Conditional Sequence Variational Autoencoder (CSE) and ensure that the actions generated by the CSE accurately reproduce the target while conforming to physical constraints and distribution characteristics, this implementation defines a composite loss function that includes multiple constraints. .

[0049] 3.1: Loss in Multidimensional Motion Reconstruction : Multidimensional motion reconstruction loss Used to calculate the predicted action pose matrix for the next frame. With the actual next frame motion pose matrix The combined error across multiple dimensions is expressed as: ; in, Represents the L1 norm; Represents the L2 norm; This represents the actual pose matrix for the next frame. The predicted pose matrix for the next frame; This represents the positive kinematics function, used to convert the motion posture matrix into the corresponding three-dimensional space coordinates of the joints; Represents the actual joint angular velocity. This indicates the predicted joint angular velocity; The loss weight for the action item can be set to... ; The loss weight for the location item can be set to... ; The loss weight for the velocity term can be set to... The joint angular velocity can be obtained using existing calculation methods, such as by calculating the difference between adjacent frames.

[0050] 3.2: KL divergence regularization loss : KL divergence regularization loss This is used to calculate the Kullback-Leibler divergence between the final latent posterior distribution and the standard normal prior distribution, to normalize the structure distribution of the latent space. The KL divergence regularization loss is used. Represented as: ; in, For the dimension of latent variables; Indexing for latent variable dimensions; for The first in One portion, for The first in Each component.

[0051] 3.3 Comparison of Losses in Clothing Dynamics Consistency : Clothing dynamics consistency contrast loss It is specifically designed for innovative modules, i.e., spatiotemporal clothing constraint graph networks. The designed loss term aims to force the network to learn to distinguish specific dynamic pattern differences arising from different target clothing categories. This is applied to historical action sequences of the same training sample. Input the embedding vector of the correct target clothing category for each. Embedded vectors of randomly sampled incorrect clothing categories The requirement is that the Euclidean distance between the two dynamic offset vectors output by the network is greater than a preset boundary threshold.

[0052] Clothing dynamics consistency contrast loss Represented as: ; in, The preset distance boundary threshold, This represents the potential dynamic offset vector when the target clothing category is correct. This represents the potential dynamic offset vector when the target clothing category is incorrect. Indicates taking 0 and The larger of the values.

[0053] Composite loss function, also known as total loss function Loss of multidimensional motion reconstruction KL divergence regularization loss Clothing dynamics consistency comparison loss The weighted sum is expressed as: ; in, The weights for the KL divergence regularization loss are dynamically adjusted during training. The weight for the contrast loss of clothing dynamic consistency can be set to... .

[0054] Step S4: Implement a two-stage course learning and training strategy: Through two stages, namely, basic motion space warm-up and joint dynamics fine-tuning, the conditional sequence variational autoencoder is trained using the multimodal dance dataset combined with the composite loss function to ensure that the conditional sequence variational autoencoder can correctly learn the dynamic constraints and obtain a trained dance generation model.

[0055] Specifically, the AdamW optimizer was used to update the network parameters, with a batch size set to 128. This was to ensure the integrity of the core innovative module, the spatiotemporal clothing constraint graph network. It can correctly learn dynamic constraints rather than noise, and adopts a two-stage course learning and training strategy.

[0056] For example, step S4 includes steps 4.1 and 4.2.

[0057] Step 4.1: Phase One: Basic Motion Space Warm-up: The first phase consists of training epochs 0-60. During this phase, basic motion space is warmed up, and the spatiotemporal clothing constraint graph network is frozen. The embedding vectors of all target clothing categories are... The values ​​are as follows: time The parameters are not updated with gradients, and the spatiotemporal clothing constraint graph network is used. The output is the potential dynamic offset vector. Forced to zero, training only the skeletal graph attention encoder. Audio context encoder and skeleton graph attention decoder These three subnetworks; KL divergence regularization loss weights An annealing strategy is adopted, which linearly increases from 0 to 0.01, enabling the conditional sequence variational autoencoder to learn the alignment relationship between the basic human motion manifold and musical rhythm, thus establishing a good basic potential motion space.

[0058] Step 4.2: Phase Two: Fine-tuning of Joint Dynamics Epochs 61-300 are used as the second stage, during which joint dynamics fine-tuning is performed to unfreeze all parameters of the conditional sequence variational autoencoder, including the spatiotemporal clothing constraint graph network. Network parameters and The values ​​are as follows: time Using the complete composite loss function Joint optimization is performed, with KL divergence regularization loss weights. The upper limit is increased to 0.1, enabling the conditional sequence variational autoencoder to learn dynamic constraints. The initial learning rate is set to... The learning rate is decayed to 0.2 times the current learning rate in rounds 150 and 250, respectively. When the total loss on the validation set composed of partial temporal training sample pairs in the multimodal dance dataset containing clothing attributes does not decrease within 20 consecutive epochs, the model is considered to have converged, training is stopped, and the network parameters of the conditional sequence variational autoencoder at this point are saved as the optimal model parameters.

[0059] Another embodiment of the present invention provides a dance generation method, the dance generation method comprising: Receives the input full-length music audio sequence and the user-specified target clothing category; Using a trained dance generation model, a complete dance sequence is generated frame-by-frame through autoregression based on the input full-length music audio sequence and the target clothing category specified by the user. The trained dance generation model is obtained by training the dance generation model as described in the above-described implementation method.

[0060] For example, using a trained dance generation model, a complete dance sequence is generated frame-by-frame through autoregression based on the input full-length music audio sequence and the user-specified target clothing category. This includes steps 5.1 and 5.2, which are described below in conjunction with... Figure 4 Steps 5.1 and 5.2 will be explained in detail.

[0061] Step 5.1: Initialization Phase: Load all network parameters of the pre-trained dance generation model, i.e., all network parameters of the conditional sequence variational autoencoder at the end of training, and extract the embedding vector of the target clothing category specified by the user. Initialize a string of length. The historical motion buffer queue, filled with standard T-Pose standing posture data, is used to initiate the autoregressive process.

[0062] Step 5.2: Frame-by-frame autoregressive loop generation: Enter the frame-by-frame generation loop, from Until the music ends, for each time step : Step 1: Audio Feature Extraction: Extract the current moment from the full-length music audio sequence. Corresponding audio context window The extracted audio context window Input the audio context encoder into the trained dance generation model To obtain the musical characteristics at the current moment ; Step 2: Basic Intent Encoding: Input the current historical action sequence from the historical action buffer queue into the skeletal map attention encoder of the trained dance generation model. The mean vector of the basic action intention at the current moment is calculated. ; Step 3: Dynamic Constraint Prediction: Simultaneously, several frames from the historical action buffer queue that are closest to the current moment are selected, such as... Embedding vectors of frame action and user-specified target clothing category Input the spatiotemporal costume constraint graph network from the trained dance generation model This yields the dynamic offset vector of the target clothing category specified by the user at the current moment. ; Step 4: Latent Space Fusion: Combine the mean vector of the basic action intent at the current moment. The dynamic offset vector of the target clothing category specified by the user at the current moment The sum of these values ​​serves as the final latent mean vector, constrained by the target clothing category. ,Right now Furthermore, to ensure the stability and consistency of the generated actions, no noise is introduced during the inference phase. The final potential mean vector is obtained through deterministic sampling. Directly used as the latent variable vector at the current moment ,Right now ; Step 5: Decode and generate action: spliced ​​vector Input the skeletal attention decoder into the trained dance generation model To obtain the predicted pose of the next frame. ; Step 6: Update the buffer queue: Add the predicted motion pose for the next frame. Add the data to the end of the historical action buffer queue, delete the oldest frame in the historical action buffer queue to squeeze out the oldest frame, complete the update of the historical action buffer queue, and advance to the next time step. Repeat steps 1 to 6 until the entire music ends, completing the generation of the entire dance and ultimately obtaining a complete dance sequence that conforms to the physical constraints of the target clothing category.

[0063] To verify the effectiveness of the trained dance generation model obtained by using the training method provided in this invention, the same music segment was used as a full-length music audio sequence and combined with eight target clothing categories, including tight-fitting sportswear, ordinary casual wear, Hanfu wide-sleeved shirts, Tibetan robes, heavy formal dresses, ballet tutus, metal armor, and heavy down jackets. Each combination was then input into the trained dance generation model to obtain the corresponding skeletal sequence frames as the corresponding dance action sequences. Figure 5 The diagram illustrates the motion characteristics corresponding to each clothing category. A comparative analysis of these motion characteristics using clothing dynamics constraints reveals the following: In terms of joint freedom of movement, the order is: tight-fitting sportswear > casual wear > ballet tutu > Hanfu wide-sleeved shirt > heavy down jacket > Tibetan robe > heavy formal dress > metal armor; in terms of motion inertia, the order is: metal armor > Hanfu wide-sleeved shirt > formal dress > Tibetan robe > heavy down jacket > casual wear > ballet tutu > tight-fitting sportswear. Therefore, it can be concluded that the training method for the dance generation model provided by this invention can utilize a spatiotemporal clothing constraint graph network. Successfully captured and reproduced the differentiated physical constraints imposed on human movement by different clothing, and the dance movements generated using the trained dance generation model conform to the real physical characteristics of various types of clothing.

[0064] This invention introduces an innovative spatiotemporal clothing constraint graph network into the latent space of a variational autoencoder, enabling explicit modeling and imposition of nonlinear physical constraints and inertial effects of complex clothing on human motion. Through the spatiotemporal clothing constraint graph network, particularly its clothing modulation graph convolutional layer and dilated temporal convolutional layer, the model accurately predicts changes in joint linkage and motion lag effects caused by different target clothing categories. By combining a two-stage learning training strategy with clothing dynamics consistency contrastive loss, the basic motion intention is effectively separated from specific clothing dynamics characteristics. The dance motion sequences generated using the trained dance generation model not only match the musical rhythm but also realistically present the physical dynamics of the specified clothing, significantly improving the realism and expressiveness of dance generation under complex clothing.

[0065] Those skilled in the art will understand that the above embodiments are specific implementations of the present invention, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A training method for a dance generation model, characterized in that, The training method for the dance generation model includes the following steps: Step S1: Collect human skeleton motion data under multiple target clothing categories and standardize them; extract music and audio features synchronized with the human skeleton motion data; construct time-series training sample pairs using sliding window technology to obtain a multimodal dance dataset containing clothing attributes. Step S2: Construct a conditional sequence variational autoencoder that couples a skeletal graph attention encoder, a spatiotemporal clothing constraint graph network, an audio context encoder, and a skeletal graph attention decoder. Step S3: Construct a composite loss function that includes multidimensional motion reconstruction loss, KL divergence regularization loss, and clothing dynamics consistency contrast loss; Step S4: Through two stages, namely basic motion space preheating and joint dynamics fine-tuning, the conditional sequence variational autoencoder is trained using the multimodal dance dataset and the composite loss function to ensure that the conditional sequence variational autoencoder can correctly learn the dynamic constraints and obtain a trained dance generation model.

2. The training method for the dance generation model according to claim 1, characterized in that, The time-series training sample pair includes a historical action sequence window under the target clothing category, an audio context window covering the specified area before and after the current prediction frame, and a target action output window for the current prediction frame.

3. The training method for the dance generation model according to claim 2, characterized in that, The skeletal graph attention encoder is used to extract a latent mean vector of basic action intent from an input historical action sequence window, without constraint on the target clothing category. Sum of logarithmic variance vector ; The spatiotemporal clothing constraint graph network uses historical action sequence windows and their corresponding target clothing category embedding vectors. As input, output the potential dynamic offset vector. ; Step S2 involves fusing the skeletal graph attention encoder with the spatiotemporal clothing constraint graph network to perform latent spatial constraint fusion, and then using the latent distribution mean vector constrained by the target clothing category. Represented as: ； Step S2 further obtains the latent variable vector through reparameterized sampling according to the following formula. : ； in, Let be the log-variance vector constrained by the target clothing category and , For exponential operations, For Hadamard product operations, This is a noise vector obtained by sampling from a standard normal distribution.

4. The training method for the dance generation model according to claim 3, characterized in that, The spatiotemporal clothing constraint graph network performs spatiotemporal feature transformation through several alternating clothing modulation graph convolutional layers and dilated temporal convolutional layers; The feature update formula for the convolutional layer of the clothing modulation map is: ； in, For the first The node feature matrix of the layer, for Activation function This is the basic adjacency matrix of the human skeleton. For the reason A specific adjacency adjustment mask generated by a multilayer perceptron. For the first The input feature matrix of the layer, For the first Layer-learnable weight matrix; The dilated temporal convolutional layer employs a causal one-dimensional convolutional kernel; The spatiotemporal clothing constraint graph network further performs global pooling and fully connected mapping on the output of the last layer of the dilated temporal convolutional layer, ultimately outputting a latent dynamic offset vector. .

5. The training method for the dance generation model according to claim 4, characterized in that, The audio context encoder is used to extract high-level music feature vectors from the input audio context window using a five-layer one-dimensional deep convolutional neural network. .

6. The training method for the dance generation model according to claim 5, characterized in that, The skeleton graph attention decoder will spliced ​​vector As input, it is first mapped back to the initial spatial feature dimension through a fully connected layer, and then deconstructed and reconstructed layer by layer through a skeleton graph attention layer, finally outputting the predicted action pose matrix for the next frame.

7. The training method for the dance generation model according to claim 6, characterized in that, The composite loss function is expressed as follows: ； in, For the total loss function, For multi-dimensional motion reconstruction loss, The loss is the KL divergence regularization loss. For the contrast loss of clothing dynamics consistency, The weights are the KL divergence regularization loss weights. Weighting of the loss for clothing dynamic consistency; Multidimensional motion reconstruction loss Represented as: ； in, Represents the L1 norm; Represents the L2 norm; This represents the actual pose matrix for the next frame. The predicted pose matrix for the next frame; This represents the positive kinematics function, used to convert the motion posture matrix into the corresponding three-dimensional space coordinates of the joints; Represents the actual joint angular velocity. This indicates the predicted joint angular velocity; The loss weights for the action items, The loss weights for the position terms. The loss weight for the velocity term; KL divergence regularization loss Represented as: ； in, For the dimension of latent variables; Indexing for latent variable dimensions; for The first in One portion, for The first in One component; Clothing dynamics consistency contrast loss Represented as: ； in, The preset distance boundary threshold, This represents the potential dynamic offset vector when the target clothing category is correct. This represents the potential dynamic offset vector when the target clothing category is incorrect. Indicates taking 0 and The larger of the values.

8. The training method for the dance generation model according to any one of claims 1 to 7, characterized in that, Step S4 includes: In the first stage, basic motion space preheating is performed, the parameters of the spatiotemporal clothing constraint graph network and the embedding vectors of all target clothing categories are frozen, and the output of the spatiotemporal clothing constraint graph network is forced to zero. Only the skeletal graph attention encoder, the audio context encoder and the skeletal graph attention decoder are trained. The KL divergence regularization loss weight is linearly increased from 0 to 0.01, so that the conditional sequence variational autoencoder learns the basic alignment relationship between human motion manifold and musical rhythm. In the second stage, joint dynamic fine-tuning is performed, all parameters of the conditional sequence variational autoencoder are unfrozen, and joint optimization is performed using the complete composite loss function. The upper limit of the KL divergence regularization loss weight is increased to 0.1, enabling the conditional sequence variational autoencoder to learn dynamic constraints.

9. A dance generation method, characterized in that, The dance generation method includes: Receives the input full-length music audio sequence and the user-specified target clothing category; Using a trained dance generation model, a complete dance sequence is generated frame-by-frame through autoregression based on the input full-length music audio sequence and the target clothing category specified by the user; wherein the trained dance generation model is trained using the training method of the dance generation model according to any one of claims 1 to 8.

10. The dance generation method according to claim 9, characterized in that, The process utilizes a trained dance generation model to generate a complete dance sequence based on the input full-length music audio sequence and the user-specified target clothing category, through frame-by-frame autoregression. This includes: Load all network parameters of the pre-trained dance generation model and extract the embedding vector of the target clothing category specified by the user. Initialize a historical action buffer queue filled with standard standing posture data; For each time step : The audio context window corresponding to the current moment is extracted from the full-length music audio sequence. The extracted audio context window is then input into the audio context encoder in the trained dance generation model to obtain the music features at the current moment. ; The current historical action sequence in the historical action buffer queue is input into the skeletal map attention encoder of the trained dance generation model to calculate the mean vector of the basic action intent at the current moment. Simultaneously, the embedding vectors of the most recent frames of actions in the historical action buffer queue and the user-specified target clothing category are also input. Input the spatiotemporal clothing constraint graph network from the trained dance generation model to obtain the dynamic offset vector of the target clothing category specified by the user at the current time. The sum of the mean vector of the basic action intent at the current moment and the dynamic offset vector of the target clothing category specified by the user at the current moment is used as the final latent mean vector constrained by the target clothing category. This final latent mean vector is then directly used as the latent variable vector at the current moment through deterministic sampling. ; Will spliced ​​vector Input the skeletal attention decoder into the trained dance generation model to obtain the predicted action pose for the next frame. Add the predicted action pose for the next frame to the end of the historical action buffer queue and delete the earliest frame data in the historical action buffer queue to complete the update of the historical action buffer queue.

Images