Dance generation method and system based on deep learning model

Abstract

The application belongs to the technical field of image data processing, and discloses a dance generation method and system based on a deep learning model. The method comprises the following steps: constructing and preprocessing a multi-modal data set containing dance videos, 3D posture sequences and annotated emotional music; extracting visual time sequence features and cross-modal fusion with skeletal time sequence features to obtain a fusion feature vector; obtaining target video visual time sequence feature constraints according to input music emotion matching; taking the fusion feature vector, target visual feature constraints and music beat features as conditions, generating an initial dance posture sequence through a conditional diffusion generation model constructed based on a multi-target loss function; performing image conversion on the initial posture sequence, and optimizing through an image optimization model containing optical flow correction to generate a high-quality dance image sequence and synthesize a final dance video. The system comprises corresponding function modules. The application realizes end-to-end generation of a high-quality dance video with controllable style, smooth action and sound-picture coordination.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a dance generation method and system based on a deep learning model. Background Technology

[0002] With the explosive growth of the digital entertainment industry, the commercialization of virtual idols, and the widespread adoption of online dance instruction, AI-based dance generation technology is becoming a core demand driver in multiple fields. In the digital entertainment sector, film, games, and short video platforms need to quickly generate diverse dance content to match the visual style of different IPs. The virtual idol field requires dance movements to conform to the laws of human movement while also fitting the character's persona. In online dance instruction, users expect the system to generate "disassembled and adjustable" standard dance movement sequences to help learners master rhythm and posture details.

[0003] Existing dance generation technologies generally rely solely on skeletal sequences or single modal features, optimizing motion transitions only through differences in joint coordinates between adjacent frames. Furthermore, the coordination between music and dance is limited to beat timing alignment, failing to establish a mapping relationship between musical emotions and video visual features. This results in a low match between the style of the generated movements and the visual expression, easily leading to unnatural movement patterns such as sudden stops and sudden movements, and a mismatch between emotional expression and music. Summary of the Invention

[0004] The present invention aims to provide a dance generation method and system based on a deep learning model to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A dance generation method based on a deep learning model includes the following steps: S1. Construct a multimodal dataset containing dance style reference videos, corresponding 3D pose sequences, and music clips labeled with emotion tags. Preprocess the video data, pose data, and music data in the dataset to obtain preprocessed continuous video image frames, aligned and completed 3D pose sequences, and music features containing emotion features and temporal rhythm features. The preprocessing of the video data includes extracting continuous image frames and extracting human pose contour lines from them. S2. Based on the continuous video image frames obtained in step S1, visual temporal features are extracted using a video-pose feature fusion model. The visual temporal features include pose contour features and inter-frame motion features obtained from the continuous video image frames, as well as line rhythm features obtained from the extracted human pose contour lines. Simultaneously, based on the 3D pose sequence obtained in step S1, skeletal temporal features are extracted. The visual temporal features and skeletal temporal features are fused across modally to obtain a fused feature vector. S3. Based on the music emotion features obtained in step S1, match them using a music-video emotion matching model to obtain visual temporal feature constraints of the target video. S4. Using the fused feature vector obtained in step S2, the target video visual temporal feature constraints obtained in step S3, and the music beat temporal features obtained in step S1 as conditions, an initial dance posture sequence is generated through a conditional diffusion generation model, which is constructed based on a preset multi-objective loss function. S5. The initial dance posture sequence generated in step S4 is converted into an image to obtain an initial dance image sequence. Then, the initial dance image sequence is optimized by inter-frame motion correction using an image optimization model to generate a high-quality dance image sequence. S6. Output a high-quality dance video synthesized from the high-quality dance image sequence generated in step S5.

[0006] Preferably, in step S2, cross-modal fusion is achieved through a cross-attention mechanism.

[0007] Preferably, in step S3, the matching is achieved through a mapping relationship established by contrastive learning, wherein the mapping relationship is a mapping relationship from music emotional features to video visual temporal features.

[0008] Preferably, in step S4, the multi-objective loss function For attitude continuity loss Video visual temporal feature matching loss and motion smoothing loss The weighted sum is expressed as: in, , and These are the preset weighting coefficients.

[0009] Preferably, the motion smoothing loss The calculation method is the average variance of the joint motion velocities over three consecutive frames, expressed as: in, Let be the motion velocity of the i-th joint in the t-th frame. The number of key points. Total number of frames Let be the variance function, and the optimization objective of the action smoothing loss is to make .

[0010] Preferably, the attitude continuity loss The calculation method is to use the average L1 norm of the differences in coordinates of all attitude joints between adjacent frames, and its optimization objective is to make .

[0011] Preferably, the video visual temporal feature matching loss The calculation method is to use the cosine distance between the visual feature vector corresponding to the generated pose sequence and the visual temporal feature constraint vector of the target video, and the objective is to maximize the cosine similarity. .

[0012] Preferably, in step S5, the inter-frame motion correction optimization is achieved through an optical flow correction module, which employs an optical flow estimation network based on a recurrent architecture.

[0013] Preferably, the optical flow estimation network uses optical flow correction loss. The optical flow correction loss is used to optimize training for the target. This is a weighted sum of the L1 loss for optical flow prediction and the MSE loss for image reconstruction.

[0014] A dance generation system based on a deep learning model, used to implement the above method, includes: The data acquisition and preprocessing module is used to construct the multimodal dataset and preprocess the video data, posture data and music data in the dataset, and output the preprocessed continuous video image frames, 3D posture sequences and music features containing emotional features and temporal features; wherein, the preprocessing of video data includes extracting continuous image frames and extracting human posture contour lines from them. The video-pose feature fusion module is connected to the data acquisition and preprocessing module. It is used to receive preprocessed video continuous image frames and 3D pose sequences, and use the extracted human pose contour lines to run the video-pose feature fusion model to extract visual temporal features and skeletal temporal features. It then performs cross-modal fusion of the two and outputs the fused feature vector to the subsequent modules. The music-video emotion matching module is connected to the data acquisition and preprocessing module. It is used to receive the preprocessed music emotion features, run the music-video emotion matching model, match the input music, and output the visual temporal features of the target video to constrain subsequent modules. The conditional diffusion generation module, connected to the video-pose feature fusion module and the music-video emotion matching module, is used to run a conditional diffusion generation model constructed based on a multi-objective loss function, with the fused feature vector, the visual temporal feature constraints of the target video, and the temporal features of the music beat output by the data acquisition and preprocessing module as conditions, to generate and output the initial dance pose sequence to the subsequent modules; The image optimization and video synthesis module is connected to the conditional diffusion generation module. It is used to perform image-based conversion on the initial dance posture sequence to obtain the initial dance image sequence, and to perform inter-frame motion correction optimization on the initial dance image sequence through the image optimization model to generate a high-quality dance image sequence and synthesize a high-quality dance video. The results output and interaction module, connected to the image optimization and video synthesis module, is used to output a high-quality dance video synthesized from a high-quality dance image sequence.

[0015] The beneficial effects of this technical solution compared to existing technologies are as follows: (1) This technical solution uses the continuous image frame sequence of the dance reference video as the core style driving source, and combines the cross attention mechanism to deeply integrate the video visual temporal features and skeletal temporal features. It also establishes the mapping relationship between music emotion and video visual features through comparative learning. This solves the core defects of isolated dance generation modalities and superficial associations in the existing technology, ensuring that the generated dance movements are not only accurate at the skeletal movement level, but also accurately reproduce the dynamic charm of the target style in terms of visual perception. At the same time, it enables the visual expression of the dance and the emotional atmosphere of the music to achieve deep synergy, thereby generating a high-quality dance sequence with a distinctive style and integrated audio-visual elements.

[0016] (2) By employing a multi-objective collaborative loss function system, including motion smoothing loss, to train the conditional diffusion generation model, the problem of stiff motion transitions is fundamentally solved from the perspective of physical motion laws. The motion smoothing loss function constrains the variance of joint motion velocities in three consecutive frames, forcing smooth acceleration changes in the generated motion and avoiding unnatural motion patterns such as sudden stops and turns. Combined with the collaborative optimization of other loss functions (posture continuity, visual feature matching, and audio-visual timing alignment), the generated dance achieves optimal performance in terms of motion trajectory smoothness, style consistency, and rhythmic accuracy, ultimately outputting a dance posture sequence that is smooth, naturally connected, and visually comfortable.

[0017] (3) By constructing an image optimization model that integrates generative adversarial networks and optical flow correction modules, end-to-end image-level rendering and temporal smoothing are performed on the generated pose sequences. Stylized texture generation and detail enhancement are performed based on the target visual features, ensuring high quality and style consistency of single frames. The optical flow correction module eliminates image jitter and tearing by physically compensating for motion offset between adjacent frames, ensuring the ultimate smoothness of video playback. This avoids the cumbersome steps and quality loss of separate rendering and post-compositing in traditional solutions, and can directly output high-quality, ready-to-use dance videos, simplifying the technical process and improving the quality of the results. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system architecture diagram of the present invention; Detailed Implementation The present invention will now be described in further detail with reference to the accompanying drawings and embodiments: like Figures 1-2 The present invention relates to a dance generation method and system based on a deep learning model, including the system and the method.

[0019] like Figure 2 As shown, the system is built in a modular manner, with each module communicating through well-defined data interfaces and ultimately scheduled by a unified system controller. Specifically, it includes: Data acquisition and preprocessing module This module serves as the system's data entry point and standardization center, consisting of hardware acquisition units and software processing pipelines.

[0020] 1. Hardware acquisition unit: Video capture: A set of multiple Blackmagic Pocket Cinema Camera 6K Pro cameras with synchronized phase locking were used to surround and film the dancers at 60fps and 4K resolution to ensure no blind spots in the capture.

[0021] Motion capture: The OptiTrack Prime series optical motion capture system is used, along with 32 high-speed infrared cameras. Reflective markers are attached to key joints of the dancers to synchronously acquire high-precision three-dimensional spatial coordinates at a frequency of 120Hz, with a positioning error of less than 0.1 mm.

[0022] Audio capture: Lossless audio tracks were recorded using a Rode NTG5 shotgun microphone and a Focusrite Scarlett 18i20 sound card at a sampling rate of 96kHz / 24-bit.

[0023] Software processing pipeline: Deployed on a Linux server, it automates processing based on a series of standard image and signal processing algorithms and open-source libraries.

[0024] Video processing pipeline: Frame synchronization and extraction: First, timestamp alignment and viewpoint synthesis are performed on multi-camera video, and then time-series downsampling is performed to unify the frame rate to 30fps.

[0025] Spatial normalization: The spatial dimensions of each frame of the image are standardized. Specifically, a bilinear interpolation algorithm is used to uniformly scale the pixel size of each frame of the image to a fixed resolution of 256 pixels × 256 pixels to eliminate scale differences in the input data.

[0026] Contour contrast enhancement: To highlight the boundary information of human pose, adaptive histogram equalization is performed on each frame of the normalized image. Specifically, the Limit Contrast Adaptive Histogram Equalization (CLAHE) algorithm is adopted, where the contrast limit value is set to 2.0, and the histogram is calculated and equalized within an 8×8 local image block.

[0027] Human contour line extraction: Single-pixel wide edges representing the spatial structure of human posture are extracted from the contrast-enhanced image. Specifically, the Canny edge detection algorithm is used, and double threshold hysteresis processing of gradient magnitude is performed by setting a high threshold (150) and a low threshold (50), and finally generating a binary line image sequence containing only human posture contour information.

[0028] Attitude processing pipeline: Coordinate system normalization: Using the collected 3D coordinates of the pelvic joints as the origin of the spatial coordinate system, the 3D coordinates of all other joints (a total of 52, including finger details) relative to this origin are calculated to achieve spatial normalization of the posture data.

[0029] Trajectory smoothing and completion: First, a Savitzky-Golay filter (window length 11, polynomial order 3) is used to smooth the 3D motion trajectory of each joint to suppress high-frequency noise. For trajectory data loss caused by temporary occlusion of marker points, a cubic spline interpolation method is used for accurate completion to ensure the spatiotemporal continuity of the attitude sequence.

[0030] Music processing pipeline: Spectral feature extraction: After pre-emphasizing, framing (window length 2048 points, number of jumps 512 points) and adding a Hamming window to the audio signal, its 128-dimensional spectral graph is calculated as a representation of the music.

[0031] Emotion feature encoding: The Mel spectrogram is input into a convolutional neural network pre-trained on a dedicated music emotion dataset for classification. The network outputs an 8-dimensional probability distribution of emotion categories, and the category with the highest probability (such as "exhilarating", "joyful", "sad") is taken as the emotion label of the music segment and encoded as the corresponding one-hot vector.

[0032] Beat timing extraction: An adaptive beat detection algorithm based on Spectral Flux is used to extract the timestamp sequence of beat points in the music. Subsequently, this timestamp sequence is aligned with the timeline of the video frames to generate a binary beat indicator vector of the same length as the video frame sequence, where the frame position corresponding to the beat point is 1 and the non-beat point is 0.

[0033] 2. Video-pose feature fusion module The software entity of this module is a deep neural network, implemented using a "dual-stream encoding-cross-fusion" architecture.

[0034] Video encoded stream: Backbone Network and Feature Extraction: An I3D (Inflated 3DConvNet) model pre-trained on the Kinetics-400 dataset is used as the spatiotemporal feature extractor. The input is 16 consecutive frames of RGB image patches (256×256×3). The backbone network contains multiple stacked 3D convolutional and pooling layers to progressively extract and compress spatiotemporal information. The output of this network after the last downsampling stage and before the global pooling layer is extracted to obtain a feature map with rich spatiotemporal semantic information, with a spatial dimension of 7×7 and a channel dimension of 832.

[0035] Temporal dependency modeling: A lightweight temporal Transformer encoder module is added to the spatiotemporal feature sequence output by the backbone network. This module consists of two layers, each containing a 4-head multi-head self-attention mechanism to model long-range inter-frame dependencies and automatically assign higher attention weights to action transition or key pose frames.

[0036] Output: Global spatiotemporal average pooling is performed on the temporally encoded feature maps to obtain a 1024-dimensional global visual temporal feature vector for the video. .

[0037] Attitude encoding stream: Backbone Network: An improved spatiotemporal graph convolutional network is employed. Fifty-two joints and their natural physiological connections are constructed into a spatiotemporal graph structure, where nodes represent joints, spatial edges represent human skeleton connections, and temporal edges connect the same joint in adjacent frames. The network contains 10 ST-GCN basic blocks, each of which sequentially performs spatial graph convolution and temporal one-dimensional convolution to simultaneously uncover spatial relationships between joints and temporal motion patterns across frames.

[0038] Output: After the last ST-GCN block, a global graph pooling operation is performed, followed by a fully connected layer to obtain a 1024-dimensional attitude spatiotemporal feature vector. .

[0039] Cross-modal fusion unit: Fusion Mechanism: A gated cross-attention mechanism is used for deep fusion. First, and Self-reinforcement is performed through a self-attention layer. Then, the enhanced... As a query vector, to enhance As key and value vectors, cross-attention is calculated to obtain the fusion context from the video perspective. Symmetrically, with... For Query, For the Key and Value, calculate the fusion context from the pose perspective. Finally, input the two context vectors into a gated fusion layer (implemented by a fully connected layer and a sigmoid activation function), dynamically learn the weighted weights, and merge them to output the final 512-dimensional fusion feature vector. .

[0040] Training details: This module was trained on four NVIDIA A100 GPUs. Triple loss function was used for metric learning, and its expression is: in, It is the fusion feature of the anchor point sample (a dance segment). These are features of positive samples (another segment of the same dance). These are features of negative samples (different dances). For Euclidean distance, Set to 0.5. The optimizer uses Adam, with an initial learning rate of 3 × 10⁻⁵. -4 The learning rate was adjusted using a cosine annealing strategy, and the system was trained for 200 epochs. On the validation set, the average intra-class similarity of the fused features reached 0.91, while the average inter-class similarity decreased to 0.15.

[0041] 3. Music-Video Emotion Matching Module At the core of this module is an embedding network based on metric learning.

[0042] Model Architecture: This module contains a music encoder and a video encoder. The music encoder is a 3-layer one-dimensional convolutional neural network that takes a Mel spectrogram as input and outputs a 256-dimensional music embedding vector. The video encoder and the video coding stream in the video-pose feature fusion module share some of the underlying structure, but are topped with different fully connected layers, outputting a 256-dimensional video style embedding vector. .

[0043] Comparative learning and training: Data pair construction: Music clips and corresponding dance videos are extracted from the dataset as positive sample pairs. Negative samples are generated by randomly replacing music or video clips within the same training batch.

[0044] Loss function: Circle Loss is used for training. This loss function learns a more accurate metric space by assigning different optimization strengths to positive and negative sample pairs with different similarities. Its core is to optimize the similarity of positive sample pairs. Similarity with negative sample pairs This makes them approach their optimal values. and ( Set the scale factor. , margin .

[0045] Mapping table construction and application: After training, all training set dance videos are processed by a video encoder to obtain their corresponding mapping tables. This data is then associated with the video's musical mood tags (such as "exciting - street dance") and stored to form an efficient memory-mapped database (e.g., an index built using the Faiss library). For new input music, its embedding is obtained through a music encoder. Then, a K-nearest neighbor (K=3) search is performed in this database, and the top-K retrieved neighbors are... The vectors are weighted and averaged, then inversely mapped using a lightweight 3-layer multilayer perceptron (MLP) to restore them to a 1024-dimensional target video visual temporal feature constraint vector. This vector serves as a "style blueprint" for subsequent generation processes.

[0046] 4. Conditional Diffusion Generation Module This module adopts the latest Latent Diffusion Model (LDM) architecture to significantly improve generation efficiency while ensuring quality.

[0047] Model architecture: Variational Autoencoder (VAE): First, a VAE is trained to compress the original 16×52×3 pose sequence into a low-dimensional continuous latent space, resulting in the latent variable z with a dimension of 16×32. This reduces the data dimensionality that the subsequent diffusion model needs to process.

[0048] Conditional Denoising U-Net: The diffusion denoising process is performed on the aforementioned latent space z. The denoising network adopts a 3D U-Net structure to adapt to time-series data. All conditional information (including...) , The beat vectors are first processed by a multi-layer Transformer encoder and then injected into the intermediate layers of the U-Net network via a cross-attention mechanism. Specifically, the beat vectors are converted into sinusoidal positional codes and added to the time-step embeddings of the diffusion process for finer rhythm control.

[0049] Multi-target loss training: Core loss function: This conditional diffusion generative model is based on a designed multi-objective loss function. Training and construction are performed. This loss function... For attitude continuity loss Video visual temporal feature matching loss and motion smoothing loss The weighted sum, its expression is: in, , and These are preset weighting coefficients used to balance the importance of different optimization objectives. A set of optimal coefficients, determined through grid search and validation set performance, is as follows: , , .

[0050] Loss term definition and optimization objective: Attitude continuity loss : Calculate the average L1 norm (absolute error) of the coordinate differences of all relevant nodes between adjacent frames in the generated pose sequence. The optimization objective is to make This loss directly constrains the smooth transition of actions over time, avoiding large positional jumps. The formula is: Visual feature matching loss To ensure that the style of the generated actions is consistent with the target, the generated pose sequence is passed through a lightweight "pose-to-vision" mapping network, similar in structure to the video coding stream in the video-pose feature fusion module, to obtain its visual feature vector. Then calculate. Constraint vector of visual temporal features of the target video The cosine distance between them is used as the loss, and the optimization objective is to maximize the cosine similarity. The formula is: motion smoothing loss The optimization objective is to calculate the variance of the motion velocity vector of each joint in three consecutive frames, and to find the average variance of all joints in all three consecutive frame groups. This loss directly constrains the smoothness of acceleration in motion from the perspective of physical laws of motion, and is key to solving the visual problem of "sudden stops and turns." The formula is: in, Let be the motion velocity of the i-th joint in the t-th frame. The number of key points. Total number of frames This is a function for calculating the variance of each component of the three velocity vectors.

[0051] Training: Distributed training was performed on eight NVIDIA A100 GPUs. The total number of denoising steps in the diffusion process was set to 1000, using linear noise scheduling. The model parameters were optimized by minimizing the aforementioned multi-objective loss function. Optimization was performed. The optimizer used was AdamW, with an initial learning rate set to 1×10⁻⁶. -4 The batch size was 64. The model was trained for a total of 500,000 steps.

[0052] 5. Image optimization and video compositing module This module is responsible for rendering the generated skeletal animation into a video with high visual fidelity and temporal consistency; it is a three-stage cascaded model.

[0053] Neural Rendering Network (Phase 1): Architecture: Employs a variant of neural radiation field, dynamic NeR, which takes spatial coordinates, time and pose parameters as joint inputs and directly outputs the color value and volume density of the spatiotemporal point.

[0054] Training: For each dance style, an independent dynamic NeRF model was trained offline as a differentiable, high-quality renderer using its multi-view, high-frame-rate synchronized video and accurate pose data.

[0055] Stylized Texture Transfer Network (Phase Two): Architecture: An image-to-image translation network based on adaptive instance normalization is employed, using an initial image rendered in stage one and visual temporal feature constraint vectors from the target video upstream. This is the input. The target video visual temporal feature constraint vector. After passing through a small mapping network, a set of modulation parameters (scaling and bias) is generated to control the normalization process of the AdaIN layer on the feature map of the rendered image, thereby transferring the visual attributes of the target dance style to the rendered image.

[0056] Temporal Consistency Enhancement Network (Phase 3): Architecture: This is a network designed specifically for video processing, which combines optical flow estimation and cyclic frame prediction. It takes consecutive image frames generated in the first two stages as input, and applies strong spatiotemporal smoothness constraints by estimating optical flow and combining it with frame prediction based on gated cyclic units.

[0057] Loss function: The training objective of this network is to minimize the optical flow correction loss. Its optimization goal is to make . It is a weighted sum of the optical flow prediction L1 loss and the image reconstruction mean square error loss, expressed as: in, and These are the predicted and actual optical flow fields, respectively. and These are the reconstructed image and the real image, respectively. , , These represent the image's width, height, and number of channels; weighting coefficients are then set. , .

[0058] 6. Results Output and Interaction Module Provide a web-based front-end interactive interface for uploading data, adjusting parameters, previewing, and downloading generated dance videos.

[0059] In addition, the system is equipped with storage and computing power support modules deployed on a private cloud platform. Storage management is achieved using the Ceph distributed file system, and computing power support for model training and high-concurrency online inference is provided through a high-performance computing pool containing dozens of NVIDIA A100 / A800 GPUs, scheduled by a Kubernetes cluster.

[0060] The method flow based on the above system: 1. Data Input and Preprocessing Users upload a target piece of music (e.g., an upbeat electronic dance track) and a style reference video (e.g., a Breaking dance video) through an interactive interface. The system controller invokes the data acquisition and preprocessing module. This module performs a series of preprocessing operations on the uploaded video, including decoding, frame extraction, spatial and contrast normalization, and contour extraction, outputting preprocessed continuous video image frames and a sequence of human posture contour images extracted from these frames. Simultaneously, the module obtains a 3D posture sequence synchronized with the video by parsing video metadata or running an online pose estimation algorithm. For uploaded music, the module extracts its Mel-spectral features, emotion classification label (e.g., "upbeat"), and beat time series.

[0061] 2. Style Feature Extraction and Matching 2.1 The video-pose feature fusion module receives the video image frames, contour line sequences, and 3D pose sequences output from step 1, runs their deep neural networks, extracts and fuses visual and motion features, and finally generates a fused feature vector. This vector encodes the association between visual elements and movement patterns specific to the dance style in the reference video.

[0062] 2.2 The music-video emotion matching module receives the music emotion tags output in step 1, and performs matching queries with the mapping database through its encoder to generate a visual temporal feature constraint vector for the target video. This vector quantitatively describes the visual style features of the dance that match the mood of the input music.

[0063] 3. Conditional dance movement generation The conditional diffusion generation module is activated. It receives the fused feature vector from step 2. and target video visual temporal feature constraint vector The music beat time series from step 1 serves as a multiple generation condition. In the model's latent space, iterative denoising is performed based on the aforementioned conditions, ultimately generating an initial dance posture sequence that highly matches the input conditions in terms of movement style, visual expression, and rhythm.

[0064] 4. Image rendering and video compositing The image optimization and video synthesis module receives the initial dance pose sequence and performs the following operations in sequence: Neuro-geometric rendering: Utilizing a pre-trained dynamic neural radiation field model, pose sequences are rendered from a specified viewpoint into an initial image sequence with three-dimensional consistency.

[0065] Stylized texture enhancement: Based on visual temporal feature constraint vectors of target video A high-fidelity style image sequence is generated by performing stylized texture transfer and detail enhancement on the initial image sequence through an adaptive instance normalization network.

[0066] Temporal consistency optimization: By combining optical flow estimation and recurrent prediction networks, the stylized image sequence is processed for inter-frame motion smoothing and visual consistency enhancement, eliminating flicker and jitter, and outputting a final high-quality, high-smoothness dance image sequence. Subsequently, this image sequence is encoded and synthesized into a high-definition dance video file in a standard format.

[0067] 5. Results Output and Interaction The generated dance video file is sent to the results output and interaction module. The user interface automatically loads and plays a preview of the video. Users can adjust parameters and trigger regeneration through the interface.

[0068] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A dance generation method based on a deep learning model, characterized in that, Includes the following steps: S1. Construct a multimodal dataset containing dance style reference videos, corresponding 3D pose sequences, and music clips labeled with emotion tags. Preprocess the video data, pose data, and music data in the dataset to obtain preprocessed continuous video image frames, aligned and completed 3D pose sequences, and music features containing emotion features and temporal rhythm features. The preprocessing of the video data includes extracting continuous image frames and extracting human pose contour lines from them. S2. Based on the continuous video image frames obtained in step S1, visual temporal features are extracted using a video-pose feature fusion model. The visual temporal features include pose contour features and inter-frame motion features obtained from the continuous video image frames, as well as line rhythm features obtained from the extracted human pose contour lines. Simultaneously, based on the 3D pose sequence obtained in step S1, skeletal temporal features are extracted. The visual temporal features and skeletal temporal features are fused across modally to obtain a fused feature vector. S3. Based on the music emotion features obtained in step S1, match them using a music-video emotion matching model to obtain visual temporal feature constraints of the target video. S4. Using the fused feature vector obtained in step S2, the target video visual temporal feature constraints obtained in step S3, and the music beat temporal features obtained in step S1 as conditions, an initial dance posture sequence is generated through a conditional diffusion generation model, which is constructed based on a preset multi-objective loss function. S5. The initial dance posture sequence generated in step S4 is converted into an image to obtain an initial dance image sequence. Then, the initial dance image sequence is optimized by inter-frame motion correction using an image optimization model to generate a high-quality dance image sequence. S6. Output a high-quality dance video synthesized from the high-quality dance image sequence generated in step S5.

2. The dance generation method based on a deep learning model as described in claim 1, characterized in that, In step S2, cross-modal fusion is achieved through a cross-attention mechanism.

3. The dance generation method based on a deep learning model as described in claim 1, characterized in that, In step S3, the matching is achieved through a mapping relationship established by contrastive learning, which is a mapping relationship from music emotional features to video visual temporal features.

4. The dance generation method based on a deep learning model as described in claim 1, characterized in that, In step S4, the multi-objective loss function For attitude continuity loss Video visual temporal feature matching loss and motion smoothing loss The weighted sum is expressed as: in, , and These are the preset weighting coefficients.

5. The dance generation method based on a deep learning model as described in claim 4, characterized in that, The motion smoothing loss The calculation method is the average variance of the joint motion velocities over three consecutive frames, expressed as: in, Let be the motion velocity of the i-th joint in the t-th frame. The number of key points. Total number of frames Let be the variance function, and the optimization objective of the action smoothing loss is to make .

6. The dance generation method based on a deep learning model as described in claim 4, characterized in that, The attitude continuity loss The calculation method is to use the average L1 norm of the differences in coordinates of all attitude joints between adjacent frames, and its optimization objective is to make .

7. The dance generation method based on a deep learning model as described in claim 4, characterized in that, The video visual temporal feature matching loss The calculation method involves the cosine distance between the visual feature vector corresponding to the generated pose sequence and the visual temporal feature constraint vector of the target video, with the objective being to maximize the cosine similarity. .

8. The dance generation method based on a deep learning model as described in claim 1, characterized in that, In step S5, the inter-frame motion correction optimization is achieved through the optical flow correction module, which employs an optical flow estimation network based on a recurrent architecture.

9. The dance generation method based on a deep learning model as described in claim 8, characterized in that, The optical flow estimation network uses optical flow correction loss. The optical flow correction loss is used to optimize the training process. This is a weighted sum of the L1 loss for optical flow prediction and the MSE loss for image reconstruction.

10. A dance generation system based on a deep learning model, used to implement the method of any one of claims 1-9, characterized in that, include: The data acquisition and preprocessing module is used to construct the multimodal dataset and preprocess the video data, posture data and music data in the dataset, and output the preprocessed video continuous image frames, 3D posture sequences and music features containing emotional features and temporal features; wherein, the preprocessing of video data includes extracting continuous image frames and extracting human posture contour lines from them. The video-pose feature fusion module is connected to the data acquisition and preprocessing module. It is used to receive preprocessed continuous video image frames and 3D pose sequences, and to run the video-pose feature fusion model using the extracted human pose contour lines to extract visual temporal features and skeletal temporal features. The two are then fused across modalities, and the fused feature vector is output to the subsequent modules. The music-video emotion matching module is connected to the data acquisition and preprocessing module. It is used to receive the preprocessed music emotion features, run the music-video emotion matching model, match the input music, and output the visual temporal features of the target video to constrain subsequent modules. The conditional diffusion generation module, connected to the video-pose feature fusion module and the music-video emotion matching module, is used to run a conditional diffusion generation model constructed based on a multi-objective loss function, with the fused feature vector, the visual temporal feature constraints of the target video, and the temporal features of the music beat output by the data acquisition and preprocessing module as conditions, to generate and output the initial dance pose sequence to the subsequent modules; The image optimization and video synthesis module is connected to the conditional diffusion generation module. It is used to perform image-based conversion on the initial dance posture sequence to obtain the initial dance image sequence, and to perform inter-frame motion correction optimization on the initial dance image sequence through the image optimization model to generate a high-quality dance image sequence and synthesize a high-quality dance video. The results output and interaction module, connected to the image optimization and video synthesis module, is used to output a high-quality dance video synthesized from a high-quality dance image sequence.

Images