Music-driven dance video generation method and electronic device based on cross-modal learning
By employing cross-modal learning methods, combined with gated recurrent units, graph convolutional networks, and diffusion models, the problems of facial blurring, identity loss, and monotonous style in music-driven dance video generation were solved, achieving high-quality and diverse dance video generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2026-04-13
- Publication Date
- 2026-07-31
AI Technical Summary
In existing technologies, music-driven dance video generation methods suffer from problems such as uncontrollable pure end-to-end generation, facial blurring or loss of identity features in long sequence video generation, and dance generation models being limited to a single style.
A music-driven dance video generation method based on cross-modal learning is constructed. It utilizes gated recurrent units and graph convolutional networks to generate 2D dance skeleton sequences. By combining a diffusion model with face enhancement technology and a phased training strategy and a comprehensive loss function, the method achieves synchronization between music and dance movements and identity preservation.
The generated dance videos maintain stylistic consistency while possessing extremely high visual quality. Facial features are clear and highly consistent with the identity of the reference images. The videos can generate diverse dance sequence sequences that adapt to different music styles.
Smart Images
Figure CN122493353A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence, computer vision, and cross-modal generation, and in particular to a video generation method and electronic device for music-driven dance based on cross-modal learning. Background Technology
[0002] With the booming development of digital entertainment and social media, dance, as an art form that transcends language barriers, has become a core medium for expressing emotions and disseminating culture on short video platforms. However, the production of high-quality dance content is often limited by the high technical threshold of professional choreography and the complexity of cross-modal mapping, hindering the creative participation of ordinary users. To address this, music-driven dance synthesis technology based on generative artificial intelligence has emerged, aiming to lower the creative threshold and promote the dissemination of cultural diversity by constructing a mapping relationship between auditory features and visual movements.
[0003] In related technologies, traditional motion synthesis methods rely on motion capture data and have poor generalization ability. Although recurrent neural networks (RNNs) have been used for motion modeling, they are limited by the vanishing gradient problem, making it difficult to capture long temporal dependencies, resulting in monotonous generated motions. While recent mainstream diffusion models have improved video quality, they still have significant drawbacks: the generated videos often suffer from facial distortion, inconsistent identity features, and visual artifacts. In addition, existing audio-driven methods are mostly limited to head animation and cannot meet the complex requirements of full-body dance synthesis.
[0004] Chinese invention patent CN117316129A discloses a method, device, and storage medium for generating dance postures from music based on multimodal feature fusion. The method comprises two main stages: 1) a training stage, where the dance is decomposed into a series of basic dance units, through which the model learns how to move; and 2) a generation stage, where the model learns how to create a dance by seamlessly organizing multiple basic dance movements based on the input music. This method solves the challenging problem of translating music into dance by establishing a correspondence between music and dance, considering multiple factors such as the style and rhythm of the music and dance. The model creates dances by learning basic dance units and seamlessly organizing dance movements, resulting in production results that are realistic, diverse, stylistically consistent, and rhythmically matched. This invention improves the fluency and naturalness of dance movements, ensures effective correlation and rhythm between audio and dance movements, enhances the system's comprehensiveness and intelligence, and produces more coordinated and consistent dance movements. However, it still suffers from the uncontrollability of pure end-to-end generation, the tendency for facial blurring or loss of identity features in long video sequence generation, and the limitation of existing dance generation models to a single dance style.
[0005] In summary, there is currently a lack of a video generation method and electronic device for music-driven dance based on cross-modal learning to solve or partially solve the above problems. Summary of the Invention
[0006] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a video generation method and electronic device for music-driven dance based on cross-modal learning, so as to solve or partially solve the problems of uncontrollability of pure end-to-end generation, facial blurring or loss of identity features that easily occur in the generation of long sequence videos, and the limitation of existing dance generation models to a single dance style.
[0007] The objective of this invention can be achieved through the following technical solutions: According to one aspect of the present invention, a method for generating music-driven dance videos based on cross-modal learning is provided, comprising: Acquire multimodal data, including dance videos and reference images; preprocess the multimodal data; extract rhythm and beat features from the music corresponding to the dance videos; and extract image features from the reference images. A music motion generation model based on gated loop units and graph convolutional networks is constructed. The bidirectional gated loop unit is used as an audio encoder to extract the temporal dependency features of music data. The graph convolutional network is used as a generator to generate the corresponding 2D dance skeleton sequence according to the temporal dependency features. A dance video rendering model based on diffusion model and face enhancement is constructed. The image features and 2D dance skeleton sequence are input into the dance video rendering model to synthesize a dance video frame sequence. A built-in face enhancement model is constructed, and face repair is performed using identity feature extraction and adaptive fusion mechanism. A phased training strategy was adopted to construct a comprehensive loss function including WGAN loss, gradient penalty loss and L1 reconstruction loss to train the music action generation model, and the face enhancement model was fine-tuned using identity consistency loss. The target music, reference image, and noise are input into a trained dance video rendering model, which outputs a rhythmically synchronized and identity-preserving dance video, thus realizing the generation of music-driven dance videos.
[0008] As a preferred technical solution, the dance videos include street dance, modern dance, pop dance, and folk dance.
[0009] As a preferred technical solution, the preprocessing includes: The music corresponding to the dance video is separated and cropped, and the sampling rate and duration are standardized. The human pose estimation model is used to extract the sequence of human key points from the dance video to form human pose annotations aligned with the music timeline. A dataset is constructed using the rhythm and beat features, the human pose annotations, and the image features. The dataset is then subjected to denoising, temporal alignment, and normalization.
[0010] As a preferred technical solution, the image features include extracting high-level semantic features of the reference image using a CLIP encoder and extracting low-level texture detail features of the reference image using a pre-trained VAE encoder.
[0011] As a preferred technical solution, the steps of generating a 2D dance skeletal sequence using the music motion generation model include: The extracted rhythm and beat features are input into a bidirectional gated loop unit to capture the long temporal contextual dependencies of the music; Based on the long-term temporal context dependency, the human skeleton is modeled as an undirected graph. A graph convolutional network is used to update the node representation by aggregating neighborhood node information. A 2D dance skeleton sequence including key points is output through a fully connected layer.
[0012] As a preferred technical solution, the steps for generating the dance video frames include: The generated 2D dance skeleton sequence is embedded as a feature representation using the pose encoder of the dance video rendering model to form pose features; The pose features, image features, and Gaussian noise are concatenated and input into a 3D U-Net model. A continuous sequence of dance video frames is generated by denoising through a reverse diffusion process.
[0013] As a preferred technical solution, the process of facial reconstruction using identity feature extraction and adaptive fusion mechanism includes: The MTCNN network is used to detect faces in generated frames and extract key points. The faces are then aligned to a uniform template space using an affine transformation matrix. The identity feature vector of the reference image is extracted using a pre-trained ArcFace model, and adaptive feature fusion is performed with the face feature vector extracted from the generated frame. The fused features are then used to generate an enhanced face through a face reconstruction network. The enhanced face is then fused pixel-wise with the original video frame using a face mask generated by BiSeNet.
[0014] As a preferred technical solution, the dance video rendering model analyzes the 2D dance skeleton sequence, combines the reference image-driven diffusion model and the face enhancement model to perform frame-level synthesis and identity feature repair, and finally generates a complete dance video.
[0015] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for performing a video generation method for music-driven dance based on cross-modal learning as described above.
[0016] According to another aspect of the invention, a computer-readable storage medium is provided, comprising one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing a video generation method for music-driven dance based on cross-modal learning as described above.
[0017] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) This invention constructs a phased coupling framework for generative adversarial network action prediction and diffusion model video synthesis, which takes into account both the temporal synchronization of generation and visual high fidelity. It utilizes GRU and GAN models to efficiently capture long temporal dependencies between music and action, ensuring precise timing of dance rhythm. Then, it uses a video synthesis module based on diffusion model to convert structured skeletal sequences into high-resolution videos, solving the problem of uncontrollability in pure end-to-end generation. This achieves the technical effect of maintaining stylistic consistency while possessing extremely high visual quality in the final generated video.
[0018] (2) By constructing a face enhancement model, introducing ArcFace to extract identity features, and combining adaptive feature fusion and affine alignment technology, the present invention explicitly repairs facial geometric details, solves the problem of facial blurring or loss of identity features that is easy to occur in the generation of long sequence videos, and achieves the technical effect that the face of the person in the generated video is always clear and highly consistent with the identity of the reference image.
[0019] (3) This invention trains the model by constructing a multi-ethnic multimodal dance video dataset with different styles, so that the trained model can generate diverse action sequences, extract corresponding music features according to the input music style, and automatically adjust the rhythm features of the generated dance. This solves the problem that the existing dance generation model is limited to a single dance style and achieves the technical effect of enhancing the multimodal generalization ability and cultural adaptability of the dance video rendering model. Attached Figure Description
[0020] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a data flow diagram of the method of the present invention; Figure 3 A comparison diagram of the dance video frame sequence generated by the method of the present invention and the frame sequence diagrams of several existing video generation methods; Figure 4 This is a sequence diagram of dance video frames generated by the method of the present invention based on different music. Figure 5 This is a comparison diagram of the results of the face enhancement module in the method of the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0022] Example 1 To address the problems existing in the prior art, this embodiment provides a video generation method for music-driven dance based on cross-modal learning, such as... Figure 1 As shown, the specific steps include: S1. Multimodal data input and preprocessing, including music and reference images. The target music is preprocessed to extract rhythm and beat features; feature extraction is performed on the reference images. The reference images used in this section are all generated by artificial intelligence and are not real-world portraits.
[0023] Specifically, this invention collects dance video data across multiple styles, rhythms, and dance genres, covering various dance types including street dance, modern dance, pop dance, and folk dance. The corresponding music audio for each dance video is separated and cropped, with standardized sampling rates and durations to obtain standardized music input data. The Openpose human pose estimation model is used to extract human keypoint sequences from the dance videos, forming human pose annotations strictly aligned with the music on the timeline. This invention extracts the audio track from the video and resamples it to 16kHz. The videos are manually screened to remove low-quality or out-of-sync segments, retaining only clear samples with distinct dance styles. All videos are standardized to 24 frames per second.
[0024] The dataset was denoised, time-aligned, and normalized, and then divided into training and test sets in an 8:2 ratio to ensure the stability and generalization ability of the model training.
[0025] An audio encoder is used to extract music features, a CLIP encoder is used to extract high-level semantic features of the reference image, and a pre-trained VAE encoder is used to extract low-level texture detail features of the reference image.
[0026] S2. Construct a music-to-pose model based on gated recurrent units (GRU) and graph convolutional networks (GCN). Use bidirectional gated recurrent units (Bi-GRU) as audio encoders to extract temporal dependency features of music, and use graph convolutional networks (GCN) as generators to generate corresponding 2D dance skeleton sequences based on audio features.
[0027] Specifically, the extracted audio features are input into a bidirectional GRU network to capture the long-term temporal contextual dependencies of the music signal. To ensure consistency between actions and musical style, this invention first uses an audio encoder to extract audio features. Then, this invention uses a bidirectional GRU to model these features to capture temporal dependencies and uses them as the initial node input to a graph convolutional network (GCN).
[0028] The human skeleton is modeled as an undirected graph. A graph convolutional network (GCN) is used as a generator to update the node representation by aggregating neighborhood node information. Finally, a 2D coordinate sequence containing 18 key points is output through a fully connected layer.
[0029] In 2D skeleton extraction, the human skeleton is modeled as an undirected graph G=(V, E), where nodes V represent 2D keypoints and edges E represent skeleton connections. The skeleton state at time t is: in, For complete skeleton sequence, This represents the total number of time frames. The number of key points in the human skeleton. The feature dimensions for each key point.
[0030] S3. Construct a dance video rendering model based on diffusion model and face enhancement. The diffusion model based on 3D U-Net architecture is adopted, and reference image features, skeletal sequence features and noise are used as input to synthesize dance video frames. A built-in face enhancement model is constructed, and facial details are repaired by identity feature extraction and adaptive fusion mechanism.
[0031] Specifically, the generated skeleton sequence is embedded as a feature representation using a pose encoder; the pose features, reference image features and Gaussian noise are concatenated and input into a 3D U-Net, and a continuous video frame sequence is generated by gradually denoising through a reverse diffusion process.
[0032] This invention introduces a pre-trained VAE encoder as a supplement. This encoder compresses multi-scale features to preserve low-level visual details such as texture and edges. For the generated pose sequences, this invention uses a ControlNet-based pose encoder to embed them into the feature representation. Then, the pose features, input noise, and reference image features are concatenated and fed into a U-Net for joint modeling. Finally, a diffusion model decodes the fused high-dimensional features into a series of dance video frames that match the input style and motion.
[0033] MTCNN detects faces in generated frames and extracts key points, calculates affine transformation matrices to align faces to a unified template space, and uses a pre-trained ArcFace model to extract identity feature vectors from reference images and face feature vectors from generated frames, and performs adaptive feature fusion.
[0034] This invention uses a multi-layer convolutional neural network (MTCNN) to detect each frame of the image. Extract five key points from the face in the image. And calculate an affine transformation matrix. Compare each frame of the image with a reference image. Align to a uniform template space to obtain a normalized image: in, Indicates the first after alignment The image generated by the frame, and This represents a reference image aligned using the same transformation.
[0035] This invention utilizes a pre-trained ArcFace model to align reference images Extract identity feature vectors For each aligned video frame This invention extracts facial feature vectors. To integrate identity information from reference images, this invention designs an adaptive feature fusion mechanism. and Fusion, generating fusion representation This process can be represented as: in, It is a weighting parameter used to control the influence of identity features.
[0036] The fused features are used to generate an enhanced face through a face reconstruction network, and the enhanced face is then fused with the original video frame at the pixel level using the face mask generated by BiSeNet.
[0037] Fusion features It is then fed into a face reconstruction network to generate enhanced face image regions. To reintegrate the enhanced region into the complete video frame, this invention uses the BiSeNet face segmentation network to obtain a binary mask. The value 1 represents a face region. This invention employs a pixel-level fusion strategy to synthesize the generated face with the original frame.
[0038] S4. A phased training strategy is adopted. First, the music motion generation model is trained using the GAN generative adversarial network framework, and a comprehensive loss function including WGAN loss, gradient penalty (GP) loss and L1 reconstruction loss is used. Then, the video rendering model is trained, and the face enhancement model is fine-tuned using identity consistency loss (ID Loss).
[0039] Specifically, the comprehensive loss function includes: 1. WGAN Loss: Utilizes Wasserstein distance to replace the traditional JS divergence in GANs for stable adversarial training between the generator and discriminator. The loss function is... Defined as: in, Indicates obedience to musical distribution Input Take the expected value. This indicates that the data follows the true skeleton data distribution. samples Take the expected value. The skeleton sequence output by the generator. The input music signal. This is a true skeleton sequence. It is a discriminator.
[0040] 2. Gradient Penalty Loss: This imposes a constraint on the gradient norm of the discriminator to satisfy the Lipschitz continuity condition. The loss function... Defined as: in, These are the weighting coefficients. The sample is obtained by interpolating real data and generated data. For the discriminator D with respect to the input gradient, This indicates that the sample follows an interpolated sample distribution. of v takes the mathematical expectation.
[0041] 3. L1 Reconstruction Loss: Directly constrains the frame-level difference between the generated skeletal joint positions and the actual joint positions to preserve motion details and prevent over-smoothing. The loss function is... Defined as: in, These are the weighting coefficients for L1 loss. For the real skeleton With corresponding music The joint distribution takes the expected value.
[0042] The final total loss function is the weighted sum of the losses of all individuals, and can be expressed as: in, and It is to adjust the weighting coefficients of different loss contributions based on the experimental results.
[0043] After the dance video rendering model outputs the initial dance video frame sequence, facial enhancement is introduced to detect, align, and map the facial regions of the characters in the video.
[0044] The face enhancement model uses Identity Consistency Loss (ID Loss) for fine-tuning. It calculates the cosine similarity between the generated face and the reference image in the identity feature space. This loss measures the cosine similarity between the generated result and the reference image in the identity feature space, thus ensuring that the generated face remains close to the reference identity. The final ID Loss is expressed as: in, This represents the total number of frames in the video sequence. For pre-trained facial recognition feature extraction networks, The input conditions for the i-th frame are... A synthetic image containing source identity features and target facial expression. For the first The image generated from the frame.
[0045] S5. Input the target music audio and a single reference image of the person into the trained model, and output a dance video with synchronized rhythm and preserved identity.
[0046] like Figure 2 As shown, the input music audio undergoes music generation motion processing and posture encoding to generate a standardized 2D skeletal keypoint sequence. The geometric posture of each frame and its temporal topology relationship with the music rhythm are defined. The inference script is executed to automatically parse the skeletal sequence data. Combined with the encoded reference image to drive the diffusion model and the face enhancement module, frame-level synthesis and identity feature repair are performed, ultimately generating a complete dance video with synchronized rhythm and realistic faces.
[0047] This example compares the generated dance video with three representative advanced human image animation methods: MagicAnimate, MooreAnimate, and MagicPose. Figure 3As shown, these baseline methods generally have certain limitations when generating dance videos, such as insufficient consistency in character identity, blurred facial details, or distortion of local structures. Specifically, while MagicAnimate supports dance motion generation, its ability to depict complex movements is limited, resulting in low motion accuracy and stiff overall human body movements, lacking the fluidity and expressiveness required for realistic dance. MooreAnimate and MagicPose perform relatively better in motion tracking and posture reconstruction, accurately capturing human movement trajectories, but they have significant shortcomings in clothing and appearance dynamic modeling. For example, skirts and other garments lack reasonable physical changes during movement, appearing almost static or exhibiting unnatural shaking, thus affecting the overall visual realism. In contrast, the method presented in this example demonstrates significant advantages in several key dimensions. First, in terms of identity consistency, the generated video can stably maintain the appearance features of the person and avoid obvious drift over time; second, in terms of facial detail, the face generated by this method is clearer and more natural, and the expression changes are more coherent; in terms of motion accuracy and temporal consistency, this method can accurately restore the details of dance movements and maintain good temporal smoothness between adjacent frames, thereby significantly reducing jitter and discontinuity.
[0048] like Figure 4 As shown, the method of this invention also demonstrates good adaptability to dance generation across different musical styles. Regardless of whether the music is fast-paced or emotionally lyrical, this method accurately captures rhythmic variations and emotional characteristics, generating distinctive, fluid, and expressive dance sequences accordingly. This further verifies the versatility and robustness of this method in multi-style dance generation tasks. To evaluate the effectiveness of the face enhancement module in maintaining identity consistency and facial clarity, this invention removes the face enhancement module for a comparative analysis.
[0049] like Figure 5 As shown, the results indicate that the video without the facial enhancement module exhibits significant visual blurring and identity drift, confirming that the facial enhancement module significantly improves facial identity consistency and video quality. Overall, the dance video generated by this method has a more natural and realistic visual effect, with a higher degree of matching between character movements, music rhythm, and dance style, resulting in a more harmonious viewing experience.
[0050] Example 2 Based on the foregoing embodiments, this embodiment provides an electronic device, including a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned video generation method for music-driven dance based on cross-modal learning. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, that is, the execution entity of the processing flow is not limited to individual logic units, but can also be hardware or logic devices.
[0051] Example 3 Building upon the foregoing embodiments, this embodiment provides a computer-readable storage medium storing a computer program that can be used to execute the aforementioned video generation method for music-driven dance based on cross-modal learning. Computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.
[0052] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for generating a music-driven dance video based on cross-modal learning, characterized in that, The method specifically includes: Acquire multimodal data, including dance videos and reference images; preprocess the multimodal data; extract rhythm and beat features from the music corresponding to the dance videos; and extract image features from the reference images. A music motion generation model based on gated loop units and graph convolutional networks is constructed. The bidirectional gated loop unit is used as an audio encoder to extract the temporal dependency features of music data. The graph convolutional network is used as a generator to generate the corresponding 2D dance skeleton sequence according to the temporal dependency features. A dance video rendering model based on diffusion model and face enhancement is constructed. The image features and 2D dance skeleton sequence are input into the dance video rendering model to synthesize a dance video frame sequence. A built-in face enhancement model is constructed, and face repair is performed using identity feature extraction and adaptive fusion mechanism. A phased training strategy was adopted to construct a comprehensive loss function including WGAN loss, gradient penalty loss and L1 reconstruction loss to train the music action generation model, and the face enhancement model was fine-tuned using identity consistency loss. The target music, reference image, and noise are input into a trained dance video rendering model, which outputs a rhythmically synchronized and identity-preserving dance video, thus realizing the generation of music-driven dance videos.
2. The video generation method for music-driven dance based on cross-modal learning according to claim 1, characterized in that, The dance videos include street dance, modern dance, pop dance, and folk dance.
3. The video generation method for music-driven dance based on cross-modal learning according to claim 1, characterized in that, The preprocessing includes: The music corresponding to the dance video is separated and cropped, and the sampling rate and duration are standardized. The human pose estimation model is used to extract the sequence of human key points from the dance video to form human pose annotations aligned with the music timeline. A dataset is constructed using the rhythm and beat features, the human pose annotations, and the image features. The dataset is then subjected to denoising, temporal alignment, and normalization.
4. The video generation method for music-driven dance based on cross-modal learning according to claim 1, characterized in that, The image features include high-level semantic features extracted from the reference image using a CLIP encoder, and low-level texture detail features extracted from the reference image using a pre-trained VAE encoder.
5. The video generation method for music-driven dance based on cross-modal learning according to claim 1, characterized in that, The steps for generating a 2D dance skeletal sequence using the aforementioned music motion generation model include: The extracted rhythm and beat features are input into a bidirectional gated loop unit to capture the long temporal contextual dependencies of the music; Based on the long-term temporal context dependency, the human skeleton is modeled as an undirected graph. A graph convolutional network is used to update the node representation by aggregating neighborhood node information. A 2D dance skeleton sequence including key points is output through a fully connected layer.
6. The video generation method for music-driven dance based on cross-modal learning according to claim 1, characterized in that, The steps for generating the dance video frames include: The generated 2D dance skeleton sequence is embedded as a feature representation using the pose encoder of the dance video rendering model to form pose features; The pose features, image features, and Gaussian noise are concatenated and input into a 3D U-Net model. A continuous sequence of dance video frames is generated by denoising through a reverse diffusion process.
7. The video generation method for music-driven dance based on cross-modal learning according to claim 1, characterized in that, The process of facial reconstruction using identity feature extraction and adaptive fusion mechanism includes: The MTCNN network is used to detect faces in generated frames and extract key points. The faces are then aligned to a uniform template space using an affine transformation matrix. The identity feature vector of the reference image is extracted using a pre-trained ArcFace model, and adaptive feature fusion is performed with the face feature vector extracted from the generated frame. The fused features are then used to generate an enhanced face through a face reconstruction network. The enhanced face is then fused pixel-wise with the original video frame using a face mask generated by BiSeNet.
8. The video generation method for music-driven dance based on cross-modal learning according to claim 1, characterized in that, The dance video rendering model parses the 2D dance skeleton sequence and combines the reference image-driven diffusion model with the face enhancement model to perform frame-level synthesis and identity feature restoration, ultimately generating a complete dance video.
9. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for performing the video generation method for music-driven dance based on cross-modal learning as described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, Includes one or more programs executable by one or more processors of an electronic device, said one or more programs including instructions for performing the video generation method for music-driven dance based on cross-modal learning as described in any one of claims 1-8.