Dance video generation method based on motion focus attention and decoupled control

Abstract

The present application relates to the technical field of music conditioned video generation, in particular to a dance video generation method based on motion focus attention and decoupling control, comprising: extracting music features, reference image features and text features; adding noise to the training video and inputting the multi-modal features into a denoising network; dividing the character area through a motion focus attention mechanism and adopting a differentiated time sequence shift step to aggregate historical frame information; realizing frame-level alignment of music and video through a cross attention mechanism; processing text and image conditions respectively through a decoupling cross attention mechanism and weighted fusion; and calculating noise prediction loss to optimize the model. The present application effectively solves the problems of inconsistent character identity, incoherent action and out-of-sync audio and video, and generates high-quality dance videos.

Description

Technical Field

[0001] This invention relates to the field of music conditional video generation technology, specifically a method for generating dance videos based on motion-focused attention and decoupled control. Background Technology

[0002] Music-conditioned dance video generation technology has significant application value in fields such as virtual human animation and digital entertainment. Its core lies in generating coherent, natural, consistent, and rhythmically synchronized dance videos based on musical characteristics.

[0003] Current mainstream methods employ a two-stage paradigm of "pose generation-rendering compositing," which suffers from error accumulation and loss of detail. While end-to-end diffusion model-based methods avoid the shortcomings of the two-stage approach, they still face three technical bottlenecks: First, the appearance of the character in different frames of the generated video is prone to identity drift, making it difficult to maintain a continuous and stable character image; second, the timing of the music rhythm and dance movements is not aligned, resulting in weak audio-visual synchronization; and third, the transitions between movements are unnatural when generating long video sequences, especially in large-scale dance movements, where issues such as limb disjointedness or incoherent movements are likely to occur. Summary of the Invention

[0004] This invention provides a dance video generation method based on motion-focused attention and decoupled control to solve the technical problems of inconsistent character identities, disjointed movements, and audio-visual asynchrony in existing dance video generation methods, thereby achieving the generation of high-quality and controllable dance videos.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention relates to a dance video generation method based on motion-focused attention and decoupled control, comprising:

[0007] S100: Obtain music sequence, reference image of person and text description, and extract music features, reference image features and text features respectively;

[0008] S200: Add noise to the training video in the latent space to obtain noisy latent features, and input the noisy latent features, the music features, the reference image features and the text features into the denoising network;

[0009] S300: In the denoising network, the motion-focused attention mechanism is used to locate the human figure region and divide it into different motion regions. Different temporal shift step sizes are used to aggregate historical frame information and perform attention calculation with the current frame features to obtain video features.

[0010] S400: The music features and video features are aligned at the frame level using a cross-attention mechanism;

[0011] S500: The text features and the reference image features are processed separately by decoupling the cross-attention mechanism. The text features are input to all cross-attention layers, and the reference image features are input to a specific cross-attention layer. The two attention outputs are then weighted and fused.

[0012] S600: The denoising network predicts the noise in the latent features after adding noise, calculates the loss between the predicted noise and the actual added noise, and optimizes the model parameters through backpropagation.

[0013] As a preferred embodiment of the present invention, the step of extracting music features includes:

[0014] The music sequence is divided into multiple consecutive segments of fixed duration;

[0015] Extract the Mel frequency cepstral coefficient features of each segment;

[0016] The music features are obtained by extracting temporal features from the Mel frequency cepstral coefficients using a one-dimensional convolutional network.

[0017] As a preferred embodiment of the present invention, the step of extracting reference image features includes:

[0018] The reference image of the person is encoded using a pre-trained image encoder to obtain static reference image features;

[0019] Extend the features of the static reference image in the time dimension;

[0020] The extended static reference image features are mapped to a dynamic reference feature sequence using a multilayer perceptron to obtain the reference image features.

[0021] As a preferred embodiment of the present invention, the step of dividing the different motion regions includes:

[0022] Locate the central region of the person on the noisy latent features;

[0023] The central area of ​​the figure is divided into a central area and an edge area.

[0024] As a preferred embodiment of the present invention, the differentiated temporal shift step size for different motion regions includes:

[0025] The feature blocks in the central region are shifted using a first timing shift step.

[0026] The feature blocks of the edge region are shifted using a second temporal shift step size.

[0027] The second timing shift step size is greater than the first timing shift step size.

[0028] As a preferred embodiment of the present invention, the aggregation of historical frame information and the attention calculation with current frame features include:

[0029] By applying multiple different historical shift step sizes to the overall frame features, a block shift operation is performed to obtain an overall hybrid feature frame;

[0030] The features of the central region and the edge region after block shifting are spliced ​​together to obtain the mixed features of human motion;

[0031] The character motion blend feature, the overall blend feature frame, and the current frame feature are concatenated as the key and value of the attention mechanism, and the current frame feature is used as the query to perform attention calculation.

[0032] As a preferred embodiment of the present invention, the frame-level alignment of the music features and the video features using a cross-attention mechanism includes:

[0033] The video features are spatially reshaped so that the frame rate of the music features matches the frame rate of the video features.

[0034] A frame-by-frame correspondence between the music features and the video features is established using a cross-attention mechanism.

[0035] As a preferred embodiment of the present invention, the step of processing the text features and the reference image features separately through the decoupled cross-attention mechanism includes:

[0036] The text features are passed through a first trainable projection layer to generate a first key-value pair, and the video features are subjected to global semantic control through a text cross-attention mechanism.

[0037] The reference image features are used to generate a second key-value pair through a second trainable projection layer. This pair is introduced only in specific cross-attention layers in the downsampling block, intermediate block, and upsampling block of the denoising network. The video features are then used to control the appearance of the person through an identity cross-attention mechanism.

[0038] The text cross-attention output and the identity cross-attention output are weighted and fused.

[0039] As a preferred embodiment of the present invention, the calculation of the loss between the predicted noise and the actual added noise includes:

[0040] The mean square error between the predicted noise and the actual added noise is calculated as the loss function.

[0041] As a preferred embodiment of the present invention, it further includes a generation stage:

[0042] Obtain the music sequence, reference images of people, and text description of the video to be generated, and extract multimodal conditional features according to step S100;

[0043] Random Gaussian noise is initialized in the latent space as initial noise latent features, and the initial noise latent features and the multimodal conditional features are input into the trained denoising network.

[0044] The denoised latent features are obtained by iteratively denoising through the denoising network according to steps S300 to S500.

[0045] For long video generation, a video sequence corresponding to a music segment is generated each time. The last frame of the generated video sequence is used as the reference image for the next music segment. This process is repeated until the complete music sequence is processed.

[0046] The decoder, which uses a pre-trained variational autoencoder, decodes the denoised latent features into video frames.

[0047] The beneficial effects of this invention are:

[0048] 1. The motion-focused attention mechanism proposed in this invention accurately captures the motion characteristics of different body parts by employing differentiated temporal shift step lengths for the trunk and limbs, thereby reducing computational complexity. This mechanism reduces the computational complexity of attention and significantly improves the coherence and naturalness of long sequences of dance movements.

[0049] 2. The decoupled cross-attention mechanism proposed in this invention introduces textual conditions at a global control level and image conditions at a specific layer, achieving independent processing of semantic control and identity preservation. This mechanism effectively avoids mutual interference between multimodal conditions, ensuring that the generated video accurately expresses textual semantics while maintaining the consistency of character identities.

[0050] 3. The frame-level audio-video alignment strategy proposed in this invention achieves strict synchronization between dance movements and music rhythm by constraining the number of music segments to match the number of video frames and establishing a frame-by-frame correspondence. This strategy optimizes the problem of audio-visual timing misalignment in traditional methods and significantly improves the rhythm matching degree of the generated video. Attached Figure Description

[0051] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0052] Figure 1 This is a schematic diagram of the overall process of the dance video generation method based on motion-focused attention and decoupled control of the present invention;

[0053] Figure 2This is a schematic diagram of the overall model framework of the present invention;

[0054] Figure 3 This is a flowchart of the dance video generation stage of the present invention;

[0055] Figure 4 This is a dance video effect image generated by the present invention. Detailed Implementation

[0056] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0057] Example 1: As Figure 1 , Figure 2 and Figure 3 As shown, this invention provides a dance video generation method based on motion-focused attention and decoupled control. Figure 1 The overall process of the method of the present invention is shown. Figure 2 The detailed model framework structure is shown. Figure 3 The invention illustrates the specific process of the generation phase and specifically includes:

[0058] S100: Obtain music sequence, reference image of person and text description, and extract music features, reference image features and text features respectively;

[0059] Furthermore, the steps for extracting musical features include:

[0060] The music sequence is divided into multiple consecutive segments of fixed duration;

[0061] Extract the Mel frequency cepstral coefficient features of each segment;

[0062] The music features are obtained by extracting temporal features from the Mel frequency cepstral coefficients using a one-dimensional convolutional network.

[0063] Furthermore, the steps for extracting features from the reference image include:

[0064] The reference image of the person is encoded using a pre-trained image encoder to obtain static reference image features;

[0065] Extend the features of the static reference image in the time dimension;

[0066] The extended static reference image features are mapped to a dynamic reference feature sequence using a multilayer perceptron to obtain the reference image features.

[0067] Specifically, users submit music sequences, reference images of characters, and text descriptions through an interactive interface as input conditions for generating dance videos.

[0068] Given a musical signal First, preprocess it: Divided into A fixed duration And non-overlapping continuous segments, i.e. Subsequently, the Mel-frequency cepstral coefficients (MFCCs) of each musical segment are calculated as input features to obtain the musical features. ,in This represents the number of feature vectors (i.e., the number of music segments obtained from preprocessing segmentation). To ensure the temporal alignment requirements in subsequent multimodal processing, its value is set to be consistent with the number of frames in the corresponding video. This indicates the dimension of the extracted MFCC features, which can be adjusted according to the complexity of the music in practical applications.

[0069] The obtained frame-level music feature sequence The input is fed into a multi-layer one-dimensional convolutional neural network (1D-Conv) for processing to extract temporal dependencies and obtain music features. The processing procedure can be formally represented as:

[0070] ;

[0071] in, It is a one-dimensional convolutional neural network used to capture temporal variation patterns of musical features.

[0072] Using a pre-trained image encoder Reference image Encode to obtain feature representation ,in These represent the height, width, and number of channels of the feature map, respectively.

[0073] To match the time dimension of video clips ( Frame alignment, spatial features Replication in the time dimension Next, get .

[0074] A trainable multilayer perceptron (MLP) is introduced to map static reference features to a time-dependent dynamic feature sequence. This MLP uses... As input, it learns the dynamic mapping relationship from the initial static state to various target dance movements, and outputs a dynamic reference feature sequence. This process can be formally represented as:

[0075] ;

[0076] Through this MLP network, the features of static human reference images are transformed into a sequence of features with temporal dynamics, enabling them to adapt to the changing needs of different dance movements.

[0077] Using a pre-trained CLIP text encoder Text description The data is processed and encoded into text features. This feature As a crucial semantic guiding signal, it is input into the diffusion model along with other conditional features. This collaboratively guides the model to perform denoising iterations in the latent space, thereby achieving controllable video generation. The processing can be formally represented as:

[0078] ;

[0079] Text features It contains high-level semantic information such as dance style and scene content, providing global semantic control for video generation.

[0080] S200: Add noise to the training video in the latent space to obtain noisy latent features, and input the noisy latent features, the music features, the reference image features and the text features into the denoising network;

[0081] Specifically, during the training phase, the system acquires training video data and performs preprocessing. The raw video data is first segmented at fixed frame intervals with equal spacing and no overlap, resulting in a series of video clips. A center-cropping operation is then performed on each frame of each clip to obtain video data with a spatial size of 1024×1024 pixels. ,in This indicates the segment length. For ease of subsequent processing, the tensor... By performing spatial downsampling, reducing its spatial dimension to 512×512 pixels, we obtain... The video frame resolution can be adjusted according to hardware conditions. Subsequently, a pre-trained variational autoencoder (VAE) encodes each frame into the latent space, obtaining the initial latent feature representation of the video segment. The processing procedure can be formally represented as:

[0082] ;

[0083] In the latent space, for the initial latent features Perform the forward diffusion process. Randomly sample time steps in each training iteration. According to the forward process of the diffusion model, towards Add Gaussian noise to obtain the latent features after adding noise. .

[0084] The forward diffusion process follows a Markov chain, iteratively adding noise to gradually transform the original data into noisy data that approximates a standard Gaussian distribution. For any time step... Latent features after adding noise You can start directly from the initial state The samples were obtained through the following sampling method:

[0085] ;

[0086] in, This represents the actual added noise sampled from the standard normal distribution. These are predefined noise scheduling parameters used to control the noise intensity at different time steps. As time steps... The increase, As the noise level gradually decreases, the proportion of noise gradually increases.

[0087] The latent features after adding noise Time step and the multimodal conditional features extracted in step S100 The data is fed into the denoising network. The goal of the denoising network is to learn to predict what to add. Noise in This allows for the gradual recovery of a clear dance video from random noise during the generation phase.

[0088] It should be noted that the time step During training, from a uniform distribution Random sampling in the middle, of which This represents the maximum number of time steps in the diffusion process, typically set to 1000. Different time steps correspond to different noise levels, enabling the model to learn denoising under various noise conditions.

[0089] In this embodiment, the denoising network adopts a diffusion model based on the U-Net architecture. This network subsequently employs a motion-focused attention mechanism, an audio / video alignment strategy, and a decoupled cross-attention mechanism to... Processing is performed to achieve coordinated control of multimodal conditions.

[0090] S300: In the denoising network, the motion-focused attention mechanism is used to locate the human figure region and divide it into different motion regions. Different temporal shift step sizes are used to aggregate historical frame information and perform attention calculation with the current frame features to obtain video features.

[0091] Within the denoising network, the motion-focusing attention module processes the latent features after adding noise. The process involves aggregating historical frame information using a differentiated block shifting strategy to enhance temporal modeling. The design goal of the motion-focused attention mechanism is to enhance the motion coherence and temporal consistency of the generated dance video while reducing computational complexity. This mechanism is based on the observation that in continuous dance motion sequences, the torso typically remains near the central region with relatively small movements, while the limbs often exhibit large swings, distributed on both sides of the center line. The differentiated block shifting strategy enables refined modeling of the character's motion regions.

[0092] Furthermore, the steps for dividing the different motion regions include:

[0093] Locate the central region of the person on the noisy latent features;

[0094] The central area of ​​the figure is divided into a central area and an edge area.

[0095] Specifically, for each frame, the central region where the person is located is located on the latent representation or feature map, and the "person central region" and "background region" are distinguished in the latent feature map. The background region usually remains static and therefore does not participate in the shifting operation; while the person region is the core target of motion modeling.

[0096] In feature map The above steps involve locating the character's center area and the static background area. The center area is then further subdivided to obtain the central region. (Mainly corresponding to the torso) and peripheral areas (Primarily corresponding to the limbs). This partitioning method allows the model to differentiate the motion characteristics of different body parts.

[0097] It should be noted that the localization of the person's region can be achieved through human pose estimation or saliency detection methods. The division between the central and peripheral regions is based on the spatial distance from the person's mass center, usually using the center of the person's bounding box as a reference, and dividing the area near the center into... The area far from the center is divided into .

[0098] Furthermore, the differentiated temporal shift step size for different motion regions includes:

[0099] The feature blocks in the central region are shifted using a first timing shift step.

[0100] The feature blocks of the edge region are shifted using a second temporal shift step size.

[0101] The second timing shift step size is greater than the first timing shift step size.

[0102] Specifically, let the current frame feature be... The historical frame set is For each feature block location Historical frame information is aggregated using different time-shift step sizes based on the region to which it belongs.

[0103] If feature block Belongs to the central area Then the first timing shift step size is used. A block shift operation is performed to capture subtle changes in the torso. Its formal representation is as follows:

[0104] ;

[0105] If feature block Belongs to the edge area Then the second timing shift step size is adopted. A block shifting operation is performed to capture the long-range motion trajectories of the limb extremities. Its formal representation is:

[0106] ;

[0107] in, .

[0108] The features of the central region of the figure obtained through the above shifting operation and edge region features By splicing the images together, we can obtain the mixed features of the central area of ​​the figure. Then, it is concatenated with the static background features to obtain a human motion blend feature with the same size as the original frame. This differentiated displacement strategy can capture subtle changes in movement in the torso while preserving long-range movement trajectories in the extremities.

[0109] In this embodiment, the shift step size range That is, the central area uses a small step size shift of 1-3 frames, and the edge area uses a large step size shift of 4-7 frames. This parameter is set to the preferred value and can be adjusted according to the amplitude characteristics of the dance movements.

[0110] In the time dimension, temporal mixing of the entire frame's features is also required to further enhance inter-frame motion modeling capabilities. For each feature block location... Through shift operations at different time steps:

[0111] ;

[0112] in, Indicates different historical shift steps, This indicates the number of historical frames. These time-shifted features are then concatenated to construct a hybrid feature frame. Its shape is similar to the features of the current frame. Maintaining consistency. This allows for the explicit introduction of multi-frame motion context information into single-frame features, further enhancing the temporal consistency of actions.

[0113] Furthermore, the aggregation of historical frame information and the attention calculation with current frame features includes:

[0114] By applying multiple different historical shift step sizes to the overall frame features, a block shift operation is performed to obtain an overall hybrid feature frame;

[0115] The features of the central region and the edge region after block shifting are spliced ​​together to obtain the mixed features of human motion;

[0116] The character motion blend feature, the overall blend feature frame, and the current frame feature are concatenated as the key and value of the attention mechanism, and the current frame feature is used as the query to perform attention calculation.

[0117] Specifically, by employing a global frame block-level differentiated shift strategy, adjacent historical frames are spliced ​​together after block-level shifting to generate a hybrid feature frame. (Including overall motion context information); The character-centric block-level differentiated shift strategy for the character's central area employs a character-specific shift for the main character block. (Includes fine motion information); combine the two with the features of the current frame. The concatenation serves as the key and value in the attention mechanism. It can be formally represented as:

[0118] ;

[0119] ;

[0120] ;

[0121] in, Indicates a query frame; This represents the hybrid feature frame obtained by region shifting; This represents the hybrid feature frame obtained by shifting the entire frame; This indicates the concatenate operation; , , It is a learnable projection matrix.

[0122] The formal representation of attention computation is as follows:

[0123] ;

[0124] in, The feature dimension is used as a scaling factor to stabilize the training process.

[0125] Through this attention mechanism, the model can adaptively extract motion information related to the current frame from historical frames, resulting in video features that enhance temporal coherence. This mechanism reduces the computational complexity of attention from the original... Down to ,in For frame number, The number of feature tokens per frame significantly reduces the computational overhead of generating long video sequences, while effectively avoiding ghosting or fragmentation in the generated video.

[0126] S400: The music features and video features are aligned at the frame level using a cross-attention mechanism;

[0127] To achieve a strict temporal correspondence between music and video frames, this invention constructs a direct mapping relationship from music features to latent video features. The music sequence is represented by frame-level feature extraction in step S100. ,in These represent the frame number, the length of the music feature sequence within a single frame, and the MFCC feature dimension, respectively. The video latent space features are denoted as... ,in and These represent the frame count, height, width, and channel dimension, respectively.

[0128] Furthermore, the step of aligning the music features with the video features at the frame level using a cross-attention mechanism includes:

[0129] The video features are spatially reshaped so that the frame rate of the music features matches the frame rate of the video features.

[0130] A frame-by-frame correspondence between the music features and the video features is established using a cross-attention mechanism.

[0131] Specifically, to establish the mapping relationship between audio and video features, the video feature sequence obtained through the Motion Focus Attention (MFA) mechanism in step S300 is spatially reshaped, and it is transformed from... Remodeling Through this reshaping operation, the spatial dimension of the video features is flattened, and the features of each frame are represented as a two-dimensional matrix, with the frame number dimension consistent with that of the music features.

[0132] It should be noted that during the music signal processing in step S100, the number of music segments... It has been set to match the frame rate of the corresponding video, therefore the music features Frame count dimension With video features Frame count dimension Natural alignment provides a fundamental guarantee for frame-level audio and video alignment.

[0133] Establishing musical features through cross-attention mechanism With video features The correlation between them. In this mechanism, video features serve as the query, and music features serve as the key and value, enabling video frames to extract rhythm and cadence information from corresponding music segments. The cross-attention calculation is formally represented as:

[0134] ;

[0135] in, A learnable projection matrix is ​​used to map video features and music features to the same representation space; For attention computation, the feature dimension of the Key is usually... .

[0136] Through the aforementioned cross-attention mechanism, each frame of the video can interact with the music features within its corresponding time window, extracting motion control signals from information such as music rhythm, melody, and intensity. Since the frame number of the music features is strictly consistent with the frame number of the video features, this mechanism achieves frame-by-frame audio-visual alignment, ensuring that the generated dance movements are strictly synchronized with the music beat.

[0137] In this embodiment, the frame-level alignment strategy significantly improves the synchronization quality between dance movements and music rhythm by constraining the length of music segments to be consistent with the number of video frames. This makes the generated dance video visually match the rhythm changes and dynamic fluctuations of the music, avoiding the audio-visual asynchrony problem commonly found in traditional methods.

[0138] S500: The text features and the reference image features are processed separately by decoupling the cross-attention mechanism. The text features are input to all cross-attention layers, and the reference image features are input to a specific cross-attention layer. The two attention outputs are then weighted and fused.

[0139] Furthermore, the step of processing the text features and the reference image features separately through the decoupled cross-attention mechanism includes:

[0140] The text features are passed through a first trainable projection layer to generate a first key-value pair, and the video features are subjected to global semantic control through a text cross-attention mechanism.

[0141] The reference image features are used to generate a second key-value pair through a second trainable projection layer. This pair is introduced only in specific cross-attention layers in the downsampling block, intermediate block, and upsampling block of the denoising network. The video features are then used to control the appearance of the person through an identity cross-attention mechanism.

[0142] The text cross-attention output and the identity cross-attention output are weighted and fused.

[0143] Specifically, after the audio and video synchronization alignment in step S400, the text conditions and the person reference image conditions are processed separately through a decoupled cross-attention mechanism. To solve the control interference problem caused by multimodal condition coupling, this invention processes the text conditions and reference image conditions separately in different paths of the U-Net, and achieves decoupled control through weighted fusion.

[0144] First, the text prompt via CLIP encoder in step S100 Projection as semantic embedding Key-value pairs are generated through the first trainable projection layer. This key-value pair is achieved through a text cross-attention mechanism. Constraining global semantic consistency and spatial layout enables control over high-level semantics such as dance style and scene content. The text cross-attention calculation is formally represented as follows:

[0145] ;

[0146] in, The query vector is obtained by projecting video features; Key-value pairs obtained by projecting text features; This is the parameter matrix of the first trainable projection layer.

[0147] Textual conditions are processed in all cross-attention layers of the denoising network, enabling textual semantics to exert global control over the entire video generation process, ensuring that the generated dance video meets the requirements of the textual description in terms of style, movement type, etc.

[0148] Static reference features The dynamic reference features obtained through MLP in step S100 Key-value pairs are generated through a second trainable projection layer. This key-value pair is achieved through an identity cross-attention mechanism. This achieves global identity preservation and character appearance control. The identity cross-attention calculation is formally represented as:

[0149] ;

[0150] in, The query vector is obtained by projecting video features; Key-value pairs obtained by projecting reference image features; This is the parameter matrix of the second trainable projection layer.

[0151] It should be noted that, unlike the global introduction of text conditions, reference image conditions are only introduced in specific cross-attention layers of the denoising network U-Net. In this embodiment, based on the StableDiffusion1.5 architecture, the reference image conditions are only introduced in specific cross-attention layers of the network. , and This is introduced in the cross-attention layer of these three blocks. This design allows the text pathway to focus on semantic alignment and the image pathway to focus on identity preservation, thereby improving control precision and identity consistency in complex dynamic generation, while avoiding excessive interference from reference image information on global semantic control.

[0152] Final output Weighted fusion is used to decouple the two-path attention, which can be formally represented as:

[0153] ;

[0154] in, This is an adjustable weight scalar used to balance the relative strength of text semantic control and character identity control. In this embodiment... This indicates that the two attention outputs are fused with equal weights. In practical applications, this can be adjusted according to specific needs. Value: Can be increased when a stronger identity preservation effect is needed. When stronger semantic control is required, the size can be reduced. .

[0155] Through the decoupled cross-attention mechanism described above, textual conditions and reference image conditions operate in their respective independent processing paths, effectively avoiding mutual interference between multimodal conditions. Textual conditions ensure the semantic accuracy and stylistic consistency of the generated video, while reference image conditions ensure the stability of the characters' appearance and identity consistency in the generated video. Together, they achieve high-quality, controllable dance video generation.

[0156] S600: The denoising network predicts the noise in the latent features after adding noise, calculates the loss between the predicted noise and the actual added noise, and optimizes the model parameters through backpropagation.

[0157] Furthermore, the loss between the calculated predicted noise and the actual added noise includes:

[0158] The mean square error between the predicted noise and the actual added noise is calculated as the loss function.

[0159] Specifically, the loss function drives the model to learn under given multimodal conditions (musical conditions). Reference images of people Text description Under these conditions, a high-quality, coherent, and synchronized dance video is reconstructed from noise. The loss function is calculated by predicting the noise level. With actual added noise The accuracy of the model prediction is measured by the mean squared error between the two sides, and the specific design is as follows:

[0160] ;

[0161] in, This represents the expectation operation, which averages the results over all training samples and randomly sampled time steps. Represents the initial latent features ; This represents the actual added noise sampled from a standard normal distribution; The time step for random sampling from a uniform distribution; This represents the square of the L2 norm, i.e., the mean square error. Let represent the prediction function of the denoising network, where This represents all the learnable parameters of the network.

[0162] This loss function ensures that the model can accurately predict noise components at different noise levels, thus enabling it to gradually recover a clear and coherent dance video from random Gaussian noise during the generation stage. By minimizing this loss, the model learns the correspondence between music rhythm and dance movements, methods for preserving character identity features, and visual expressions of text semantics.

[0163] In each training iteration, based on the calculated loss The loss is calculated using the backpropagation algorithm on the model parameters. gradient And update the model parameters using the optimizer:

[0164] ;

[0165] in, The learning rate is set to [value]. Through iterative optimization, the model parameters gradually converge, and the predictive ability of the denoising network continuously improves. Preferably, this embodiment uses the Adam optimizer, and the learning rate can be set to [value]. .

[0166] This method also includes a generation phase, such as Figure 3 As shown:

[0167] Obtain the music sequence, reference images of people, and text description of the video to be generated, and extract multimodal conditional features according to step S100;

[0168] Random Gaussian noise is initialized in the latent space as initial noise latent features, and the initial noise latent features and the multimodal conditional features are input into the trained denoising network.

[0169] The denoised latent features are obtained by iteratively denoising through the denoising network according to steps S300 to S500.

[0170] For long video generation, a video sequence corresponding to a music segment is generated each time. The last frame of the generated video sequence is used as the reference image for the next music segment. This process is repeated until the complete music sequence is processed.

[0171] The decoder, which uses a pre-trained variational autoencoder, decodes the denoised latent features into video frames.

[0172] Specifically, in the generation stage, the user inputs a music clip, a reference image of a character, and a text description as conditions. These inputs are then processed according to the method described in step S100 to extract features, resulting in music features. Dynamic reference features and text features .

[0173] Initialize random Gaussian noise in the latent space As a potential feature of initial noise, among which This represents the maximum number of time steps in the diffusion process. The initial noise, along with the extracted multimodal conditional features, is then input into the trained denoising network.

[0174] The denoising network follows the processing mechanism of the training phase, starting from the time step. The process begins with progressive denoising, where at each step, noise is predicted and the latent features are updated based on the current latent noise features and multimodal conditional features, until... To obtain the denoised latent features In this process, motion-focused attention mechanisms enhance movement coherence, audio-visual alignment strategies ensure rhythmic synchronization, and decoupled cross-attention mechanisms maintain identity consistency.

[0175] For long video generation, a sliding window strategy is adopted. After generating a video sequence corresponding to a music segment each time, the last frame of the sequence is used as the reference image for the next segment. This process is repeated until the entire music segment is processed, ensuring the temporal continuity of the long video.

[0176] Denoising the latent features The decoder, which uses a pre-trained variational autoencoder, decodes the video into pixel-space frames. Output the final dance video sequence.

[0177] Example 2: This example provides a specific application example of the dance video generation method based on motion-focused attention and decoupling control described in this invention.

[0178] In this embodiment, a 30-second clip of popular music is selected as input, a frontal photograph of a real person is selected as the reference image, and the text description is set as "a woman performing modern street dance".

[0179] First, the 30-second music track is divided into several fixed-length segments. MFCC features are extracted from each segment and processed through a one-dimensional convolutional network to obtain the music features. Features are extracted from the reference image using a CLIP image encoder and mapped to dynamic reference features via an MLP. Semantic features are extracted from the text description using a CLIP text encoder. Then, random Gaussian noise is initialized in the latent space and input into a trained denoising network along with the extracted multimodal conditional features. The denoising network performs differential shifting of the character region using a motion-focused attention mechanism, establishes frame-level correspondences through an audio-video alignment strategy, and processes text and image conditions separately through a decoupled cross-attention mechanism. A sliding window approach is used to generate a video sequence corresponding to each music segment, using the last frame of the generated segment as the reference image for the next segment, repeating this process until the entire music is processed. Finally, a VAE decoder decodes the latent features into video frames, outputting the complete dance video.

[0180] The generated dance video effect is as follows Figure 4 As shown. From the generated results, we can observe that:

[0181] Throughout all frames of the video, facial features, skin tone, hairstyle, and other physical characteristics of the individuals remained highly consistent, with no identity drift observed. This is attributed to the decoupled cross-attention mechanism, which introduces reference image conditions at specific cross-attention layers, achieving stable control over the individuals' identities.

[0182] The character's torso movements are smooth, and the large-amplitude swings of the limbs are fluid and continuous, without any disjointed or ghosting issues. The motion-focused attention mechanism effectively captures the movement characteristics of different body parts by employing a differentiated strategy of small-step shifts for the torso and large-step shifts for the limbs, while simultaneously reducing computational complexity. Down to .

[0183] The dance movements are precisely synchronized with the music rhythm, with increased amplitude during strong beats and smooth, flowing movements during gentler melodies. A frame-level audio-visual alignment strategy ensures a frame-by-frame correspondence between musical and video features, achieving strict synchronization between movement and rhythm.

[0184] The generated dance style conforms to the characteristics of "modern street dance," with dynamic and rhythmic movements that are highly consistent with the text description. The decoupled cross-attention mechanism enables semantic control of textual conditions globally, ensuring the semantic accuracy of the generated content.

[0185] This embodiment verifies that the method of the present invention can generate high-quality, consistent, coherent, and synchronized dance videos in practical applications, effectively solving the problems of character identity drift, discontinuous movements, and audio-visual asynchrony existing in the prior art.

[0186] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dance video generation method based on motion focused attention and decoupled control, characterized in that, The method comprises the following steps: S100: Obtain a music sequence, a character reference image and a text description, and extract music features, reference image features and text features respectively; S200: Add noise to a training video in a latent space to obtain a noisy latent feature, and input the noisy latent feature into a denoising network together with the music features, the reference image features and the text features; S300: In the denoising network, locate a character region in the noisy latent feature and divide it into different motion regions through a motion-focused attention mechanism, and aggregate historical frame information and perform attention calculation with current frame features in different motion regions by using a differentiated time shift step to obtain video features; S400: Align the music features and the video features at a frame level through a cross-attention mechanism; S500: Process the text features and the reference image features respectively through a decoupling cross-attention mechanism, input the text features into all cross-attention layers, input the reference image features into specific cross-attention layers, and perform weighted fusion on the outputs of the two attention mechanisms; S600: The denoising network predicts the noise in the noisy latent feature, calculates the loss between the predicted noise and the actual added noise, and optimizes the model parameters through back propagation; The division step of the different motion regions comprises: Locating a character center region on the noisy latent feature; Dividing the character center region into a center region and an edge region; The differentiated time shift step for different motion regions comprises: Performing a block shift operation on the feature blocks of the center region by using a first time shift step; Performing a block shift operation on the feature blocks of the edge region by using a second time shift step; The second time shift step is greater than the first time shift step.

2. The dance video generation method based on motion focus attention and decoupled control according to claim 1, characterized in that, The step of extracting music features comprises: Dividing the music sequence into a plurality of continuous segments of a fixed time length; Extracting a mel-frequency cepstral coefficient feature of each segment; Extracting the music features through a one-dimensional convolution network.

3. The dance video generation method based on motion focus attention and decoupled control according to claim 1, characterized in that, The step of extracting reference image features comprises: Encoding the character reference image using a pre-trained image encoder to obtain a static reference image feature; Expanding the static reference image feature in the time dimension; Mapping the expanded static reference image feature to a dynamic reference feature sequence through a multi-layer perceptron to obtain the reference image features.

4. The dance video generation method based on motion focus attention and decoupled control according to claim 1, characterized in that, The step of aggregating historical frame information and performing attention calculation with current frame features comprises: Performing a block shift operation on the overall frame features by using a plurality of different historical shift steps to obtain an overall mixed feature frame; Splicing the features of the center region and the edge region after the block shift to obtain a character motion mixed feature; Splicing the character motion mixed feature, the overall mixed feature frame and the current frame feature as the key and value of the attention mechanism, and taking the current frame feature as the query to perform attention calculation.

5. The dance video generation method based on motion focus attention and decoupled control according to claim 1, characterized in that, The step of aligning the music features and the video features at a frame level through a cross-attention mechanism comprises: The video features are spatially dimensionally reshaped to make the number of frames of the music features consistent with the number of frames of the video features; The cross-attention mechanism is used to establish the frame-by-frame correspondence between the music features and the video features.

6. The dance video generation method based on motion focus attention and decoupled control according to claim 1, characterized in that, The step of processing the text features and the reference image features respectively by decoupling the cross-attention mechanism comprises: The text features are generated into first key-value pairs by a first trainable projection layer, and the video features are globally semantically controlled by a text cross-attention mechanism; The reference image features are generated into second key-value pairs by a second trainable projection layer, and are only introduced in specific cross-attention layers in the down-sampling block, the intermediate block and the up-sampling block of the denoising network, and the video features are controlled in terms of appearance by an identity cross-attention mechanism; The text cross-attention output and the identity cross-attention output are weighted and fused.

7. The dance video generation method based on motion focus attention and decoupled control according to claim 1, characterized in that, The loss between the predicted noise and the actual added noise is calculated, comprising: The mean square error between the predicted noise and the actual added noise is calculated as the loss function.

8. The dance video generation method based on motion focus attention and decoupled control according to claim 1, characterized in that, It also includes a generation phase: The music sequence, the reference image of the character and the text description of the video to be generated are obtained, and the multi-modal conditional features are extracted according to step S100; Random Gaussian noise is initialized in the latent space as the initial noise latent feature, and the initial noise latent feature and the multi-modal conditional features are input into the trained denoising network; The denoising network is used to iteratively denoise the denoised latent features according to steps S300 to S500; For long video generation, one music segment corresponding video sequence is generated each time, the last frame of the generated video sequence is taken as the reference image of the next music segment, and the process is repeated until the complete music sequence is processed; The denoised latent features are decoded into video frames by the decoder of the pre-trained variational autoencoder.

Images