Audiovisual dance generation method and system based on diffusion model and cross-modal collaboration

Abstract

The present invention discloses a method and system for generating audiovisual dance based on diffusion model and cross-modal collaboration, belonging to the field of generative artificial intelligence. The method comprises: obtaining a dance video and a corresponding background music combination as a training set, respectively obtaining a low-dimensional embedding feature vector sequence through a cross-modal bidirectional collaborative representation variational autoencoder; constructing and training a diffusion model based on the Transformer architecture to learn the data distribution of audiovisual dance data; using the trained model in combination with the proposed classifier-free guided sampling to generate the corresponding low-dimensional embedding feature vector sequence; and finally realizing the audiovisual dance generation task through the corresponding decoder. The present invention realizes high-quality, high-matching, and long-sequence audiovisual dance generation by introducing cross-modal collaborative representation learning in a low-dimensional embedding space and an improved diffusion model with an attention mechanism as the main architecture.

Description

Technical Field

[0001] The present invention relates to the field of creative generative artificial intelligence, and in particular to an audiovisual dance generation method and system based on a diffusion model and cross-modal collaboration. Background Art

[0002] Generative AI aims to use AI algorithms to learn and identify data, and to generate relevant content through appropriate generalization capabilities. At present, some generative AI technologies have gained wide attention and application, such as generating corresponding pictures through text descriptions, generating corresponding anthropomorphic voices through given text fragments, and realizing conversational AI through large models. However, for the more artistic field of audio-visual dance, there is still no perfect work to achieve high-quality and high-matching audio-visual dance generation. In response to this challenging topic, the present invention proposes a model algorithm based on diffusion model and cross-modal collaboration.

[0003] In the field of audio-visual dance generation, the existing solutions in the world are only for one-way generation algorithms, such as generating dance postures given music or generating matching music given dance postures. In addition, the existing model algorithms cannot support high-quality generation effects for long sequences.

[0004] Specifically, existing technologies have the following shortcomings: 1) Existing technologies only support one-way generation and have poor flexibility. 2) Existing technologies do not perform collaborative learning on multimodal data, but simply use it as part of the conditional control input. 3) Existing technologies support limited generation lengths, such as audio or gesture sequences of about 5 seconds.

[0005] The technical problems solved by the present invention include three points:

[0006] 1. How to compress long sequences of audio and video data to reduce the difficulty of the model learning long sequence data, and improve the training efficiency and reasoning efficiency of the model, so as to achieve the generation of long sequences (more than 25 seconds).

[0007] 2. How to perform cross-modal knowledge collaborative representation learning on multimodal data (audio, video) to achieve highly matched bidirectional audiovisual dance generation.

[0008] 3. How to design a more effective generative model for long sequence data so that it can capture the temporal characteristics of long sequences and thus achieve high-quality audiovisual dance generation. Summary of the invention

[0009] In order to overcome the shortcomings of the prior art, the present invention provides an audio-visual dance generation method and system based on a diffusion model and cross-modal collaboration.

[0010] The specific technical solution adopted by the present invention is:

[0011] In a first aspect, the present invention proposes an audio-visual dance generation method based on a diffusion model and cross-modal collaboration, comprising the following steps:

[0012] 1) Obtain a dance video and audio data corresponding to the dance video, extract an audio Mel spectrum sequence for the audio data; extract human body posture motion joints for each frame of the dance video to form a posture motion sequence;

[0013] 2) constructing a variational autoencoder for the audio part and a variational autoencoder for the posture motion part, wherein the variational autoencoders are each composed of an encoder and a decoder;

[0014] 3) Input the audio Mel spectrum sequence into the variational autoencoder of the audio part, train the variational autoencoder of the audio part through the variational autoencoder reconstruction loss, and fix the trained variational autoencoder;

[0015] 4) Input the audio Mel spectrum sequence into the variational autoencoder of the trained audio part, and input the posture motion sequence into the variational autoencoder of the posture motion part, respectively output the low-dimensional embedding features of the posture motion sequence and the low-dimensional embedding features of the audio Mel spectrum sequence, perform cross-modal knowledge collaborative contrast learning on the low-dimensional embedding features of the two modalities to achieve alignment, and calculate the contrastive learning loss function based on the aligned low-dimensional embedding features; train the variational autoencoder of the posture motion part through the reconstruction loss of the variational autoencoder and the contrastive learning loss function;

[0016] 5) Construct a diffusion model based on the Transformer architecture; use the two trained variational autoencoders to obtain low-dimensional embedding features of the posture motion sequence and the audio Mel spectrum sequence, and use the obtained low-dimensional embedding features to train the diffusion model based on the Transformer architecture;

[0017] 6) Based on the trained diffusion model based on the Transformer architecture, combined with the classifier-free guided sampling strategy, audio generation based on human motion posture or audio-based human motion posture generation is realized.

[0018] In a second aspect, the present invention proposes an audio-visual dance generation system based on a diffusion model and cross-modal collaboration, which is used to implement the above-mentioned audio-visual dance generation method based on a diffusion model and cross-modal collaboration.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The present invention is a method for solving the audiovisual dance generation problem in creative generative artificial intelligence by using a diffusion model and cross-modal collaborative representation learning. In implementation, the present invention uses a diffusion model based on a Transformer architecture and cross-modal collaborative contrastive learning.

[0021] (1) Through cross-modal collaborative contrast learning, the present invention can effectively extract the intrinsic synchronization and matching relationship between dance movement beats and music rhythm. At the same time, by constructing a cross-modal collaborative bidirectional low-dimensional embedding space, the present invention can effectively solve the long-standing problems of long sequence generation and bidirectional synchronization generation in this field.

[0022] (2) By constructing a diffusion model with the Transformer architecture as the core, the model can be helped to better extract the temporal features of long-sequence data, thus achieving high-quality, high-matching, and long-sequence audio-visual dance generation. At the same time, by introducing the Transformer architecture, the problem of generating audio-visual dances of custom lengths can be effectively solved, which greatly improves the practicality and flexibility of the present invention.

[0023] In summary, by jointly using the diffusion model based on the Transformer architecture and cross-modal collaborative representation learning, the present invention can fully learn the synchronization and matching relationship between dance movements and the intrinsic rhythm of music, thereby achieving high-quality, high-matching, and long-sequence audiovisual dance generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall flow chart of the present invention;

[0025] Figure 2 It is the bidirectional collaborative variational autoencoder proposed in the present invention;

[0026] Figure 3 This is a performance comparison chart of the present invention and the existing most advanced algorithm LORIS in the task of audio generation based on human motion posture;

[0027] Figure 4 This is a comparison chart of subjective evaluation of the present invention with the most advanced algorithms currently available on the task of audio generation based on human motion posture;

[0028] Figure 5 This is a subjective evaluation comparison chart of the present invention with the most advanced algorithm currently available on the task of human motion posture generation based on audio input. DETAILED DESCRIPTION

[0029] The present invention will be further described and illustrated below in conjunction with the accompanying drawings and specific implementation methods.

[0030] The present invention discloses a method for solving the audio-visual dance generation problem in creative generative artificial intelligence by using a diffusion model and cross-modal collaborative representation learning, comprising the following steps:

[0031] Step 1: obtain a dance video and audio data corresponding to the dance video as a training set, extract the Mel spectrum features of the audio data, obtain an audio Mel spectrum sequence, extract the human body posture motion joints in the dance video through the OpenPose open source library, and all the human body posture motion joints extracted from the dance video form a posture motion sequence;

[0032] Step 2, constructing a multimodal variational autoencoder, including a variational autoencoder for the audio part and a variational autoencoder for the posture motion part; wherein the variational autoencoder includes an encoder and a decoder, which can adopt a common codec structure, which is well known to those skilled in the art and will not be described in detail in the present invention;

[0033] Step 3, train the encoder and decoder of the audio part through the variational autoencoder reconstruction loss;

[0034] Step 4: fix the trained audio encoder, train the encoder and decoder of the posture motion part through the variational autoencoder reconstruction loss and the cross-modal collaborative contrastive learning loss function based on the posture spectrogram to construct a low-dimensional embedding space for cross-modal collaborative representation;

[0035] Step 5: construct a diffusion model based on the Transformer architecture (DiT model); use the posture motion obtained by the trained multimodal variational autoencoder and the low-dimensional embedded feature vector of the audio to train the diffusion model based on the Transformer architecture;

[0036] Step 6: Based on the trained DiT model, the classifier-free guided sampling strategy is combined to achieve cross-modal audiovisual dance generation.

[0037] The specific process of extracting the posture motion sequence of the dance video and the audio Mel spectrum sequence of the audio data in the above step 1 is as follows.

[0038] For dance videos, the present invention extracts 2D human key points and processes videos at a speed of 60 frames per second. The pre-trained Body-25 model is used to extract 25 key points of the human body. Because some key points are difficult to extract continuously and stably, and some are not very relevant to dance movements. Therefore, the present invention finally selects 14 most relevant key points to represent postures, including nose, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees and left and right ankles. In order to ensure that there are no missing key points in all extracted segments, the key points of missing detection will be interpolated from adjacent frames. These 14 key points are defined as human posture motion joints in the present invention. For music files, the present invention uses Mel spectrograms as audio feature representations. First, the audio is resampled to 16kHz. When processing the Mel spectrogram, 80 filters are used, and the fast Fourier transform parameters are set to 1024, the jump length is set to 256, and the Mel spectrum is processed using a Hann window of size 1024.

[0039] In the above step 2, the multimodal variational autoencoder is constructed as follows:

[0040] Encoder of the posture motion part: Set is a motion posture sequence, which contains all the human body motion joints extracted from the corresponding video, where T m is the number of motion frames, and J is the number of nodes containing x-coordinate values ​​and y-coordinate values. Then, the present invention encodes the spatial position of the joint points of the human posture motion into a low-dimensional embedding space representation Where m is the downsampled motion pose sequence, T zm <T m is the number of downsampled motion frames, d is the low-dimensional spatial dimension of the posture data, and the low-dimensional embedding feature z of the motion posture sequence m Output of the variational autoencoder as the pose motion part.

[0041] Audio codec: is the audio data, where T u Indicates the waveform length. The present invention can extract the Mel-spectrogram of audio data: Where Mel is a predefined Mel spectrogram extraction function, C a is the number of channels, and T a <<T u is the number of audio frames. Then, the encoder is used to embed the Mel spectrogram into a low-dimensional space to obtain a low-dimensional embedding feature Where T za is the number of downsampled audio frames, d' is the dimension of the embedded Mel spectrum graph, d' = d, and the low-dimensional embedding feature z of the audio Mel spectrum sequence a The output of the variational autoencoder as the audio part.

[0042] Decoder: The two decoders in the multimodal variational autoencoder can use a universal decoder to calculate the variational autoencoder reconstruction loss during the training phase. The low-dimensional embedding features of the motion posture sequence output by the variational autoencoder of the posture motion part can be reconstructed into the corresponding posture motion sequence through a decoder: m′=D m (m). The low-dimensional embedding features of the audio Mel spectrum sequence output by the variational autoencoder of the audio part can be reconstructed into the corresponding audio Mel spectrum sequence a′=D through another decoder a (a).

[0043] The above steps 3 and 4 are two-stage training of the multimodal variational autoencoder. In the present invention, the goal is to learn two cross-modal collaborative variational autoencoders for low-dimensional embedding space learning of visual posture motion and audio input signals, and to achieve timing and rhythm alignment in the low-dimensional embedding space. However, the goal of maintaining fine-grained details in the variational autoencoder to achieve accurate reconstruction conflicts with the goal of contrastive learning. In other words, there is a trade-off between the restoration fidelity of low-dimensional features and cross-modal alignment learning. To solve this problem, the present invention proposes a two-stage training strategy: first, the present invention uses the reconstruction loss function of the variational autoencoder to train the variational autoencoder of the audio part; then, the present invention trains the variational autoencoder for human motion posture data in the second stage. The training at this stage will use the variational autoencoder reconstruction loss function and the above-mentioned contrastive learning loss function, in which the variational autoencoder parameters of the music part will be fixed. The insight behind this strategy is that the Mel-spectrogram, with its rich and complex acoustic features, requires a more complex optimization process than processing limited body joint data. The variational autoencoder based on cross-modal collaborative representation learning of the present invention is summarized as follows Figure 2 shown.

[0044] In step 3, the first training stage is to train the variational autoencoder of the audio part through the variational autoencoder reconstruction loss, wherein the variational autoencoder reconstruction loss is to decode and reconstruct the low-dimensional embedding features output by the variational autoencoder into a corresponding posture motion sequence or audio mel spectrum sequence through a decoder, and measure the difference between the original input sequence and the reconstructed sequence. Calculating the variational autoencoder reconstruction loss is a method well known to those skilled in the art.

[0045] In step 4, the second training stage is to train the variational autoencoder of the posture motion part through the variational autoencoder reconstruction loss and the cross-modal co-contrastive learning loss function based on the posture spectrogram. The variational autoencoder reconstruction loss described in this stage is calculated in the same way as the reconstruction loss in the first training stage.

[0046] To obtain the cross-modal collaborative contrastive learning loss function based on the posture spectrogram, it is necessary to realize rhythm beat contrastive learning by constructing the posture spectrogram. The specific steps are as follows:

[0047] 4.1) Based on the variational autoencoder of visual motion gesture and audio, the present invention can obtain low-dimensional embedding representation of motion gesture sequence respectively and low-dimensional embedding representation of audio mel-spectrogram sequences In order to synchronize visual gesture motion and audio data sampled in different ways, the present invention adopts preprocessing techniques, such as resampling the gesture motion sequence to match the length of the audio Mel spectrum sequence to ensure that T zm =T za .

[0048] 4.2) In the field of audio-visual dance generation, due to the inherent irregularity of human movement, its rapid and sudden movements can often significantly affect the rhythm of the music. In order to synchronize these rhythmic patterns, the present invention uses a posture motion amplitude index as the basis for constructing cross-modal knowledge collaborative comparative learning. First, the present invention uses a metric to quantify motion changes, namely, the posture spectrogram extracts the displacement of human motion over time, which is defined as the motion amplitude index. Let F(r,j) be the first-order difference of the jth joint point in the 2D posture motion at time r, and through tan -1 (y / x) is divided into K intervals according to its Euclidean angle with the x-axis, where x represents the coordinate of F(r,j) on the x-axis and y represents the coordinate of F(r,j) on the y-axis. Then, the posture spectrum D(r,θ) of 2D motion can be expressed as the aggregation of F(r,j) in each angle interval:

[0049]

[0050] Where D(r,θ) is the posture spectrum of time r and angle interval θ, J is the number of joint points of human posture motion, ‖·‖ 2 represents the L2 norm, 1 θ (∠F(r,j)) is the indicator function. When |θ-∠F(r,j)|≤2π / K, its value is 1, otherwise it is 0. ∠F(r,j) represents the angle corresponding to the first-order difference of the j-th joint point in the 2D posture motion at time r. The indicator function distributes the joint points of the human body posture motion into K angle intervals.

[0051] 4.3) Then, the motion amplitude index is calculated by summing the pose spectrogram differences in each angle interval:

[0052]

[0053] Where D(r,k) is the attitude spectrum at time r and the kth angle interval.

[0054] Furthermore, the steps of cross-modal collaborative contrastive learning are as follows:

[0055] 4.4) After establishing the motion amplitude index based on the posture spectrogram, the present invention proposes a cross-modal collaborative representation learning method to mine the potential rhythm alignment information between human motion and music beat. zm =T za There is a one-to-one correspondence between the low-dimensional embedding features of the preprocessed posture motion sequence and the low-dimensional embedding features of the audio Mel spectrum sequence. The present invention refers to the low-dimensional embedding features at the same time as paired low-dimensional features. For all paired low-dimensional features of posture motion and audio, N are randomly sampled according to the time dimension. T The slice lengths are consistent. Each gesture motion-audio slice contains gesture motion slices and audio slices of the same length. Calculate the maximum motion amplitude index Q(r s :r e ), and N T The slices are divided into N according to the size of the maximum motion amplitude index of the slice. C In order to maximize the similarity of gesture motion and audio feature pairs at the same time (i.e., time alignment) and minimize the similarity of gesture motion and audio feature pairs from different time and different rhythm features, the present invention uses N T Randomly sample N S Slices of different pose motion and audio feature pairs for timing and cross-modal rhythm alignment:

[0056]

[0057] Q(r s :r e )=max(Q(r s ):Q(r e ))

[0058] where r s and r e Respectively represent the start and end time of the sampling slice, represents the gesture motion slice, Indicates that from r s Time to e Slices of the low-dimensional embedding representation of the music mel-spectrogram at time instant, Represents an audio slice, Indicates from r s Time to e A slice of the low-dimensional embedding representation of the motion posture sequence at the moment, Q(r s :r e ) indicates that from r sTime to e The movement amplitude index at the moment, Q(r s )、Q(r e ) represent r s The movement range index at the moment and r e The movement amplitude index at the moment, P max Represents the maximum pooling operation in the time dimension.

[0059] 4.5) Sampling-based gesture motion-audio slices in Represents an audio slice of category i, represents the pose motion slice of category i. The contrastive learning loss function of cross-modal collaborative representation is:

[0060]

[0061] Among them, exp represents the square root, sim is the similarity measurement function between cross-modal low-dimensional embedding features, and τ is the contrastive learning temperature parameter, which is set to 1.0 here.

[0062] In step 5, the diffusion transformer model (DiT) based on the transformer architecture is an existing model in the prior art. By introducing noise disturbance and training the neural network to reverse the process of noise increase, the generation of motion posture or audio is achieved. The specific steps of training the DiT model are as follows:

[0063] During the forward diffusion process of the DiT model, the embedded low-dimensional data is gradually perturbed toward the standard Gaussian distribution. This process of adding noise perturbation is based on the predefined perturbation parameter α 1 ,…,α T To control, where T is the total number of diffusion steps,

[0064]

[0065] where z a (t) represents the low-dimensional embedding feature of the audio when the diffusion step is t, q(z a (t)|z a (t-1)) means, represents the standard Gaussian distribution and I is the identity matrix.

[0066] The training objectives of the diffusion model based on the Transformer architecture are:

[0067]

[0068] in represents the noise in the forward diffusion process, θ a represents the parameterized DiT model, represents the noise predicted by the model when the diffusion step number is t, z a (t) represents the parameterized Transformer model when the number of diffusion steps is t, Represents the square of the L2 norm.

[0069] In step 6, the specific steps of using classifier-free guided sampling and cross-guided sampling to generate audio-visual dance samples are as follows, taking the audio generation task based on human motion posture as an example:

[0070] 6.1) For the task of generating audio based on human motion posture, the input posture motion sequence is extracted through the encoder of the trained posture motion part to obtain the low-dimensional embedding feature z of the motion posture sequence m ;

[0071] 6.2) The generation process based on the diffusion model adopts an iterative denoising method. Specifically, a random noise is first sampled from a standard Gaussian distribution; then, the low-dimensional embedding feature z of the motion posture sequence is used. m As a conditional control, the classifier-free guided strategy is combined to perform iterative denoising on the sampled random noise to finally obtain the low-dimensional embedding features of the target audio. The classifier-free guided strategy is expressed as follows:

[0072]

[0073] in, It represents the output result of the model when the diffusion step number is t. represents an unconditionally controlled tensor (in a specific implementation, it can be a tensor with all zero values), T≥t>0, s>1 is a hyperparameter used to control the sampling strategy, which is used to balance the diversity and quality of the synthesized samples. In the process of sampling steps from T to 0, the random noise is gradually transformed into the desired audio low-dimensional embedding features.

[0074] 6.3) After the iterative denoising process of the diffusion model, the motion posture embedding feature z can be obtained m The low-dimensional embedding feature z of the target audio under control a Then, the audio decoder is used to generate a low-dimensional embedding representation of the generated audio Mel spectrum sequence z a The generated audio mel-spectrogram sequence is decoded and then passed through a pre-trained high-fidelity vocoder to obtain the generated audio that matches the input gesture motion sequence;

[0075] Since the present invention constructs a low-dimensional embedding space for cross-modal collaborative representation, a symmetrical task, namely, human posture generation based on audio input, can be achieved by exchanging low-dimensional embedding features. Similar to the above-mentioned audio generation task based on human motion posture, for the human motion posture generation task based on audio, the input audio Mel spectrum sequence is extracted by the encoder of the trained audio part to obtain the low-dimensional embedding feature z of the audio Mel spectrum sequence. a , with z a The diffusion model combines the classifier-free guidance strategy to iteratively denoise the random noise and finally obtain the low-dimensional embedding feature z of the motion posture sequence. m , z m Then, through the decoder of the posture motion part, a motion posture sequence matching the input audio Mel spectrum sequence is obtained.

[0076] The above method is applied to the following embodiments to demonstrate the technical effects of the present invention, and the specific steps in the embodiments are not repeated here.

[0077] The present invention has been fully experimentally verified on two cross-modal generation tasks.

[0078] Specifically, for the task of audio generation based on human body motion posture, the present invention conducts experiments on three real video datasets based on human body posture movement: AIST++Dance, Floor Exercise, and Figure Skating. The total duration of the video data exceeds 80 hours. In order to objectively evaluate the performance of the present invention, the present invention uses the beat coverage score (BCS) and the beat hit score (BHS) in the corresponding test set to verify the rhythm alignment and cross-modal alignment of the synthesized audio, and their standard deviations CSD and HSD are also taken into account to test the stability of the present invention. In combination with BCS and BHS, the present invention uses the F1 score as a comprehensive evaluation indicator for beat alignment. In order to reflect the advancement of the present invention, it is compared with the following best existing technical models:

[0079] Compare 1. Foley Music model, which is a graph Transformer framework based on music MIDI feature representation;

[0080] Comparison 2. CMT model, a Transformer-based model for controllable music generation, used to learn rhythmic consistency between video and music;

[0081] Compare 3.D2M-GAN model, which is a model based on adversarial generative network and uses vector quantized music representation;

[0082] Comparison 4. CDCD model, which is a diffusion model that introduces contrastive learning loss with conditional control and has achieved excellent results in the field of gesture-guided music generation;

[0083] Compare 5. LORIS model, which is a diffusion model with multi-layer conditional control mechanism, and has reached the state-of-the-art level in the field of gesture-guided music generation.

[0084] According to the steps described in the specific implementation, the experimental results are shown in Tables 1 to 5. ↑ means that the larger the value of the indicator, the better, and ↓ means that the smaller the value of the indicator, the better.

[0085] Table 1: Test results of the audio generation task based on human motion posture obtained by the present invention for the AIST++ dataset

[0086]

[0087]

[0088] Table 2: Test results of the audio generation task based on human motion posture obtained by the present invention for the Floor Exercise (25 second slice) dataset

[0089] Model BCS↑ CSD↓ BHS↑ HSD↓ F1↑ Foley Music 36.0 36.2 32.3 15.0 57.5 CMT 46.4 30.1 57.4 18.6 62.6 D2M-GAN 45.3 27.7 58.7 19.0 93.1 CDCD 49.0 21.1 61.0 18.1 92.7 LORIS 58.8 19.4 67.1 13.9 94.5 The present invention 66.6 14.3 76.9 2.8 98.1

[0090] Table 3: Test results of the audio generation task based on human motion posture obtained by the present invention for the Floor Exercise (50-second slice) dataset

[0091] Model BCS↑ CSD↓ BHS↑ HSD↓ F1↑ Foley Music 32.6 38.0 28.4 32.5 30.4 CMT 42.3 32.0 53.8 31.7 47.4 D2M-GAN 41.9 29.2 54.7 32.7 47.5 CDCD 45.9 23.8 57.5 29.3 51.0 LORIS 54.7 21.6 63.8 24.5 58.9 The present invention 62.7 24.0 68.1 20.2 65.3

[0092] Table 4: Test results of the audio generation task based on human motion posture obtained by the present invention for the Figure Skating (25 second slice) dataset

[0093] Model BCS↑ CSD↓ BHS↑ HSD↓ F1↑ Foley Music 36.0 36.2 32.3 30.7 34.1 CMT 46.4 30.1 57.4 29.8 51.3 D2M-GAN 45.3 27.7 58.7 30.1 51.1 CDCD 49.0 21.1 61.0 27.0 57.3 LORIS 58.8 19.4 67.1 21.1 62.7 The present invention 63.5 16.3 75.6 28.8 69.0

[0094] Table 5: Test results of the audio generation task based on human motion posture obtained by the present invention for the Figure Skating (50 second slice) dataset

[0095] Model BCS↑ CSD↓ BHS↑ HSD↓ F1↑ Foley Music 32.6 38.0 28.4 32.5 30.4 CMT 42.3 32.0 53.8 31.7 47.4 D2M-GAN 41.9 29.2 54.7 32.7 47.5 CDCD 45.9 23.8 57.5 29.3 21.0 LORIS 54.7 21.6 63.8 24.5 58.9 The present invention 59.9 22.6 68.7 22.2 64.0

[0096] In order to detect the quality and diversity of the generated audio, this embodiment further uses the FAD score and the Diversity score. Figure 3The performance comparison between the present invention and the existing most advanced algorithm LORIS on the task of audio generation based on human motion posture is shown. DANCE represents the AIST++Dance dataset, FE25 represents the Floor Exercise (25-second slice) dataset, FE50 represents the Floor Exercise (50-second slice) dataset, FS25 represents the Figure Skating (25-second slice) dataset, and FS50 represents the Figure Skating (50-second slice) dataset. Figure 3 It can be seen that the performance of the present invention in FAD score and Diversity score is significantly better than that of LORIS algorithm.

[0097] For the task of generating human motion postures based on audio, the present invention conducts experiments on two real video data sets AIST++Dance and BHSDance, which are mainly based on human motion postures. The total duration of the video data is approximately 80 hours. In order to objectively evaluate the performance of the present invention, the present invention has continued to use the above-mentioned beat coverage score (BCS), beat hit score (BHS) and F1 score in the corresponding test set to verify the rhythm alignment and cross-modal alignment of the synthesized audio, and their standard deviations CSD and HSD are also taken into account to test the stability of the present invention. In order to detect the quality and diversity of the generated motion postures, this paper further uses FID scores, Mean KLD scores and Diversity scores. In order to reflect the advancement of the present invention, it is compared with the following existing best technical models:

[0098] Compare 1.D2M, a two-stage algorithm framework for generating motion gestures based on audio. The framework first divides the motion gesture into several basic motion units, and then combines them into a complete motion gesture. This two-stage process of decomposition and recombination is implemented by variational autoencoders and adversarial generative networks respectively.

[0099] Compare 2. DiffGesture, a recent work on speech-guided gesture generation based on diffusion model, which we generalize to the audio-guided gesture generation task.

[0100] According to the steps described in the specific implementation, the experimental results are shown in Tables 6 to 9. ↑ means that the larger the value of the indicator, the better, and ↓ means that the smaller the value of the indicator, the better.

[0101] Table 6: Test results of the audio-based human motion gesture generation task obtained by the present invention for the AIST++Dance dataset

[0102] Model BCS↑ CSD↓ BHS↑ HSD↓ F1↑ D2M 23.7 13.8 42.8 23.6 30.5 DiffGesture 28.5 16.7 40.4 25.7 33.4 The present invention 39.2 10.2 56.3 12.0 46.2

[0103] Table 7: Test results of the audio-based human motion posture generation task obtained by the present invention for the BHSDance dataset

[0104] Model BCS↑ CSD↓ BHS↑ HSD↓ F1↑ D2M 35.1 15.9 57.5 35.0 43.6 DiffGesture 42.8 21.3 61.1 23.9 50.3 The present invention 47.9 8.4 78.5 12.1 59.5

[0105] Table 8: Test results of the audio-based human motion gesture generation task obtained by the present invention for the AIST++Dance dataset

[0106]

[0107] Table 9: Test results of the audio-based human motion posture generation task obtained by the present invention on the BHSDance dataset

[0108]

[0109] It can be found from Tables 6 to 9 that the present invention is significantly superior to the existing most advanced audio-based human posture generation model in terms of beat matching as well as generation quality and diversity.

[0110] In order to further verify the effectiveness of the present invention, this paper provides corresponding subjective evaluation indicators. Specifically, the present invention will be compared with LORIS, the most advanced algorithm for generating audio from posture, and D2M, the most advanced algorithm for generating posture from audio. The subjective evaluation invited 20 professionals to evaluate the generation performance of different methods on the AIST++ dance dataset. Each method first generated 200 samples, and then randomly selected 20 paired samples for each comparison group for evaluation. The subjects were asked to answer the following evaluation question on the spot: "Which dance / music is more realistic and matches the music / dance better?". The corresponding evaluation results are shown as follows. Figure 4 As shown. Figure 4 The results show that the present invention significantly outperforms the most advanced models in both tasks. At the same time, the variational autoencoder used in the present invention for constructing cross-modal collaborative representation learning plays a key role.

[0111] In this embodiment, an audio-visual dance generation system based on a diffusion model and cross-modal collaboration is also provided, which is used to implement the above embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0112] A dance video preprocessing module, which is used to extract the posture sequence of human body movements in dance videos;

[0113] A music file preprocessing module, which is used to obtain the Mel spectrum features of the music file;

[0114] The motion posture variational autoencoder module is used to obtain the low-dimensional embedding feature vector of the motion posture sequence;

[0115] Mel spectrum variational autoencoder module, used to obtain the low-dimensional embedding feature vector of Mel spectrum;

[0116] The vocoder module is used to convert the Mel spectrum into the corresponding sound waveform;

[0117] The diffusion model decoder module based on the Transformer architecture is used to learn the corresponding data distribution according to the low-dimensional embedding vectors of motion posture and Mel spectrum in the training phase; and to generate audio-visual dance samples by relying on sampling strategies in the generation phase.

[0118] The training module is divided into three parts: the first is to train the corresponding multimodal variational autoencoder by combining the cross-modal collaborative contrastive learning loss function and the reconstruction loss function of the variational autoencoder; the second is to train the vocoder module according to the Mel spectrum and sound waveform; the third is to train the diffusion model decoder module according to the loss function of the diffusion model to fit the data distribution of the low-dimensional embedding feature vector of the audio-visual dance data.

[0119] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0120] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0121] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A method for audio-visual dance generation based on diffusion model and cross-modal collaboration, characterized in that: The steps include: 1) Obtain a dance video and audio data corresponding to the dance video, extract an audio Mel spectrum sequence for the audio data; extract human body posture motion joints for each frame of the dance video to form a posture motion sequence; 2) constructing a variational autoencoder for the audio part and a variational autoencoder for the posture motion part, wherein the variational autoencoders are each composed of an encoder and a decoder; 3) Input the audio Mel spectrum sequence into the variational autoencoder of the audio part, train the variational autoencoder of the audio part through the variational autoencoder reconstruction loss, and fix the trained variational autoencoder; 4) Input the audio Mel spectrum sequence into the variational autoencoder of the trained audio part, and input the posture motion sequence into the variational autoencoder of the posture motion part, respectively output the low-dimensional embedding features of the posture motion sequence and the low-dimensional embedding features of the audio Mel spectrum sequence, perform cross-modal knowledge collaborative contrast learning on the low-dimensional embedding features of the two modalities to achieve alignment, and calculate the contrastive learning loss function based on the aligned low-dimensional embedding features; train the variational autoencoder of the posture motion part through the reconstruction loss of the variational autoencoder and the contrastive learning loss function; 5) Construct a diffusion model based on the Transformer architecture; use the two trained variational autoencoders to obtain low-dimensional embedding features of the posture motion sequence and the audio Mel spectrum sequence, and use the obtained low-dimensional embedding features to train the diffusion model based on the Transformer architecture; 6) Based on the trained diffusion model based on Transformer architecture, combined with the classifier-free guided sampling strategy, audio generation based on human motion posture or audio-based human motion posture generation is realized; In step 2), the encoder in the variational autoencoder of the posture motion part is used to convert the posture motion sequence Encode into a low-dimensional embedding space to obtain a low-dimensional embedding representation of the posture motion sequence Where T m is the number of motion frames, J is the number of joint points of human body posture motion, T zm <T m is the number of downsampled motion frames, d is the low-dimensional space dimension of the posture data; the encoder in the variational autoencoder of the audio part is used to convert the audio Mel spectrum sequence Encode into a low-dimensional embedding space to obtain a low-dimensional embedding representation of the audio Mel spectrum sequence Among them C a is the number of channels, T a <<T u is the number of audio frames, T u is the waveform length, T za is the number of downsampled audio frames, d′ is the dimension of the encoded Mel-spectrogram; In step 4), the cross-modal knowledge collaborative comparative learning is performed on the low-dimensional embedding features of the two modalities to achieve alignment, specifically: Adjusting low-dimensional embedding representations of pose motion sequences and low-dimensional embedding representation of audio mel-spectrogram sequences The number of frames, so that T zm =T za ; The low-dimensional embedding features of the posture motion sequence and the low-dimensional embedding features of the audio Mel spectrum sequence at the same time are used as paired low-dimensional features, and the paired low-dimensional features are randomly sampled to obtain N T gesture motion-audio slices, each gesture motion-audio slice is composed of a gesture motion slice and an audio slice, and the maximum motion amplitude index of the gesture motion-audio slice is calculated based on the gesture spectrogram; In N T N gesture movements are randomly sampled and pooled from the audio slices S Gesture Motion-Audio Slices Achieve timing and cross-modal rhythm alignment: where r s and r e Respectively represent the start and end time of the gesture motion-audio slice, represents the pooled pose motion slice, Indicates that from r s Time to e The low-dimensional embedding representation of the audio Mel spectrum sequence at time instant, Represents the pooled audio slice, Indicates that from r s Time to e The low-dimensional embedding representation of the motion posture sequence at each moment, P max Represents the maximum pooling operation in the time dimension; In step 4), the specific calculation method of the maximum motion amplitude index of the gesture motion-audio slice is as follows: Get the first-order difference F(r,j) of the jth joint point in the motion posture sequence in the posture motion-audio slice at time r, and divide F(r,j) into K angle intervals according to its Euclidean angle with the x-axis. The posture spectrum at time r and angle interval θ is: Where J is the number of joints of human body posture motion, ‖.‖2 represents the binary norm, 1 θ (∠F(r,j)) is the indicator function. When |θ-∠F(r,j)|≤2π / K, the indicator function value is 1, otherwise it is 0. ∠F(r,j) represents the angle corresponding to the first-order difference of the j-th joint point at time r. ‖.‖2 is the L2 norm, and D(.) is the posture spectrum. The motion amplitude index Q(r) is calculated by summing the attitude spectrogram differences in each angle interval: The maximum motion amplitude index calculated within the time period of the posture motion-audio slice is the maximum motion amplitude index of the posture motion-audio slice; In step 4), the contrastive learning loss function is calculated based on the aligned low-dimensional embedding features as follows: The gesture motion-audio slices are divided into N groups according to the size of their maximum motion amplitude index. C categories; Redefine the gesture motion-audio slice according to the category of gesture motion-audio slice, and record the audio slice of category i as The gesture motion slice of category j is recorded as The contrastive learning loss between the two is: Among them, exp represents the quadratic root, sim is the similarity measurement function between low-dimensional embedded features, Indicates audio slices of category i and c. represents the gesture motion slice of category j, c, τ is the contrastive learning temperature parameter, N C is the number of categories.

2. The audio-visual dance generation method based on diffusion model and cross-modal collaboration according to claim 1 is characterized in that: In step 1), for each frame of the dance video, the human body posture motion joint points are extracted by the following extraction method: For each frame image in the dance video, 2D human key points are extracted, and 14 2D human key points representing the nose, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles are selected as human posture motion joints; if the current image lacks a human posture motion joint, the missing human posture motion joint is interpolated from the adjacent frames.

3. The audio-visual dance generation method based on diffusion model and cross-modal collaboration according to claim 1 is characterized in that: In step 5), the training of the diffusion model based on the Transformer architecture adopts the following loss function: in, represents the noise of the diffusion model in the forward diffusion process, represents the noise predicted by the diffusion model when the number of diffusion steps is t, Represents the square of the L2 norm.

4. The audio-visual dance generation method based on diffusion model and cross-modal collaboration according to claim 1 is characterized in that: The method of combining the non-classifier guided sampling strategy to realize the audio generation based on human motion posture or the human motion posture generation based on audio is specifically as follows: 6.1) For the input posture motion sequence or audio mel spectrum sequence, use the encoder in the trained strain autoencoder to extract features to obtain the low-dimensional embedding features of the input modality; 6.2) Obtaining a diffusion model based on a Transformer architecture trained with random noise input, taking the low-dimensional embedding features of the input modality as a conditional control, the diffusion model is combined with a classifier-free guidance strategy to iteratively denoise the random noise to finally obtain the low-dimensional embedding features of the target modality; 6.3) For the low-dimensional embedded features of the target modality, the decoder in the trained corresponding strain autoencoder is used to decode to obtain a generated audio Mel spectrum sequence or a posture motion sequence; if an audio Mel spectrum sequence is generated, the generated audio Mel spectrum sequence is passed through a vocoder to obtain the generated audio.

5. An audio-visual dance generation system based on diffusion model and cross-modal collaboration, characterized in that: The system for implementing the audio-visual dance generation method based on diffusion model and cross-modal collaboration as described in any one of claims 1 to 4 comprises: Dance video preprocessing module, used to extract the posture motion sequence of dance videos; An audio data preprocessing module, used to obtain an audio Mel spectrum sequence of audio data; Motion posture variational autoencoder module, used to obtain low-dimensional embedding features of motion posture sequences; Mel spectrum variational autoencoder module, used to obtain low-dimensional embedding features of audio Mel spectrum sequences; The diffusion model decoder module based on the Transformer architecture is used to learn the corresponding data distribution according to the low-dimensional embedding representation of motion posture and audio Mel spectrum sequence in the training phase, and to generate audio based on human motion posture and generate human motion posture based on audio by relying on the sampling strategy based on the low-dimensional embedding features of motion posture sequence and audio Mel spectrum sequence in the generation phase; The vocoder module is used to convert the audio mel-spectrogram sequence into a sound waveform in the generation phase; The training module is used to train the motion posture variational autoencoder module, the Mel spectrum variational autoencoder module, and the diffusion model decoder module based on the Transformer architecture.

Images