Voice-driven face video generation method and device based on depth perception fusion

Abstract

The invention discloses a voice-driven face video generation method and device based on depth perception fusion, and relates to the field of computer vision and image processing, and the method comprises the steps: S1, obtaining a face speaking video data set with an audio clip and a reference image, carrying out the preprocessing of the data set, and dividing the data set into a training data set and a test data set; s2, constructing a face video generation model; comprising an audio encoder, an image encoder, a depth encoder, a cross reference module and a cross-modal attention module. S3, training a face video generation model by using the training data set to obtain a trained face video generation model; and S4, inputting the test data set into the trained face video generation model, and outputting the generated face video combining the audio and the video. According to the method, the cross reference module and the cross-modal attention module are introduced into the face video generation model, so that the fine-grained details of motion are considered while the accuracy of the face structure of the face video is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing, and in particular to a method and device for generating voice-driven face videos based on depth perception fusion. Background Art

[0002] Speech-driven face video generation technology is a cutting-edge technology that combines speech processing and computer vision. Its main goal is to generate high-quality human videos that match speech input by analyzing and understanding it. In this field, researchers not only need to overcome the semantic gap between speech and video, but also need to solve the problems of timing synchronization and matching of action expressions between speech and video.

[0003] At present, most studies mainly rely on learning 2D facial representations (such as appearance and motion) from input images, but ignore the use of 3D facial geometric structure relationships in images. There is insufficient perception and exploration of facial depth information, and there is a lack of corresponding 2D-3D feature fusion optimization research, which results in the generated video facial details are not vivid, limiting the promotion and application of face video generation in actual scenarios. Summary of the invention

[0004] In response to the problems raised in the above background technology, the present invention proposes a voice-driven face video generation method and device based on depth perception fusion, constructs a face video generation model, constructs a cross-reference module according to facial motion attributes, mines the rich semantic information and texture information in the RGB image and the key depth information in the depth map, and combines with a cross-modal attention learning network to effectively improve the facial structure accuracy of the face video while taking into account the fine-grained details of the motion, making it more realistic and natural.

[0005] On the one hand, the voice-driven face video generation method based on depth perception fusion has the following specific steps:

[0006] S1, obtain a face speaking video dataset with audio clips and reference images, and divide the dataset into a training dataset and a test dataset after preprocessing;

[0007] S2, constructing a face video generation model; the face video generation model includes an audio encoder, an image encoder, a depth encoder, a cross-reference module and a cross-modal attention module; the audio encoder extracts the Melp features of the audio in the data set; the image encoder extracts the RGB features of the image in the data set; the depth encoder extracts the depth map features of the image in the data set; the cross-reference module fuses the depth map features and the RGB features; the cross-modal attention module introduces a self-attention mechanism to enhance the ability to retain facial structure;

[0008] S3, using the training data set to train the face video generation model to obtain a trained face video generation model;

[0009] S4, input the test dataset into the trained face video generation model, and output the face video combining audio and video.

[0010] Preferably, the cross-reference module is as follows:

[0011] S21, perform global average pooling on the depth map features and RGB features to obtain RGB feature vectors and depth map feature vectors; wherein, the RGB feature vector The input features generated by the convolutional blocks are expressed as , the depth map feature The convolutional blocks generate input features represented as ;

[0012] S22, the RGB feature vector and the depth map feature vector are respectively input into the fully connected layer and the Softmax activation function to obtain the two-channel attention vector. The formula is as follows:

[0013] ;

[0014] ;

[0015] in, and Indicates The parameters of the fully connected layer for each feature; represents the global average pooling operation; Channel attention vector representing RGB features; Channel attention vector representing the depth map features; Represents the Softmax activation function;

[0016] S23, multiply the two channel attention vectors by the corresponding input features channel by channel to generate channel enhanced features, as follows:

[0017] ;

[0018] ;

[0019] in, Indicates channel-wise multiplication; Channel enhancement features representing RGB features; Channel enhancement features representing depth map features;

[0020] S24, the channel attention vector and Aggregate through the maximum function and perform normalization to obtain the cross-referenced fused channel attention vector, expressed as:

[0021] ;

[0022] in, represents the fused channel attention vector; Represents a normalization operation; Represents the maximum function aggregation;

[0023] S25, based on fused channel attention vector ,use right and Perform feature enhancement to obtain enhanced features and ; The two enhanced features are further cascaded and fed into a 1×1 convolutional layer to generate cross-modal fusion features , the process is expressed as:

[0024] ;

[0025] ;

[0026] ;

[0027] in, Represents the enhanced features of RGB features; Representing enhanced features of depth map features; represents a 1×1 convolutional layer; Indicates cascade connection; Represents cross-modal fusion features.

[0028] Preferably, the cross-modal attention module is specifically as follows:

[0029] Source Depth Map and the distorted source image features is the input; the source depth map Obtained from a reference image in a dataset; Generated by a motion field generator; the motion field generator predicts dynamic motion information of a face in a video in a face video generation model;

[0030] Source depth map Input the depth encoder to get the depth feature map , and then through a 1×1 convolutional layer to Perform linear projection to latent feature map ; Through another two 1×1 convolutional layers Perform linear projection to obtain the latent feature maps and ;Will As the query parameter in the self-attention mechanism, As the key parameter in the self-attention mechanism, As the value parameter in the self-attention mechanism; the final feature obtained The formula is as follows:

[0031] ;

[0032] Among them, Softmax represents the Softmax normalization function; Represents a linear projection of query parameters; represents the linear projection of key parameters; Represents a linear projection of the value argument.

[0033] Preferably, the final objective function of the face video generation model is expressed as:

[0034] ;

[0035] in, represents the loss function of the generative adversarial network, represents the loss weight of the generative adversarial network loss; represents the perceptual reconstruction loss function, The loss weight representing the perceptual reconstruction loss; represents the key point loss function, Represents the loss weight of the key point loss function; Indicates the input drive frame, Represents the corresponding reconstructed frame.

[0036] Preferably, the loss function of the generative adversarial network adopts the least squares loss; the loss function of the generative adversarial network is expressed as follows:

[0037] ;

[0038] in, Indicates adjusting the parameters of the discriminator D so that the discriminator The loss function Reach a minimum value; Represents the adjustment generator Make the generator The loss function Reach a minimum value; A function representing a generator; represents the function of the discriminator; Represents noise, following a normalized or Gaussian distribution; Represents real data The probability distribution of Represents probability distribution Down Expected value; express The probability distribution of Represents probability distribution Down Expected value; Represents probability distribution Down Expected value; Represents the label of the generated sample; Represents the target label of the generated sample; Represents the label of the real sample.

[0039] Preferably, the perceptual reconstruction loss uses the pre-trained VGG-19 network as the network structure for evaluating the loss; the perceptual reconstruction loss is expressed as:

[0040] ;

[0041] in, represents the first Channel features, Indicates the number of feature channels in this layer; Indicates absolute value.

[0042] Preferably, the key point loss function is expressed as:

[0043] ;

[0044] in, Represents the intermediate representation of key points of reference images in the dataset; Represents the intermediate representation of the key points of the generated image; The Jacobian matrix intermediate representation of the reference image in the dataset; represents the Jacobian matrix intermediate representation of the generated image; T represents time; represents L1 loss.

[0045] On the other hand, a voice-driven face video generation device based on depth perception fusion includes the following:

[0046] A data acquisition and preprocessing module is used to acquire a face speech video dataset with audio clips and reference images, and divide the dataset into a training dataset and a test dataset in proportion after preprocessing the dataset;

[0047] A model building module is used to build a face video generation model; the face video generation model includes an audio encoder, an image encoder, a depth encoder, a cross-reference module and a cross-modal attention module; the audio encoder extracts the Melp features of the audio in the data set; the image encoder extracts the RGB features of the image in the data set; the depth encoder extracts the depth map features of the image in the data set; the cross-reference module fuses the depth map features and the RGB features; the cross-modal attention module introduces a self-attention mechanism to enhance the ability to retain facial structure;

[0048] A model training module is used to train a face video generation model using a training data set to obtain a trained face video generation model;

[0049] The model testing module is used to input the test data set into the trained face video generation model and output the generated face video combining audio and video.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The face video generation model of the present invention uses the two-dimensional face representation of the reference image and the three-dimensional face geometric structure relationship to model the facial motion representation, constructs a cross-reference module according to the facial motion attributes, mines the rich semantic information and texture information in the RGB image and the key depth information in the depth map, and combines with the cross-modal attention learning network to effectively improve the facial structure accuracy of the face video while taking into account the fine-grained details of the motion, making it more realistic and natural, thereby providing more flexibility for various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings;

[0053] Figure 1 It is a flow chart of a method for generating a voice-driven face video based on depth perception fusion according to an embodiment of the present invention;

[0054] Figure 2 This is an overall model diagram of the voice-driven face video generation method based on depth perception fusion according to an embodiment of the present invention;

[0055] Figure 3 It is a structural diagram of a cross-reference module of a voice-driven face video generation method based on depth perception fusion according to an embodiment of the present invention;

[0056] Figure 4 This is a structural block diagram of a voice-driven face video generation device based on depth perception fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention is further described below through specific implementation modes.

[0058] See also Figure 1 As shown, the voice-driven face video generation method based on depth perception fusion includes the following specific steps:

[0059] S1, obtain a face speaking video dataset with audio clips and reference images, and divide the dataset into a training dataset and a test dataset after preprocessing.

[0060] Specifically, we extracted various audio and video clips including speeches, lectures, interviews, etc. from existing public videos to construct the required data set. The extracted content was cut into 4 seconds in length, and the video frame rate was converted to 25FPS and framed, and the corresponding speech content sampling rate was converted to 16k. The librosa audio processing library was used to preprocess and extract features from the speech data. The obtained audio features contained a total of 41 acoustic features, including 13 Mel frequency cepstral coefficients, 26 Mel filter bank energy features, pitch, and silent parts.

[0061] In addition, the dataset also includes a validation dataset for performance evaluation of subsequent face video generation models.

[0062] S2, construct a face video generation model; the face video generation model includes an audio encoder, an image encoder, a depth encoder, a cross-reference module and a cross-modal attention module; the audio encoder extracts the Melp features of the audio in the dataset; the image encoder extracts the RGB features of the images in the dataset; the depth encoder extracts the depth map features of the images in the dataset; the cross-reference module fuses the depth map features and RGB features; the cross-modal attention module introduces a self-attention mechanism to enhance the ability to retain facial structure. See the overall model diagram of the face video generation model for details. Figure 2 shown.

[0063] See also Figure 3 As shown in Figure 2, the Cross Reference Module (CRM) aims to mine and combine the most discriminative feature channels in depth map features and RGB features and generate more informative features. Specifically, given the first The two input features generated by the convolutional blocks and First, global average pooling (GAP) is used to obtain the global information in the RGB image and depth view. Then, the two feature vectors are input into the fully connected layer (FC layer) and the Softmax activation function respectively. To obtain the channel attention vector and , which reflect the importance of RGB features and depth features respectively. Then the attention vector is multiplied by the input feature channel by channel, and the weights of each feature channel are dynamically adjusted. In this way, the cross-reference module CRM will automatically identify the feature channels that are closely related to the current task, while suppressing those redundant or irrelevant feature channels, thereby improving the quality and discrimination of feature representation. The mathematical formula of this process is as follows:

[0064] ;

[0065] ;

[0066] in, and Indicates The parameters of the FC layer of features, and Represents a global average pooling operation.

[0067] Then, channel enhancement features are generated , ;in, represents channel-wise multiplication. In addition, the attention vector and Aggregation is performed through the max function to retain useful feature channels from the RGB image and depth map, which are then fed into the normalization operation. Normalize the output to the range from 0 to 1. Thus, we obtain the cross-referenced fused channel attention vector . The process is expressed as:

[0068] ;

[0069] Based on fusion channel attention vector ,Will and and Enhanced features are added to obtain enhanced features. and The enhanced features of the RGB branch and the depth branch are further cascaded and fed into a 1×1 convolutional layer to generate cross-modal fusion features. , the process is expressed as:

[0070] ;

[0071] ;

[0072] ;

[0073] in, Represents the enhanced features of RGB features; Representing enhanced features of depth map features; represents a 1×1 convolutional layer; Indicates a cascade connection.

[0074] Cross-modal attention module. A cross-modal attention module is used to generate dense depth-aware attention maps to guide source image features. Perform face generation. Source image features The motion field generator in the face video generation model predicts the dynamic motion information of the face in the video based on audio features. The cross-modal attention learning network in the cross-modal attention module aims to enhance the model's ability to retain facial structure and generate tiny facial movements that are closely related to expressions. The introduction of depth information is based on its ability to provide the model with detailed three-dimensional geometric information, which is crucial for maintaining the integrity of the facial structure and identifying key facial expressions.

[0075] Specifically, the depth encoder Take the source depth map of the input reference image As input, the encoding gets the deep feature map , then through three different 1×1 convolutional layers And the source image features obtained after distortion and deformation Perform linear projection to three latent feature maps , and . , and denote the query, key, and value in the self-attention mechanism, respectively. Therefore, the geometrically related query features generated by the depth map Key features that can be related to appearance Fusion, generate a dense motion field for face generation. And get the final feature Used to generate facial images, the formula is as follows:

[0076] ;

[0077] in, Represents a linear projection of query parameters; represents the linear projection of key parameters; represents the linear projection of the value parameter; Softmax represents the Softmax normalization function, which outputs a dense depth-aware attention map, which contains important 3D geometric information for generating faces with finer-grained facial structure and motion details. Finally, the decoder refines the warped features As input, the final composite image is generated.

[0078] The final objective function of the face video generation model is:

[0079] ;

[0080] in, , and are hyperparameters during training, representing the loss weights of the generative adversarial network loss, perceptual reconstruction loss, and key point loss used to constrain the model to generate videos. Specifically, the least squares loss is introduced to replace the binary cross entropy loss in the traditional generative adversarial network. The loss function of the generative adversarial network in this method is expressed as follows:

[0081] ;

[0082] Among them, G represents the generator; D represents the discriminator, Indicates adjusting the parameters of the discriminator D so that the discriminator The loss function Reach a minimum value; Represents the adjustment generator Make the generator The loss function Reach a minimum value; A function representing a generator; A function representing the discriminator; used to evaluate how synchronized the lip movements in the generated video frames are with the input audio; Represents noise, following a normalized or Gaussian distribution; Represents real data The probability distribution of express The probability distribution of is the expected value, The same is the expected value; the perceptual reconstruction loss uses the pre-trained VGG-19 network as the network structure for evaluating the loss. For the input drive frame and the corresponding reconstructed frame , the perceptual reconstruction loss is expressed as:

[0083] ;

[0084] in, is extracted from a specific VGG-19 layer. Channel features, is the number of feature channels in this layer; Represents the absolute value. The key point loss function constrains the key points and Jacobian matrix of the motion field generated in this method, and its mathematical formula is expressed as follows:

[0085] ;

[0086] Where T represents time, represents L1 loss, and Respectively represent the key point intermediate representation of the input reference image and the generated image, and They represent the Jacobian matrix intermediate representation of the input reference image and the generated image, respectively, and they are both obtained by the key point detection network.

[0087] S3, using the training data set to train the face video generation model to obtain a trained face video generation model.

[0088] S4, input the test data set into the trained face video generation model, and output the generated face video combining audio and video.

[0089] like Figure 4 As shown, the present invention also discloses a method and device for generating a voice-driven face video based on depth perception fusion, comprising:

[0090] The data acquisition and preprocessing module 401 is used to acquire a face speech video dataset with audio clips and reference images, and divide the dataset into a training dataset and a test dataset in proportion after preprocessing the dataset.

[0091] The model building module 402 is used to build a face video generation model; the face video generation model includes an audio encoder, an image encoder, a depth encoder, a cross-reference module and a cross-modal attention module; the audio encoder extracts the Melp features of the audio in the data set; the image encoder extracts the RGB features of the image in the data set; the depth encoder extracts the depth map features of the image in the data set; the cross-reference module fuses the depth map features and the RGB features; the cross-modal attention module introduces a self-attention mechanism to enhance the ability to retain facial structure.

[0092] The model training module 403 is used to train the face video generation model using the training data set to obtain a trained face video generation model.

[0093] The model testing module 404 is used to input the test data set into the trained face video generation model and output the generated face video combining audio and video.

[0094] The specific implementation of the voice-driven facial video generation device based on depth perception fusion is the same as the voice-driven facial video generation method based on depth perception fusion, and will not be repeated in this embodiment.

[0095] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A method for generating voice-driven face videos based on depth perception fusion, characterized in that: The steps include: S1, obtain a face speaking video dataset with audio clips and reference images, and divide the dataset into a training dataset and a test dataset after preprocessing; S2, constructing a face video generation model; the face video generation model includes an audio encoder, an image encoder, a depth encoder, a cross-reference module and a cross-modal attention module; the audio encoder extracts the Melp features of the audio in the data set; the image encoder extracts the RGB features of the image in the data set; the depth encoder extracts the depth map features of the image in the data set; the cross-reference module fuses the depth map features and the RGB features; the cross-modal attention module introduces a self-attention mechanism to enhance the ability to retain facial structure; S3, using the training data set to train the face video generation model to obtain a trained face video generation model; S4, input the test data set into the trained face video generation model, and output the face video combined with audio and video.

2. The method for generating voice-driven face videos based on depth perception fusion according to claim 1, characterized in that: The cross-reference module is as follows: S21, perform global average pooling on the depth map features and RGB features to obtain RGB feature vectors and depth map feature vectors; wherein, the RGB feature vector The input features generated by the convolutional blocks are expressed as , the depth map feature The convolutional blocks generate input features represented as ; S22, the RGB feature vector and the depth map feature vector are respectively input into the fully connected layer and the Softmax activation function to obtain the two-channel attention vector. The formula is as follows: ； ； in, and Indicates The parameters of the fully connected layer for each feature; represents the global average pooling operation; Channel attention vector representing RGB features; Channel attention vector representing the depth map features; Represents the Softmax activation function; S23, multiply the two channel attention vectors by the corresponding input features channel by channel to generate channel enhanced features, as follows: ； ； in, Indicates channel-wise multiplication; Channel enhancement features representing RGB features; Channel enhancement features representing depth map features; S24, the channel attention vector and Aggregate through the maximum function and perform normalization to obtain the cross-referenced fused channel attention vector, expressed as: ； in, represents the fused channel attention vector; Represents a normalization operation; Represents the maximum function aggregation; S25, based on fused channel attention vector ,use right and Perform feature enhancement to obtain enhanced features and ; The two enhanced features are further cascaded and fed into a 1×1 convolutional layer to generate cross-modal fusion features , the process is expressed as: ； ； ； in, Represents the enhanced features of RGB features; Representing enhanced features of depth map features; represents a 1×1 convolutional layer; Indicates cascade connection; Represents cross-modal fusion features.

3. The method for generating voice-driven face videos based on depth perception fusion according to claim 1, characterized in that: The cross-modal attention module is specifically as follows: Source Depth Map and the distorted source image features is the input; the source depth map Obtained from a reference image in a dataset; Generated by a motion field generator; the motion field generator predicts dynamic motion information of a face in a video in a face video generation model; Source Depth Map Input the depth encoder to get the depth feature map , and then through a 1×1 convolutional layer to Perform linear projection to latent feature map ; Through another two 1×1 convolutional layers Perform linear projection to obtain the latent feature maps and ;Will As the query parameter in the self-attention mechanism, As the key parameter in the self-attention mechanism, As the value parameter in the self-attention mechanism; the final feature obtained The formula is as follows: ； Among them, Softmax represents the Softmax normalization function; Represents a linear projection of query parameters; represents the linear projection of key parameters; Represents a linear projection of the value argument.

4. The method for generating voice-driven face videos based on depth perception fusion according to claim 1, characterized in that: The final objective function of the face video generation model is expressed as: ； in, represents the loss function of the generative adversarial network, represents the loss weight of the generative adversarial network loss; represents the perceptual reconstruction loss function, The loss weight representing the perceptual reconstruction loss; represents the key point loss function, Represents the loss weight of the key point loss function; Indicates the input drive frame, Represents the corresponding reconstructed frame.

5. The method for generating voice-driven face videos based on depth perception fusion according to claim 4, characterized in that: The loss function of the generative adversarial network adopts the least squares loss; the loss function of the generative adversarial network is expressed as follows: ； in, Indicates adjusting the parameters of the discriminator D so that the discriminator The loss function Reach a minimum value; Represents the adjustment generator Make the generator The loss function Reach a minimum value; A function representing a generator; represents the function of the discriminator; Represents noise, following a normalized or Gaussian distribution; Represents real data The probability distribution of Represents probability distribution Down Expected value; express The probability distribution of Represents probability distribution Down Expected value; Represents probability distribution Down Expected value; Represents the label of the generated sample; Represents the target label of the generated sample; Represents the label of the real sample.

6. The method for generating voice-driven face videos based on depth perception fusion according to claim 4, characterized in that: The perceptual reconstruction loss uses the pre-trained VGG-19 network as the network structure for evaluating the loss; the perceptual reconstruction loss is expressed as: ； in, represents the first Channel features, Indicates the number of feature channels in this layer; Indicates absolute value.

7. The method for generating voice-driven face videos based on depth perception fusion according to claim 4, characterized in that: The key point loss function is expressed as: ； in, Represents the intermediate representation of key points of reference images in the dataset; Represents the intermediate representation of the key points of the generated image; The Jacobian matrix intermediate representation of the reference image in the dataset; represents the Jacobian matrix intermediate representation of the generated image; T represents time; represents L1 loss.

8. A voice-driven face video generation device based on depth perception fusion, comprising the following: A data acquisition and preprocessing module is used to acquire a face speech video dataset with audio clips and reference images, and divide the dataset into a training dataset and a test dataset in proportion after preprocessing the dataset; A model building module is used to build a face video generation model; the face video generation model includes an audio encoder, an image encoder, a depth encoder, a cross-reference module and a cross-modal attention module; the audio encoder extracts the Melp features of the audio in the data set; the image encoder extracts the RGB features of the image in the data set; the depth encoder extracts the depth map features of the image in the data set; the cross-reference module fuses the depth map features and the RGB features; the cross-modal attention module introduces a self-attention mechanism to enhance the ability to retain facial structure; A model training module is used to train a face video generation model using a training data set to obtain a trained face video generation model; The model testing module is used to input the test data set into the trained face video generation model and output the generated face video combining audio and video.

Images