Method for training lip image generation model, method for generating lip image

Abstract

The application relates to a training method of a mouth shape image generation model and a mouth shape image generation method. The method comprises the following steps: acquiring first sample data used for training a generator in a mouth shape image generation model; the first sample data comprises a plurality of sample pairs; inputting a sample face image and a sample audio into the generator to obtain a synthesized frame image containing a complete face; taking a first mouth region image cut from a real frame image and a second mouth region image cut from the synthesized frame image as second sample data used for training a discriminator in the mouth shape image generation model; and performing adversarial training on the generator and the discriminator by using the first sample data and the second sample data until a training end condition is met, so that a trained mouth shape image generation model is obtained. By using the method, the trained mouth shape image generation model can generate a high-definition tooth image, and the overall quality of mouth shape image generation is effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method for a lip shape image generation model, a lip shape image generation method, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of computer technology, deep learning models can be used to generate lip-sync videos, such as those generated from speech-driven mouth and face images.

[0003] In related technologies, traditional lip-syncing generation techniques typically involve mapping speech to lip shapes. While they have made some progress in generating lip-syncing videos for specific speakers, learning and reproducing lip movements synchronized with speech, such as generating and synchronizing lip movements by learning the correspondence between input audio and lip shape markers, traditional methods fall short in terms of clarity and detail in the tooth area. This results in generated lip-syncing videos that fail to clearly depict tooth details, appearing blurry and lacking realism, thus affecting the overall quality of the generated lip-syncing video. Summary of the Invention

[0004] Therefore, it is necessary to provide a training method for a mouth shape image generation model, a mouth shape image generation method, a computer device, a computer-readable storage medium, and a computer program product that can improve the display effect of the tooth area, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for training a lip-shape image generation model. The method includes:

[0006] First sample data is obtained for training the generator in the lip-shape image generation model; the first sample data includes multiple sample pairs, each sample pair including a sample face image and sample audio of the corresponding time interval of the sample face image; the sample face image includes a real frame image with the mouth area covered and a fake sample image with a complete face, the real frame image is a frame image sampled from a certain frame in the sample video, and the fake sample image is a frame image sampled from frames in the sample video other than the certain frame;

[0007] The sample face image and the sample audio are input into the generator to obtain a synthetic frame image containing the complete face;

[0008] The first mouth region image cropped from the real frame image and the second mouth region image cropped from the synthesized frame image are used as the second sample data for training the discriminator in the mouth shape image generation model.

[0009] Using the first sample data and the second sample data, the generator and the discriminator are subjected to adversarial training until the training termination condition is met, resulting in a trained lip-shape image generation model; the lip-shape image generation model is used to generate a face image with lip shapes based on the input speech conversion.

[0010] In one embodiment, the method further includes:

[0011] The images in the sample video are temporally sampled to obtain multiple real frame images, and the audio in the sample video is temporally sampled to obtain sample audio corresponding to the real frame images; the time interval of the sample audio matches the time interval of the real frame images.

[0012] For any real frame image, a frame image different from any real frame image is obtained by randomly sampling the sample video and used as the fake sample image.

[0013] In one embodiment, inputting the sample face image and the sample audio into the generator to obtain a synthetic frame image containing the complete face includes:

[0014] The sample audio is processed by the audio encoder in the generator to obtain an audio feature vector; wherein the audio encoder has a one-dimensional convolutional layer and a neural network based on a self-attention mechanism.

[0015] The encoder in the generator fuses the audio feature vector and the image feature vector of the sample face image to obtain the encoded output result.

[0016] The encoded output is decoded by the face decoder in the generator to obtain the synthesized frame image.

[0017] In one embodiment, the face encoder includes multiple convolutional layers connected sequentially; the process of fusing the audio feature vector and the image feature vector of the sample face image through the face encoder in the generator to obtain the encoded output includes:

[0018] The sample face image is convolved by the intermediate convolutional layer among the multiple convolutional layers of the face encoder to obtain an intermediate feature vector;

[0019] The audio feature vector and the intermediate feature vector are fused based on the cross-attention mechanism to obtain the fused feature vector output by the intermediate convolutional layer;

[0020] The encoding output result is obtained based on the image feature vectors output by each convolutional layer in the face encoder except for the intermediate convolutional layer, and the fused feature vector output by the intermediate convolutional layer.

[0021] In one embodiment, the steps of training the discriminator include:

[0022] The first mouth region image and the second mouth region image are respectively used as input images and input to the discriminator;

[0023] The mouth features of the input image are obtained by extracting features from each convolutional layer in the discriminator.

[0024] Based on the mouth features of the input image, the discrimination probability information output by the discriminator is obtained; the discrimination probability information is used to characterize the probability that the first mouth region image belongs to the real frame image and the probability that the second mouth region image belongs to the real frame image.

[0025] In one embodiment, the step of performing adversarial training on the generator and the discriminator using the first sample data and the second sample data includes:

[0026] During the training of the generator using the first sample data, the parameters of the generator are adjusted based on the adversarial loss value and the content loss value; the training objective of the generator is to enhance the realism of the synthesized frame image output by the generator; the adversarial loss value is calculated based on the random noise vector of the synthesized frame image, and the content loss value is calculated based on the spatial metric distance of the real frame image;

[0027] During the training of the discriminator using the second sample data, the parameters of the discriminator are adjusted based on the loss value corresponding to the real frame image, the loss value corresponding to the synthetic frame image, and the feature matching loss; the training objective of the discriminator is to reduce the error degree of the discrimination probability information output by the discriminator; the feature matching loss is determined based on the intermediate layer feature representation of the discriminator.

[0028] Secondly, this application also provides a method for generating a lip shape image. The method includes:

[0029] Based on the current processing frame in the original video, obtain the audio to be processed and the face image to be processed; the audio to be processed is obtained based on the video speech corresponding to the current processing frame in the original video; the face image to be processed includes a first face image and a second face image, the first face image is the face image in the current processing frame, and the second face image is the face image obtained after occluding the mouth area of ​​the face image in the current processing frame;

[0030] The audio to be processed and the face image to be processed are input into a trained lip-sync image generation model to obtain a target face image with lip movements that match the audio to be processed; the target face image is used to replace the face image in the current processing frame of the original video to generate the target video;

[0031] The trained lip shape image generation model is obtained by training the lip shape image generation model according to any one of the above methods.

[0032] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the training method for the lip-shape image generation model as described in the first aspect, and / or the steps of the lip-shape image generation method as described in the second aspect.

[0033] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the training method for the lip-shape image generation model as described in the first aspect, and / or the steps of the lip-shape image generation method as described in the second aspect.

[0034] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the training method for the lip-shape image generation model as described in the first aspect, and / or the steps of the lip-shape image generation method as described in the second aspect.

[0035] The aforementioned training method, lip-shape image generation method, computer device, computer-readable storage medium, and computer program product for a lip-shape image generation model acquire first sample data for training the generator in the lip-shape image generation model. This first sample data includes multiple sample pairs, each sample pair including a sample face image and sample audio from the corresponding time interval of the sample face image. The sample face image includes a real frame image with the mouth area obscured and a fake sample image with a complete face. The real frame image is a frame image sampled from a certain frame in the sample video, and the fake sample image is a frame image sampled from all frames in the sample video except for that particular frame. Then, the sample face image and sample audio are input into the generator to obtain a synthetic frame image containing a complete face. The first mouth region image cropped from the real frame image and the second mouth region image cropped from the synthesized frame image are used as the second sample data for training the discriminator in the lip shape image generation model. Then, the generator and discriminator are subjected to adversarial training using the first and second sample data until the training termination condition is met, resulting in a trained lip shape image generation model. This lip shape image generation model is used to generate face images with lip shapes based on input speech conversion, thus optimizing the lip shape image generation model. Based on the introduced discriminator, which pays special attention to the generation effect of the mouth region in the image, and combined with the adversarial training of the generator and discriminator, the trained lip shape image generation model can produce high-definition teeth images, effectively improving the overall quality of lip shape image generation. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a training method for a lip-shape image generation model in one embodiment.

[0037] Figure 2a This is a schematic diagram of a generator processing flow in one embodiment;

[0038] Figure 2b This is a schematic diagram of facial landmark detection in one embodiment;

[0039] Figure 2c This is a schematic diagram of a mouth area in one embodiment;

[0040] Figure 3a This is a schematic diagram of a face region in one embodiment;

[0041] Figure 3b This is a schematic diagram of an audio feature sampling method in one embodiment;

[0042] Figure 4 This is a schematic diagram of a cross-attention mechanism in one embodiment;

[0043] Figure 5This is a flowchart illustrating a lip shape image generation method in one embodiment;

[0044] Figure 6 This is a flowchart illustrating the training method for the lip-shape image generation model in another embodiment;

[0045] Figure 7 This is a structural block diagram of a training device for a lip-shape image generation model in one embodiment;

[0046] Figure 8 This is a structural block diagram of a lip-shape image generation device in one embodiment;

[0047] Figure 9 This is an internal structural diagram of a computer device in one embodiment;

[0048] Figure 10 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In one exemplary embodiment, such as Figure 1 As shown, a training method for a lip-shape image generation model is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and can be implemented through the interaction between the terminal and the server. The server can be a standalone server or a server cluster consisting of multiple servers; the terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0051] In this embodiment, the method includes the following steps S101 to S104. Wherein:

[0052] In step S101, the first sample data for training the generator in the lip-shape image generation model is obtained; the first sample data includes multiple sample pairs, each sample pair including a sample face image and sample audio of the corresponding time interval of the sample face image.

[0053] As an example, a lip-shape image generation model can be a deep learning model based on generative adversarial networks, which can include a generator and a discriminator.

[0054] The sample face images may include real frame images with the mouth area covered and fake sample images with a complete face. The real frame image is a frame image obtained by sampling a certain frame in the sample video, and the fake sample image is a frame image obtained by sampling frames other than a certain frame in the sample video.

[0055] As an example, the time interval can be the time range of the real frame image in the sample video, or it can be the time range of the frames of the images within 5 frames (which can be other values, and are not specifically limited in this embodiment) of the real frame image.

[0056] In practical applications, during the training phase of a lip-shape image generation model, multiple sample pairs can be constructed as the first sample data, which can then be used to train the generator in the lip-shape image generation model.

[0057] For example, during the training of the generator, such as Figure 2a The generator processing flow shown involves acquiring a sample video, performing time-series sampling based on the sample video to obtain a real frame image, and performing time-series sampling on the audio in the sample video to obtain sample audio within the time interval corresponding to the real frame image. It also allows for occlusion processing of the lower half of the face in the real frame image (e.g., ...). Figure 2a The image obtained by temporal sampling of the lower half of the face can be used to randomly sample the video and obtain frame images that are different from the sampled real frame images, which can be used as fake sample images (e.g., ...). Figure 2a Randomly sample complete human faces from the image (e.g., images of the face and audio) to construct image-audio based sample pairs (e.g., ...). Figure 2a The dashed box contains two images and an audio clip.

[0058] In step S102, the sample face image and sample audio are input into the generator to obtain a synthetic frame image containing the complete face.

[0059] In practical implementation, the deep learning-based generator aims to generate an image that corresponds to the lip movements based on a reference image and audio (i.e., a sample face image and sample audio) that do not match the lip movements. Figure 2a As shown, a synthetic frame image containing the complete face is obtained.

[0060] In step S103, the first mouth region image cropped from the real frame image and the second mouth region image cropped from the synthetic frame image are used as the second sample data for training the discriminator in the mouth shape image generation model.

[0061] After obtaining the synthesized frame image, a first mouth region image can be cropped from the real frame image, and a second mouth region image can be cropped from the synthesized frame image, by identifying the mouth region in the image. Optionally, a facial landmark detection algorithm can be used to identify the mouth region in the image, such as... Figure 2b As shown, this facial landmark detection algorithm can obtain 68 facial landmarks, where the two-dimensional coordinates of each facial landmark can be [x1, y1], ..., [x...]. 68 y 68 ], where x is the horizontal axis coordinate and y is the vertical axis coordinate.

[0062] In one example, the coordinates of each facial landmark can determine the position and size of the mouth region. Therefore, based on the identified positional information of the facial landmarks, the mouth region can be accurately identified. Figure 2c As shown. For example, the left and right boundaries of the mouth region can be determined based on the minimum and maximum X coordinates of the mouth key points, and the upper and lower boundaries of the mouth region can be determined based on the minimum and maximum Y coordinates of the mouth key points. Thus, cropping the corresponding mouth region image based on the accurately identified mouth region can help generate an image that corresponds to the mouth shape.

[0063] In step S104, the generator and discriminator are trained adversarially using the first sample data and the second sample data until the training termination condition is met, thus obtaining the trained lip shape image generation model.

[0064] Among them, the lip-shape image generation model can be used to generate facial images with lip shapes based on the input speech.

[0065] Specifically, by sampling image data from random noise to generate fake images (i.e., fake sample images) and creating multiple sample pairs, a first sample data is obtained, which can be used to train the generator. A second sample data is obtained by sampling real images from real data (i.e., real frame images) and sampling fake images based on the generator's output data (i.e., synthetic frame images), which can be used to train the discriminator. Simultaneously, adversarial training can be performed on the generator and discriminator, such as alternating the training processes until a certain number of iterations or performance metrics (i.e., meeting the training termination condition) are reached. This training strategy reflects the competitive relationship between the generator and discriminator in the adversarial network, allowing them to improve their respective performance through continuous competition.

[0066] Compared to traditional methods, the technical solution in this embodiment, by introducing a discriminator focused on mouth features, can optimize the clarity of tooth details in the generative network, so as to more accurately measure the tooth generation effect of the model in complex videos and effectively improve the image quality of mouth images, especially the tooth region.

[0067] In the training method of the above-mentioned lip shape image generation model, the first sample data for training the generator in the lip shape image generation model is obtained. Then, the sample face image and sample audio are input into the generator to obtain a synthetic frame image containing a complete face. The first mouth region image cropped from the real frame image and the second mouth region image cropped from the synthetic frame image are used as the second sample data for training the discriminator in the lip shape image generation model. Then, the generator and discriminator are trained adversarially using the first and second sample data until the training termination condition is met, and the trained lip shape image generation model is obtained. This optimizes the lip shape image generation model. Based on the introduced discriminator, which pays special attention to the generation effect of the mouth region in the image, and combined with the adversarial training of the generator and discriminator, the trained lip shape image generation model can produce high-definition tooth images, effectively improving the overall quality of lip shape image generation.

[0068] In one exemplary embodiment, the following steps may also be included:

[0069] Multiple real frame images are obtained by temporal sampling of images in the sample video, and sample audio corresponding to the real frame images is obtained by temporal sampling of audio in the sample video; the time interval of the sample audio matches the time interval of the real frame images; for any real frame image, a frame image different from any real frame image is obtained by randomly sampling the sample video as a fake sample image.

[0070] In practical applications, for the data preprocessing stage, by resampling the sample video to 25 FPS and sampling all audio from the sample video to 16 kHz, the sample video can be processed frame by frame. Face recognition algorithms can be used to obtain the face regions in each frame, such as... Figure 3a As shown, the resulting face image can be resized to 256x256 and used as a real frame image.

[0071] In an optional embodiment, other preprocessing operations can be performed on the obtained face image, such as normalization to scale the pixel values ​​of the image to the range of [-1, 1], and whitening, to facilitate subsequent model training.

[0072] In one example, for audio processing, the Mel (Mell-to-Screen) features of the sample audio can be extracted. The Mel feature extraction parameters can be a frame length of 25ms (i.e., 400 sampling points). The sample audio is then converted into an 80-dimensional vector using a short-time Fourier transform. For example, with a step size of 10ms, one second of audio can be converted into an 80*100 feature. For instance, as shown... Figure 3bAs shown, a 1-second video contains 25 images, and its corresponding MEL feature is 80*100.

[0073] In another example, when constructing image-audio feature pairs (i.e., sample pairs), each image can correspond to a matrix with dimensions [3, 256, 256] (e.g., an image with 256x256 pixels, each pixel with three channels; other values ​​are also possible, but not specifically limited in this embodiment). The audio feature dimension for the corresponding time interval of the image is [80, 4] (other values ​​are also possible, but not specifically limited in this embodiment). To accommodate dataset errors, such as... Figure 3b As shown, the audio corresponding to the 5 frames near each frame can be mapped to that frame. The audio feature dimension of the time interval corresponding to each frame (i.e., the real frame image) is [80, 20] (other values ​​are also possible, but not specifically limited in this embodiment). The reference image in the image-audio feature pair consists of two parts: one part is a real frame image with the lower half of the face obscured, and the other part is a randomly sampled complete face image (i.e., a fake sample image). The dimension of the image input to the generator corresponds to [6, 256, 256] (other values ​​are also possible, but not specifically limited in this embodiment).

[0074] In this embodiment, multiple real frame images are obtained by temporally sampling the images in the sample video, and sample audio corresponding to the real frame images is obtained by temporally sampling the audio in the sample video. Then, for any real frame image, a frame image different from any real frame image is obtained as a fake sample image by randomly sampling the sample video, which provides data support for training the generator.

[0075] In an exemplary embodiment, step S102, inputting the sample face image and sample audio into the generator to obtain a synthetic frame image containing the complete face, may include the following steps:

[0076] The audio encoder in the generator processes the sample audio to obtain an audio feature vector; the face encoder in the generator fuses the audio feature vector and the image feature vector of the sample face image to obtain the encoded output; the face decoder in the generator decodes the encoded output to obtain the synthesized frame image.

[0077] The audio encoder has a one-dimensional convolutional layer and a neural network based on a self-attention mechanism.

[0078] In the specific implementation, the mel feature dimension corresponding to the sample audio input to the Audio Encoder is [80, 20] (other values ​​are also possible, but not specifically limited in this embodiment). The combination of neural network layers in the audio encoder can be 1D Conv + BN + ReLU, which can include a one-dimensional convolutional layer, batch normalization (BN), and ReLU activation function. The number of filters in the one-dimensional convolutional layer can be set to 512, the filter size can be set to 10, and the stride is 2. Therefore, the feature size output by the one-dimensional convolutional layer is [512, 11] (other values ​​are also possible, but not specifically limited in this embodiment).

[0079] In an alternative embodiment, a Transformer encoder (i.e., a neural network based on a self-attention mechanism) can be used, which may contain multiple self-attention layers and feedforward neural network layers. The input to the Transformer encoder is a sequence, such as the features output by a one-dimensional convolutional layer being considered as a sequence of length 11, with each element having a dimension of 512. By inputting this sequence into the Transformer encoder and processing it using a self-attention layer and a feedforward neural network layer, the output of the Transformer encoder can be obtained, where the output dimension of the self-attention layer is [512, 11].

[0080] In one example, global average pooling can be performed on the output of the Transformer encoder. Global average pooling can convert a sequence into a vector, and each element of the sequence can be averaged, resulting in an output feature size of

[512] . Based on the obtained 512-dimensional vector (i.e., the audio feature vector), this vector can be used as the output of the audio encoder for subsequent computation or tasks. Since the audio encoder only uses a one-dimensional convolutional layer and a Transformer encoder, its computational complexity is relatively small, which can effectively improve processing efficiency.

[0081] In this embodiment, the audio sample is processed by the audio encoder in the generator to obtain an audio feature vector. Then, the audio feature vector and the image feature vector of the sample face image are fused by the face encoder in the generator to obtain an encoded output result. Finally, the encoded output result is decoded by the face decoder in the generator to obtain a synthetic frame image. The audio feature vector can be obtained based on the audio encoder processing, which provides data support for further fusion of audio feature vector and image feature vector.

[0082] In one exemplary embodiment, the face encoder may include multiple convolutional layers connected sequentially. The encoded output is obtained by fusing audio feature vectors and image feature vectors from sample face images through the face encoder in the generator. This process may include the following steps:

[0083] The sample face image is convolved by the intermediate convolutional layer in the face encoder to obtain the intermediate feature vector. The audio feature vector and the intermediate feature vector are fused based on the cross-attention mechanism to obtain the fused feature vector output by the intermediate convolutional layer. The encoding output result is obtained based on the image feature vectors output by each convolutional layer in the face encoder except the intermediate convolutional layer and the fused feature vector output by the intermediate convolutional layer.

[0084] In practical applications, a Face Encoder can be used to process sample face images. The purpose of this face encoder is to extract feature representations from the image, which can be a CNN (Convolutional Neural Network) with multiple convolutional and pooling layers. The feature dimension of the sample face image input to the face encoder is [6, 256, 256] (other values ​​are also possible, but not specifically limited in this embodiment). It is obtained by stitching together two images, such as a real frame image with the mouth area obscured and a fake sample image with a complete face.

[0085] In one example, the face encoder can contain a series of convolutional layers, each followed by batch normalization (BN) and ReLU activation functions. For example, the output size of the first layer can be [64, 128, 128] (other values ​​are also possible, but not specifically limited in this embodiment), the output size of the second layer can be [128, 64, 64] (other values ​​are also possible, but not specifically limited in this embodiment), and so on, with each layer's output size gradually decreasing and the number of channels increasing. Each convolutional layer can be followed by a max-pooling layer, which can be used to gradually reduce the spatial size of the feature map; after processing by multiple convolutional and pooling layers, a smaller feature map, such as [1024, 8, 8] (other values ​​are also possible, but not specifically limited in this embodiment), can be obtained as the output of the face encoder (i.e., the encoded output result).

[0086] Specifically, in the intermediate layer of the facial encoder, a feature map can be selected for cross-attention fusion of audio features based on the Cross Attention module. For example, the audio feature vector can be a 512-dimensional vector obtained from the audio encoder; the image feature vector can be an intermediate feature map (i.e., an intermediate feature vector) from the selected facial encoder, such as [1024, 8, 8] (or other values, which are not specifically limited in this embodiment). During the cross-attention process, the Query can be set to each position of the image feature map, such as [8*8, 1024], the Key can be set to map the audio feature vector to a space with the same dimension as the image features through a linear layer (fully connected layer), such as [L (fully connected layer), 1024], and the Value can be set to be the same as the Key. Through attention calculation, the similarity between each Query and the Key can be calculated, such as by using a dot product, and the similarity score can be normalized using the softmax function to determine the attention weight. Then, the Value can be weighted by this attention weight and added or multiplied with the original image features to achieve feature fusion.

[0087] In an alternative embodiment, the cross-attention mechanism allows the model to reference another related sequence while processing an input sequence. This mechanism can be used for natural language processing and computer vision tasks, such as machine translation, text summarization, and image caption generation. Cross-attention includes a Query (Q), a Key (K), and a Value (V), such as... Figure 4 As shown, the query comes from the input being processed (such as...). Figure 4 (In the latent array), the key and value come from the sequence that needs to be referenced (e.g., Figure 4 The goal of the attention mechanism is to calculate the similarity between the query and each key, and then use the similarity to perform a weighted sum of the values ​​to generate a new context vector that combines information from the input array and the latent array.

[0088] like Figure 4 As shown, the specific calculation process of the cross-attention mechanism is as follows:

[0089] First, through linear transformations (such as a fully connected layer), the input array and latent array can be converted into a query, key, and value. For example, Q = latent array * W. q K = input array * W k V = input array * Wv W q W k and W v It is the weight matrix that needs to be learned.

[0090] exist Figure 4 In the example, the input array has dimensions [M, C], where M is the length of the input sequence and C is the dimension of the input features. The latent array has dimensions [N, D], where N is the length of the latent sequence and D is the dimension of the latent features. Linear transformations can convert the input array and latent array into Query, Key, and Value. Assume the weight matrix W... q The dimension is [D, D'], and the weight matrix W k and W v If the dimensions are [C, D'] and [C, C'], then:

[0091] Q = latent array * W q Therefore, the dimension of Q is [N, D'].

[0092] K = input array *W k Therefore, the dimension of K is [M, D'].

[0093] V = input array * W v Therefore, the dimension of V is [N, C'].

[0094] The resulting D' and C' are new feature dimensions, which can be set according to the specific task and model structure. In practical applications, D' and C' can be set to a small value to reduce computational complexity and prevent overfitting.

[0095] Then, the dot product between Q and each K can be calculated to obtain a similarity matrix, which can be [M, N]. By scaling the similarity matrix, such as dividing it by the square root of D', and applying the softmax function, the sum of each row is equal to 1, thus obtaining an attention weight matrix.

[0096] Finally, the attention weight matrix [M, N] can be multiplied by the [N, C']-dimensional V to obtain the [M, C']-dimensional output vector, which integrates information from the input array and the latent array.

[0097] In another example, regarding the audio-video feature fusion process, audio feature vectors are extracted (e.g., using an audio encoder to extract a 512-dimensional feature vector), and image feature vectors are extracted and intermediate layer features are selected (e.g., using a facial encoder to extract multi-scale feature representations of the image). An intermediate layer feature map from the facial encoder is then selected for cross-attention fusion. During the fusion process, the audio feature vector can be mapped to the same dimension as the image feature vector through a linear layer. Then, the cross-attention between the image feature vector and the mapped audio feature vector can be calculated. The obtained attention weights are then used to weight the audio feature vector before fusing it with the image feature vector (i.e., fusing the audio feature vector and intermediate feature vector based on the cross-attention mechanism to obtain the fused feature vector output by the intermediate convolutional layer). Since the input to the self-attention mechanism is in sequence form, when processing image data, the two-dimensional feature map needs to be converted to sequence form. Therefore, when using cross-attention, the Query and Key need to be reshaped into a two-dimensional form to facilitate the calculation of attention weights.

[0098] For example, the reshaping process and calculation steps of Query and Key in cross-attention are as follows:

[0099] 1. Reshaping the Query: For an image feature vector, if the feature map size obtained from the encoder is [C, H, W] (C is the number of channels, H and W are the height and width respectively), it needs to be reshaped to [H * W, C] to form a sequence; the dimension of the output Q is [H * W, C].

[0100] 2. Key and Value Reshaping: For audio feature vectors, the original dimension is

[512] . In cross-attention, it needs to be expanded to the same number of channels C as the image feature vector. For example, it can be mapped to [L, C] through a linear layer, so that the Key and Value can be matched with the reshaped Query. The dimension of output K is [L, C], and the dimension of output V is [L, C].

[0101] 3. Calculate attention weights: Attention scores can be calculated using the reshaped Query and Key, such as through dot product. Then, the softmax function can be applied to obtain normalized attention weights. The output softmax(Q*K^T) has dimensions [H * W, L].

[0102] 4. Apply attention weights. The obtained attention weights can be used to weight the Value, and then the weighted Value can be reshaped back to the original image feature vector dimension as needed for fusion with image features; the dimension of the output weighted Value is [H*W, C].

[0103] 5. Feature Fusion: The weighted value can be fused with the original image features, such as through addition, multiplication, or other fusion techniques; the output dimension is [H*W, C];

[0104] 6. Feature Reshaping Back to Image Size: The fused features can be reshaped from the sequence form [H*W, C] back into a two-dimensional feature map [C, H, W] for further processing or as input to the decoder.

[0105] Through the steps described above, cross-attention allows the model to effectively exchange information between different modalities, and audio features can guide the updating of image features, and vice versa, thereby enhancing the model's ability to understand and process multimodal data. The generative network is optimized using a supervised learning framework to produce high-resolution tooth images, enhancing visual realism.

[0106] In practical applications, a Face Decoder can be used to progressively upsample the output of the face encoder (i.e., the encoded output) to restore the spatial dimensions of the original image. For example, the input to the face decoder can be the feature map output by the face encoder, such as [1024, 8, 8] (or other values, which are not specifically limited in this embodiment). During upsampling, transposed convolution or upsampling techniques can be used to progressively increase the spatial dimensions of the feature map. Then, a convolutional layer can be appended after each upsampling step. This convolutional layer can be used to refine the features. Subsequently, the outputs of corresponding layers in the encoder and decoder can be concatenated, and spatial information can be restored through skip connections. The output of the face decoder can be an image upsampled to the original image size [3, 256, 256] (or other values, which are not specifically limited in this embodiment) (i.e., a synthesized frame image). Thus, through the above processing method, the model can effectively combine audio and image information in the intermediate layer, enhance feature representation, and effectively improve the performance of multimodal tasks.

[0107] In this embodiment, the sample face image is convolved by the intermediate convolutional layer among the multiple convolutional layers of the face encoder to obtain the intermediate feature vector. Then, the audio feature vector and the intermediate feature vector are fused based on the cross-attention mechanism to obtain the fused feature vector output by the intermediate convolutional layer. Then, the encoding output result is obtained based on the image feature vectors output by each convolutional layer other than the intermediate convolutional layer in the face encoder, and the fused feature vector output by the intermediate convolutional layer. This can effectively combine audio and image information, enhance feature representation, and improve the performance of the lip shape image generation model.

[0108] In one exemplary embodiment, the following steps may also be included:

[0109] The first and second mouth region images are input images to the discriminator. The discriminator extracts features from the input images through each convolutional layer to obtain the mouth features of the input images. Based on the mouth features of the input images, the discriminator outputs the discrimination probability information. The discrimination probability information is used to characterize the probability that the first mouth region image and the second mouth region image belong to the real frame image.

[0110] In one example, the mouth region is determined by using a face recognition algorithm in the data preprocessing stage. The mouth region can be cropped from the real frame image and the synthetic frame image respectively, and can be adjusted to a fixed size, such as [64, 128] (other values ​​are also possible, but no specific limitation is made in this embodiment); the input of the discriminator is the resized mouth region image (i.e., the first mouth region image and the second mouth region image), with a size of [3, 64, 128] (other values ​​are also possible, but no specific limitation is made in this embodiment).

[0111] In another example, regarding the network structure of the discriminator, the discriminator can extract image features through a series of convolutional layers. Each convolutional layer can be followed by batch normalization (BN) and LeakyReLU activation functions to enhance the non-linear expressive power of the model. For example (other values ​​can be used below, but no specific restrictions are made in this embodiment), the input of the first convolutional layer is [3, 64, 128] and the output is [64, 32, 64], the input of the second convolutional layer is [64, 32, 64] and the output is [128, 16, 32], the input of the third convolutional layer is [128, 16, 32] and the output is [256, 8, 16], and the input of the fourth convolutional layer is [256, 8, 16] and the output is [512, 4, 8]. The output of the fourth convolutional layer can be flattened by a fully connected layer, and then processed by a fully connected layer with an output size of [1]. The output of this layer can represent the discriminator's judgment that the input image is a real image (rather than a generated image). By using the sigmoid activation function, the output of the discriminator can be converted into a probability value between 0 and 1, which can represent the probability that the discriminator considers the input image to be a real image (i.e., the discrimination probability information).

[0112] In an alternative embodiment, the stride of the convolutional layer can be greater than 1 in the discriminator to replace the pooling layer for downsampling, thereby avoiding the information loss that may be caused by pooling operations.

[0113] In this embodiment, the first mouth region image and the second mouth region image are respectively used as input images and input to the discriminator. Then, the discriminator extracts features from the input images through each convolutional layer to obtain the mouth features of the input images. Based on the mouth features of the input images, the discrimination probability information output by the discriminator is obtained. By introducing a discriminator that focuses on mouth features, the clarity of tooth details can be optimized in the generative network.

[0114] In an exemplary embodiment, step S104, using the first sample data and the second sample data, performs adversarial training on the generator and the discriminator, which may include the following steps:

[0115] During the training of the generator using the first sample data, the generator parameters are adjusted based on the adversarial loss and content loss values. The training objective of the generator is to enhance the realism of the synthesized frame images output by the generator. The adversarial loss value is calculated based on the random noise vector of the synthesized frame images, and the content loss value is calculated based on the spatial metric distance of the real frame images. During the training of the discriminator using the second sample data, the discriminator parameters are adjusted based on the loss values ​​corresponding to the real frame images, the loss values ​​corresponding to the synthesized frame images, and the feature matching loss. The training objective of the discriminator is to reduce the error degree of the discrimination probability information output by the discriminator. The feature matching loss is determined based on the intermediate layer feature representation of the discriminator.

[0116] In the specific implementation, through adversarial training, the generator G aims to maximize the probability that the discriminator D is fooled, while the discriminator D aims to minimize the error rate in classifying real and fake images.

[0117] For example, content loss can be used for training the generator:

[0118]

[0119] Where x is the real image (i.e., the real frame image). It is Euclidean distance (i.e., spatial distance metric).

[0120] And employing adversarial loss:

[0121]

[0122] Where G(z) is the fake image generated by the generator G, and z is a random noise vector.

[0123] The generator G can be trained in the following way:

[0124] 1. Fixed discriminator D;

[0125] 2. Sample a batch of data from random noise z to generate a batch of fake images. (i.e., fake sample images);

[0126] 3. Calculate the adversarial loss L of generator G. adv (i.e., adversarial loss value) and content loss L content (i.e., content loss value);

[0127] 4. Update the parameters of generator G to minimize the parameters of generator G, L G = L content +λ adv L adv , where λ adv It is the weight that counteracts losses.

[0128] For example, the training of the discriminator can employ Real / Fake Image Loss:

[0129]

[0130]

[0131] in, express It comes from real data (i.e., the first mouth area image cropped from real frame images). This indicates that it comes from the generated data (i.e., the second mouth region image cropped from the synthetic frame image).

[0132] And Feature Matching Loss:

[0133]

[0134] in, It is the intermediate layer feature representation of discriminator D.

[0135] The discriminator D can be trained in the following way:

[0136] 1. Fixed generator G;

[0137] 2. Sample a batch of real images from real data;

[0138] 3. Sample a batch of fake images from generator G. ;

[0139] 4. Calculate the loss L of the discriminator D on real and fake images. real and L fake .

[0140] 5. Update the parameters of discriminator D to minimize L. total-D =L real +L fake +λ feat L feat , where λ fake These are the weights of the feature matching loss.

[0141] In this embodiment, by adjusting the generator parameters based on adversarial loss and content loss during the generator training process using the first sample data, and by adjusting the discriminator parameters based on the loss values ​​corresponding to real frame images, synthetic frame images, and feature matching loss during the discriminator training process using the second sample data, the clarity of optimized tooth details in the generator network can be improved based on adversarial training between the generator and the discriminator.

[0142] In one exemplary embodiment, such as Figure 5 As shown, a method for generating lip-shape images is provided. Taking the application of this method to a server as an example, the method includes the following steps S501 to S502. Wherein:

[0143] In step S501, based on the current processing frame in the original video, the audio to be processed and the face image to be processed are obtained;

[0144] The audio to be processed can be obtained from the video speech corresponding to the current processing frame in the original video. The face image to be processed can include a first face image and a second face image. The first face image can be the face image in the current processing frame, and the second face image can be the face image obtained by occluding the mouth area of ​​the face image in the current processing frame.

[0145] In step S502, the audio to be processed and the face image to be processed are input into the trained lip-sync image generation model to obtain a target face image with lip movements; the target face image is used to replace the face image in the current processing frame of the original video to generate the target video.

[0146] The trained lip shape image generation model can be trained using any of the lip shape image generation model training methods.

[0147] In practical applications, the MEL spectral features can be extracted from the input speech (i.e., the audio to be processed) to obtain speech features. Features can also be extracted from two sets of face images (i.e., the face images to be processed) with the full face and the lower half of the face removed. The extracted features can then be input into the trained wav2lip network (i.e., the trained lip-shape image generation model) to generate a face image with lip shape (i.e., the target face image).

[0148] In one example, the generated face image can be pasted back to the corresponding position in the original video (i.e., replace the face image corresponding to the current processing frame), and a new video can be generated frame by frame. Then, an audio-video stream conversion tool can be used to synthesize the generated video and the audio to be processed to obtain a video with lip movements (i.e., the target video).

[0149] In the above-mentioned lip shape image generation method, the audio to be processed and the face image to be processed are obtained based on the current processing frame in the original video. Then, the audio to be processed and the face image to be processed are input into the trained lip shape image generation model to obtain a target face image with lip shape. This target face image is used to replace the face image in the current processing frame of the original video to generate the target video. This method can optimize the clarity of tooth details and effectively improve the image quality of mouth images, especially the tooth area.

[0150] In one embodiment, such as Figure 6 The diagram illustrates a training method for another lip-shape image generation model. In this embodiment, the method includes the following steps:

[0151] In step S601, multiple real frame images are obtained by temporal sampling of the images in the sample video, and sample audio corresponding to the real frame images is obtained by temporal sampling of the audio in the sample video. For any real frame image, a frame image different from any real frame image is obtained by randomly sampling the sample video as a fake sample image. In step S602, the first sample data for training the generator in the lip-sync image generation model is obtained. The sample audio is processed by the audio encoder in the generator to obtain an audio feature vector. In step S603, the audio feature vector and the image feature vector of the sample face image are fused by the face encoder in the generator to obtain an encoded output result. The encoded output result is decoded by the face decoder in the generator to obtain a synthesized frame image. In step S604, the first mouth region image cropped from the real frame image and the second mouth region image cropped from the synthesized frame image are used as the second sample data for training the discriminator in the lip-sync image generation model. In step S605, during the process of training the generator using the first sample data, the parameters of the generator are adjusted based on the adversarial loss value and the content loss value. In step S606, during the training of the discriminator using the second sample data, the parameters of the discriminator are adjusted based on the loss values ​​corresponding to the real frame images, the loss values ​​corresponding to the synthesized frame images, and the feature matching loss, until the training termination condition is met, resulting in a trained lip shape image generation model. It should be noted that the specific limitations of the above steps can be found in the above description of the specific limitations of a training method for a lip shape image generation model, and will not be repeated here.

[0152] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0153] Based on the same inventive concept, this application also provides a training apparatus for a lip-shape image generation model to implement the training method for the lip-shape image generation model described above. The solution provided by this apparatus is similar to the implementation described in the above method. Therefore, the specific limitations of one or more lip-shape image generation model training apparatus embodiments provided below can be found in the limitations of the lip-shape image generation model training method described above, and will not be repeated here.

[0154] In one embodiment, such as Figure 7 As shown, a training device for a lip-shape image generation model is provided, comprising:

[0155] The first sample data acquisition module 701 is used to acquire first sample data for training the generator in the lip shape image generation model; the first sample data includes multiple sample pairs, each sample pair including a sample face image and sample audio of the corresponding time interval of the sample face image; the sample face image includes a real frame image with the mouth area covered and a fake sample image with a complete face, the real frame image is a frame image obtained by sampling a certain frame in the sample video, and the fake sample image is a frame image obtained by sampling frames in the sample video other than the certain frame;

[0156] The synthetic frame image acquisition module 702 is used to input the sample face image and the sample audio into the generator to obtain a synthetic frame image containing a complete face;

[0157] The second sample data acquisition module 703 is used to extract a first mouth region image from the real frame image and a second mouth region image from the synthesized frame image as second sample data for training the discriminator in the mouth shape image generation model.

[0158] The adversarial training module 704 is used to perform adversarial training on the generator and the discriminator using the first sample data and the second sample data until the training termination condition is met, thereby obtaining a trained lip-shape image generation model; the lip-shape image generation model is used to generate a face image with lip shape based on the input speech conversion.

[0159] In one embodiment, the apparatus further includes:

[0160] The sample acquisition module is used to perform time-series sampling of images in the sample video to obtain multiple real frame images, and to perform time-series sampling of audio in the sample video to obtain sample audio corresponding to the real frame images; the time interval of the sample audio matches the time interval of the real frame images.

[0161] The fake sample image acquisition module is used to obtain a frame image that is different from any real frame image by randomly sampling the sample video for any real frame image, and use it as the fake sample image.

[0162] In one embodiment, the synthetic frame image acquisition module 702 includes:

[0163] An audio encoding submodule is used to process the sample audio through an audio encoder in the generator to obtain an audio feature vector; wherein the audio encoder has a one-dimensional convolutional layer and a neural network based on a self-attention mechanism.

[0164] The generator fusion submodule is used to fuse the audio feature vector and the image feature vector of the sample face image through the face encoder in the generator to obtain the encoded output result;

[0165] The decoding output submodule is used to decode the encoded output result through the face decoder in the generator to obtain the synthesized frame image.

[0166] In one embodiment, the facial encoder includes multiple convolutional layers connected sequentially, and the generator fusion submodule includes:

[0167] The intermediate feature vector acquisition unit is used to perform convolution processing on the sample face image through the intermediate convolution layer among the multiple convolution layers of the face encoder to obtain the intermediate feature vector;

[0168] The feature vector fusion unit is used to fuse the audio feature vector and the intermediate feature vector based on the cross-attention mechanism to obtain the fused feature vector output by the intermediate convolutional layer;

[0169] The encoding output result obtaining unit is used to obtain the encoding output result based on the image feature vectors output by each convolutional layer in the face encoder except for the intermediate convolutional layer, and the fusion feature vector output by the intermediate convolutional layer.

[0170] In one embodiment, the apparatus further includes:

[0171] An image input module is used to input the first mouth region image and the second mouth region image as input images to the discriminator;

[0172] The mouth feature extraction module is used to extract features from the input image through each convolutional layer in the discriminator to obtain the mouth features of the input image;

[0173] The discriminator output module is used to obtain the discrimination probability information output by the discriminator based on the mouth features of the input image; the discrimination probability information is used to characterize the probability that the first mouth region image belongs to the real frame image and the probability that the second mouth region image belongs to the real frame image.

[0174] In one embodiment, the adversarial training module 704 includes:

[0175] The generator training submodule is used to adjust the parameters of the generator based on adversarial loss and content loss during the training process using the first sample data; the training objective of the generator is to enhance the realism of the synthesized frame image output by the generator; the adversarial loss is calculated based on the random noise vector of the synthesized frame image, and the content loss is calculated based on the spatial metric distance of the real frame image;

[0176] The discriminator training submodule is used to adjust the parameters of the discriminator based on the loss value corresponding to the real frame image, the loss value corresponding to the synthetic frame image, and the feature matching loss during the training process of the discriminator using the second sample data; the training objective of the discriminator is to reduce the error degree of the discrimination probability information output by the discriminator; the feature matching loss is determined based on the intermediate layer feature representation of the discriminator.

[0177] Each module in the training device for the aforementioned lip shape image generation model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0178] Based on the same inventive concept, this application also provides a lip shape image generation apparatus for implementing the lip shape image generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the lip shape image generation apparatus provided below can be found in the limitations of the lip shape image generation method described above, and will not be repeated here.

[0179] In one embodiment, such as Figure 8 As shown, a lip-shape image generation apparatus is provided, comprising:

[0180] The current processing frame data acquisition module 801 is used to acquire audio to be processed and face image to be processed based on the current processing frame in the original video; the audio to be processed is obtained based on the video speech corresponding to the current processing frame in the original video; the face image to be processed includes a first face image and a second face image, the first face image is the face image in the current processing frame, and the second face image is the face image obtained after occluding the mouth area of ​​the face image in the current processing frame;

[0181] The target face image acquisition module 802 is used to input the audio to be processed and the face image to be processed into a trained lip-sync image generation model to obtain a target face image with lip movements that match the audio to be processed; the target face image is used to replace the face image in the current processing frame of the original video to generate a target video;

[0182] The trained lip shape image generation model is obtained by training the lip shape image generation model according to any one of the above methods.

[0183] Each module in the aforementioned lip-shape image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0184] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a training method for a lip-sync image generation model.

[0185] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores lip-shape image generation data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a lip-shape image generation method.

[0186] Those skilled in the art will understand that Figure 9 and Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0187] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0188] In one exemplary embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0189] In one exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0190] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0191] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for a lip-shape image generation model, characterized in that, The method includes: First sample data is obtained for training the generator in the lip-shape image generation model; the first sample data includes multiple sample pairs, each sample pair including a sample face image and sample audio; the sample face image includes a real frame image with the mouth area obscured and a fake sample image with a complete face, the real frame image is a frame image sampled from a certain frame in the sample video, the fake sample image is a frame image sampled from frames in the sample video other than the certain frame, and the time interval of the sample audio matches the time interval of the real frame image; The sample face image and the sample audio are input into the generator. The intermediate convolutional layer of the face encoder in the generator performs convolution processing on the sample face image to obtain an intermediate feature vector. The audio feature vector of the sample audio and the intermediate feature vector are fused based on a cross-attention mechanism to obtain a fused feature vector output by the intermediate convolutional layer. Based on the image feature vectors output by each convolutional layer in the face encoder other than the intermediate convolutional layer and the fused feature vector, an encoded output result is obtained. The encoded output result is decoded and output to obtain a synthetic frame image containing a complete face. The face encoder includes multiple convolutional layers connected sequentially, and the intermediate convolutional layer is one of the multiple convolutional layers. The first mouth region image cropped from the real frame image and the second mouth region image cropped from the synthesized frame image are used as the second sample data for training the discriminator in the mouth shape image generation model. Using the first sample data and the second sample data, the generator and the discriminator are subjected to adversarial training until the training termination condition is met, resulting in a trained lip-shape image generation model; the lip-shape image generation model is used to generate a face image with lip shapes based on the input speech conversion.

2. The method according to claim 1, characterized in that, The method further includes: The images in the sample video are temporally sampled to obtain multiple real frame images, and the audio in the sample video is temporally sampled to obtain sample audio corresponding to the real frame images; the time interval of the sample audio matches the time interval of the real frame images. For any real frame image, a frame image different from any real frame image is obtained by randomly sampling the sample video and used as the fake sample image.

3. The method according to claim 1, characterized in that, The step of inputting the sample face image and the sample audio into the generator to obtain a synthetic frame image containing a complete face includes: The sample audio is processed by the audio encoder in the generator to obtain an audio feature vector; wherein the audio encoder has a one-dimensional convolutional layer and a neural network based on a self-attention mechanism. The encoder in the generator fuses the audio feature vector and the image feature vector of the sample face image to obtain the encoded output result. The encoded output is decoded by the face decoder in the generator to obtain the synthesized frame image.

4. The method according to claim 1, characterized in that, The steps for training the discriminator include: The first mouth region image and the second mouth region image are respectively used as input images and input to the discriminator; The mouth features of the input image are obtained by extracting features from each convolutional layer in the discriminator. Based on the mouth features of the input image, the discrimination probability information output by the discriminator is obtained; the discrimination probability information is used to characterize the probability that the first mouth region image belongs to the real frame image and the probability that the second mouth region image belongs to the real frame image.

5. The method according to claim 1, characterized in that, The step of using the first sample data and the second sample data to perform adversarial training on the generator and the discriminator includes: During the training of the generator using the first sample data, the parameters of the generator are adjusted based on the adversarial loss value and the content loss value; the training objective of the generator is to enhance the realism of the synthesized frame image output by the generator; the adversarial loss value is calculated based on the random noise vector of the synthesized frame image, and the content loss value is calculated based on the spatial metric distance of the real frame image; During the training of the discriminator using the second sample data, the parameters of the discriminator are adjusted based on the loss value corresponding to the real frame image, the loss value corresponding to the synthetic frame image, and the feature matching loss; the training objective of the discriminator is to reduce the error degree of the discrimination probability information output by the discriminator; the feature matching loss is determined based on the intermediate layer feature representation of the discriminator.

6. A method for generating lip shape images, characterized in that, The method includes: Based on the current processing frame in the original video, obtain the audio to be processed and the face image to be processed; the audio to be processed is obtained based on the video speech corresponding to the current processing frame in the original video; the face image to be processed includes a first face image and a second face image, the first face image is the face image in the current processing frame, and the second face image is the face image obtained after occluding the mouth area of ​​the face image in the current processing frame; The audio to be processed and the face image to be processed are input into a trained lip-sync image generation model to obtain a target face image with lip movements that match the audio to be processed; the target face image is used to replace the face image in the current processing frame of the original video to generate the target video; The trained lip shape image generation model is obtained by training the lip shape image generation model according to any one of claims 1-5.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images