Mouth shape image generation model training method and mouth shape image generation method

Abstract

The invention relates to a training method of a mouth shape image generation model and a mouth shape image generation method. The method comprises the following steps: acquiring first sample data used for training a generator in a mouth shape image generation model; the first sample data comprises a plurality of sample pairs; inputting the sample face image and the sample audio into a generator to obtain a synthetic frame image containing a complete face; taking a first mouth region image cut from the real frame image and a second mouth region image cut from the synthetic frame image as second sample data for training a discriminator in the mouth shape image generation model; and carrying out confrontation training on the generator and the discriminator by adopting the first sample data and the second sample data until a training ending condition is met, and obtaining a trained mouth shape image generation model. By adopting the method, the trained mouth shape image generation model can generate a high-definition tooth image, and the overall quality of mouth shape image generation is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for training a lip image generation model, a lip image generation method, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of computer technology, deep learning models can be used to generate lip-sync videos, such as voice-driven mouth and face generation videos.

[0003] In the related art, traditional lip-sync generation technology is usually aimed at mapping from speech to lip shape. It has made some progress in generating lip-sync videos of specific speakers and can learn and reproduce lip movements synchronized with speech. For example, by learning the correspondence between input audio and lip-sync marks, the system can achieve the generation and synchronization of lip shapes for specific speakers. However, traditional lip-sync generation methods are insufficient in terms of clarity and detail in the tooth area, resulting in the inability of the generated lip-sync video to clearly present the details of the teeth. The tooth area appears blurred and lacks realism, which affects the generation effect of the lip-sync video. Summary of the invention

[0004] Based on this, it is necessary to provide a training method for a lip image generation model, a lip image generation method, a computer device, a computer-readable storage medium and a computer program product that can improve the display effect of the dental area in order to solve the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a method for training a lip image generation model. The method comprises:

[0006] Acquire first sample data for training a generator in the lip-image generation model; the first sample data includes a plurality of sample pairs, each of which includes a sample face image and a sample audio of a time interval corresponding to the sample face image; the sample face image includes a real frame image with a mouth area blocked and a fake sample image with a complete face, the real frame image is a frame image obtained by sampling a certain frame in a sample video, and the fake sample image is a frame image obtained by sampling frames in the sample video other than the certain frame;

[0007] Inputting the sample face image and the sample audio into the generator to obtain a synthetic frame image containing a complete face;

[0008] Using a first mouth region image cropped from the real frame image and a second mouth region image cropped from the synthetic frame image as second sample data for training a discriminator in the lip shape image generation model;

[0009] The generator and the discriminator are adversarially trained using the first sample data and the second sample data until the training end condition is met, thereby obtaining a trained lip image generation model; the lip image generation model is used to generate a face image with lip shape according to input speech conversion.

[0010] In one embodiment, the method further comprises:

[0011] Performing time-sequential sampling on the images in the sample video to obtain a plurality of the real frame images, and performing time-sequential sampling on the audio in the sample video to obtain sample audio corresponding to the real frame image; the time interval of the sample audio matches the time interval of the real frame image;

[0012] For any real frame image, by randomly sampling the sample video, a frame image different from the real frame image is obtained as the false sample image.

[0013] In one embodiment, the step of inputting the sample face image and the sample audio into the generator to obtain a synthetic frame image containing a complete face includes:

[0014] The sample audio is processed by an audio encoder in the generator to obtain an audio feature vector; wherein the audio encoder has a one-dimensional convolutional layer and a neural network based on a self-attention mechanism;

[0015] The facial encoder in the generator fuses the audio feature vector and the image feature vector of the sample face image to obtain a coding output result;

[0016] The encoded output result is decoded and outputted by a facial decoder in the generator to obtain the synthesized frame image.

[0017] In one embodiment, the facial encoder includes a plurality of convolutional layers connected front and back; the encoding output result obtained by fusing the audio feature vector and the image feature vector of the sample face image through the facial encoder in the generator includes:

[0018] Performing convolution processing on the sample face image through an intermediate convolution layer among the multiple convolution layers of the face encoder to obtain an intermediate feature vector;

[0019] Fusing the audio feature vector and the intermediate feature vector based on a cross attention mechanism to obtain a fused feature vector output by the intermediate convolutional layer;

[0020] The encoding output result is obtained according to the image feature vectors output by each convolution layer except the middle convolution layer in the facial encoder and the fusion feature vector output by the middle convolution layer.

[0021] In one embodiment, the step of training the discriminator comprises:

[0022] inputting the first mouth region image and the second mouth region image as input images to the discriminator;

[0023] Extracting features of the input image through each convolutional layer in the discriminator to obtain mouth features of the input image;

[0024] According to the mouth features of the input image, the discriminant probability information output by the discriminator is obtained; the discriminant probability information is used to characterize the probability that the first mouth area image input belongs to the real frame image and the probability that the second mouth area image belongs to the real frame image.

[0025] In one embodiment, the using the first sample data and the second sample data to perform adversarial training on the generator and the discriminator includes:

[0026] In the process of training the generator using the first sample data, the parameters of the generator are adjusted based on the adversarial loss value and the content loss value; the training goal of the generator is to enhance the authenticity of the synthetic frame image output by the generator; the adversarial loss value is calculated based on the random noise vector of the synthetic frame image, and the content loss value is calculated based on the spatial metric distance of the real frame image;

[0027] In the process of training the discriminator using the second sample data, the parameters of the discriminator are adjusted based on the loss value corresponding to the real frame image, the loss value corresponding to the synthetic frame image, and the feature matching loss; the training goal of the discriminator is to reduce the error degree of the discriminant probability information output by the discriminator; the feature matching loss is determined based on the intermediate layer feature representation of the discriminator.

[0028] In a second aspect, the present application also provides a method for generating a lip shape image. The method comprises:

[0029] Based on the current processing frame in the original video, obtain the audio to be processed and the facial image to be processed; the audio to be processed is obtained based on the video voice corresponding to the current processing frame in the original video; the facial image to be processed includes a first facial image and a second facial image, the first facial image is the facial image in the current processing frame, and the second facial image is the facial image obtained after the mouth area of ​​the facial image in the current processing frame is blocked;

[0030] Inputting the audio to be processed and the facial image to be processed into a trained lip-image generation model to obtain a target facial image with a lip shape that matches the audio to be processed; the target facial image is used to replace the facial image in the current processing frame in the original video to generate a target video;

[0031] Wherein, the trained lip-image generation model is obtained by training using any one of the lip-image generation model training methods.

[0032] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the training method of the lip image generation model as described in the first aspect and / or the steps of the lip image generation method as described in the second aspect when executing the computer program.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for training a lip-image generation model as described in the first aspect and / or the steps of the method for generating a lip-image as described in the second aspect are implemented.

[0034] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, which, when executed by a processor, implements the steps of the method for training a lip-image generation model as described in the first aspect, and / or the steps of the method for generating a lip-image as described in the second aspect.

[0035] The above-mentioned training method of a lip-shaped image generation model, lip-shaped image generation method, computer device, computer-readable storage medium and computer program product obtain first sample data for training a generator in a lip-shaped image generation model, the first sample data includes multiple sample pairs, each sample pair includes a sample face image and a sample audio of a time interval corresponding to the sample face image, the sample face image includes a real frame image with a mouth area blocked and a fake sample image with a complete face, the real frame image is a frame image obtained by sampling a certain frame in a sample video, and the fake sample image is a frame image obtained by sampling frames other than a certain frame in the sample video, and then the sample face image and the sample audio are input into the generator to obtain a synthetic frame image containing a complete face, and the real frame image and the sample audio are input into the generator to obtain a synthetic frame image containing a complete face. A first mouth area image cropped from a real frame image and a second mouth area image cropped from a synthetic frame image are used as second sample data for training a discriminator in a lip image generation model. The first sample data and the second sample data are then used to perform adversarial training on the generator and the discriminator until the training end condition is met, thereby obtaining a trained lip image generation model. The lip image generation model is used to generate a face image with a lip shape according to input speech conversion, thereby optimizing the lip image generation model. The introduced discriminator pays special attention to the generation effect of the mouth area in the image, and combined with the adversarial training of the generator and the discriminator, the trained lip image generation model can produce high-definition teeth images, thereby effectively improving the overall quality of lip image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic flow chart of a method for training a lip image generation model in one embodiment;

[0037] FIG2a is a schematic diagram of a generator processing flow in one embodiment;

[0038] FIG2b is a schematic diagram of a facial key point detection in one embodiment;

[0039] FIG2c is a schematic diagram of a mouth region in one embodiment;

[0040] FIG3a is a schematic diagram of a face region in one embodiment;

[0041] FIG3b is a schematic diagram of audio feature sampling in one embodiment;

[0042] FIG4 is a schematic diagram of a cross-attention mechanism in one embodiment;

[0043] FIG5 is a schematic flow chart of a method for generating a mouth shape image in one embodiment;

[0044] FIG6 is a flow chart of a method for training a lip shape image generation model in another embodiment;

[0045] FIG7 is a block diagram of a training device for a lip image generation model according to an embodiment;

[0046] FIG8 is a block diagram of a device for generating a mouth shape image in one embodiment;

[0047] FIG9 is a diagram showing the internal structure of a computer device in one embodiment;

[0048] FIG. 10 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] In an exemplary embodiment, as shown in FIG1 , a method for training a lip image generation model is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be implemented as an independent server or a server cluster composed of multiple servers; the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, portable wearable devices, etc.

[0051] In this embodiment, the method includes the following steps S101 to S104. Among them:

[0052] In step S101, first sample data for training a generator in a lip-image generation model is obtained; the first sample data includes a plurality of sample pairs, each sample pair including a sample face image and a sample audio of a time interval corresponding to the sample face image.

[0053] As an example, the lip image generation model may be a deep learning model designed based on a generative adversarial network, which may include a generator and a discriminator.

[0054] Among them, the sample face image may include a real frame image with the mouth area blocked and a fake sample image with a complete face, the real frame image is a frame image sampled from a certain frame in the sample video, and the fake sample image is a frame image sampled from frames other than a certain frame in the sample video.

[0055] As an example, the time interval may be the time range of the frame where the real frame image is located in the sample video, or the time range of the frame where the image is located 5 frames (or other values, not specifically limited in this embodiment) near the real frame image.

[0056] In practical applications, in the training phase of the lip-image generation model, multiple sample pairs may be constructed as first sample data, so as to train a generator in the training lip-image generation model using the first sample data.

[0057] Exemplarily, in the process of training the generator, the generator processing flow shown in Figure 2a, by obtaining a sample video, time-series sampling can be performed based on the sample video to obtain a real frame image, and the audio in the sample video can be time-series sampled to obtain a sample audio of the time interval corresponding to the real frame image, the lower half of the face in the real frame image can be occluded (such as the time-series sampling image without the lower half of the face in Figure 2a), and random sampling can be performed based on the sample video to obtain a frame image different from the sampled real frame image as a false sample image (such as the randomly sampled complete face image in Figure 2a), so as to construct a sample pair based on image and audio (such as the two images and a piece of audio contained in the dotted box in Figure 2a).

[0058] In step S102, a sample face image and a sample audio are input into a generator to obtain a synthetic frame image containing a complete face.

[0059] In the specific implementation, the goal of the deep learning-based generator is to generate an image corresponding to the lip shape based on the reference image and audio (i.e., sample face image and sample audio) that do not correspond to the lip shape, as shown in Figure 2a, that is, to obtain a synthetic frame image containing a complete face.

[0060] In step S103, the first mouth region image cropped from the real frame image and the second mouth region image cropped from the synthetic frame image are used as second sample data for training the discriminator in the lip shape image generation model.

[0061] After obtaining the synthesized frame image, the first mouth region image can be cropped from the real frame image and the second mouth region image can be cropped from the synthesized frame image by identifying the mouth region in the image. Optionally, a facial key point detection algorithm can be used to identify the mouth region in the image. As shown in FIG2b , 68 facial key points can be obtained by the facial key point detection algorithm, where the two-dimensional coordinate point of each facial key point can be [x1, y1], ..., [x 68 ,y 68 ], x is the horizontal axis coordinate and y is the vertical axis coordinate.

[0062] In one example, the position and size of the mouth area can be determined by the coordinates of each facial key point, and then the mouth area can be accurately identified based on the position information of the identified facial key points, as shown in FIG2c. For example, the left and right boundaries of the mouth area can be determined based on the minimum X coordinate and the maximum X coordinate of the mouth key points, and the upper and lower boundaries of the mouth area can be determined based on the minimum Y coordinate and the maximum Y coordinate of the mouth key points. Thus, the corresponding mouth area image can be cropped based on the accurately identified mouth area, which can help generate an image corresponding to the mouth shape.

[0063] In step S104, the first sample data and the second sample data are used to perform adversarial training on the generator and the discriminator until the training end condition is met, thereby obtaining a trained lip image generation model.

[0064] Among them, the lip shape image generation model can be used to generate a face image with lip shape according to the input speech conversion.

[0065] Specifically, by sampling image data from random noise to generate a fake image (i.e., a fake sample image), and creating multiple sample pairs, a first sample data is obtained, and the first sample data can be used to train the generator; by sampling a real image (i.e., a real frame image) from real data, and sampling a fake image (i.e., a synthetic frame image) based on the output data of the generator, a second sample data is obtained, and the discriminator can be trained with the second sample data. At the same time, adversarial training can be performed for the generator and the discriminator, such as alternating the training process of the generator and the discriminator until a certain number of iterations or performance indicators are reached (i.e., the training end condition is met). The above training strategy can reflect the competitive relationship between the generator and the discriminator in the adversarial network, so that the two can improve their respective performance through continuous confrontation.

[0066] Compared with traditional methods, the technical solution of this embodiment, by introducing a discriminator focusing on mouth features, can optimize the clarity of tooth details in the generative network, so as to more accurately measure the tooth generation effect of the model in complex videos, and effectively improve the image quality of the mouth image, especially the tooth area.

[0067] In the training method of the above-mentioned lip image generation model, by obtaining the first sample data for training the generator in the lip image generation model, and then inputting the sample face image and the sample audio into the generator to obtain a synthetic frame image containing a complete face, the first mouth area image cropped from the real frame image and the second mouth area image cropped from the synthetic frame image are used as the second sample data for training the discriminator in the lip image generation model, and then using the first sample data and the second sample data, adversarial training is performed on the generator and the discriminator until the training end condition is met, and the trained lip image generation model is obtained, thereby realizing the optimization of the lip image generation model. Based on the introduced discriminator, special attention is paid to the generation effect of the mouth area in the image, and combined with the adversarial training of the generator and the discriminator, the trained lip image generation model can produce high-definition teeth images, effectively improving the overall quality of lip image generation.

[0068] In an exemplary embodiment, the following steps may also be included:

[0069] The images in the sample video are sampled in time series to obtain multiple real frame images, and the audio in the sample video is sampled in time series to obtain sample audio corresponding to the real frame image; the time interval of the sample audio matches the time interval of the real frame image; for any real frame image, the sample video is randomly sampled to obtain a frame image different from any real frame image as a false sample image.

[0070] In practical applications, for the data preprocessing stage, the sample video can be processed frame by frame by resampling the sample video to 25FPS and sampling all the audio in the sample video to 16KHz; the face recognition algorithm can be used to obtain the face area in each frame image, as shown in Figure 3a, and the obtained face image can be resized to 256x256 as a real frame image.

[0071] In an optional embodiment, other preprocessing operations may be performed on the obtained facial image, such as normalization processing to scale the pixel values ​​of the image to the range of [-1, 1], and whitening processing, etc., to facilitate subsequent model training.

[0072] In one example, for audio processing, the mel (mel spectrum) feature of the sample audio can be extracted, and the mel feature extraction parameter can be a frame length of 25ms (i.e., 400 sampling points); the sample audio is converted into an 80-dimensional vector through short-time Fourier transform. If the step length is 10ms, 1 second of audio can be converted into an 80*100 feature. For example, as shown in Figure 3b, a 1-second video contains 25 images, and the corresponding mel feature is 80*100.

[0073] In another example, when constructing image and audio feature pairs (i.e., sample pairs), each image may correspond to a matrix with a dimension of [3, 256, 256] (e.g., an image has 256x256 pixels, each pixel has three channels, and may also be other values, which are not specifically limited in this embodiment), and the audio feature dimension of the time interval corresponding to the image is [80, 4] (it may also be other values, which are not specifically limited in this embodiment); in order to accommodate the error of the data set, as shown in FIG3b, the audio corresponding to the 5 frames of images near each frame of the image may be matched with the frame of the image, and the audio feature dimension of the time interval corresponding to each frame of the image (i.e., the real frame image) is [80, 20] (it may also be other values, which are not specifically limited in this embodiment). For the reference image in the image and audio feature pair, it consists of two parts. One part is the real frame image with the lower half of the face covered, and the other part is a randomly sampled complete face image (i.e., a fake sample image). The dimension of the image input to the generator corresponds to [6, 256, 256] (it can also be other values, which are not specifically limited in this embodiment).

[0074] In this embodiment, multiple real frame images are obtained by time-series sampling of images in a sample video, and sample audio corresponding to the real frame images is obtained by time-series sampling of audio in the sample video. Then, for any real frame image, a frame image different from any real frame image is obtained as a false sample image by randomly sampling the sample video, thereby providing data support for the training generator.

[0075] In an exemplary embodiment, in step S102, inputting a sample face image and a sample audio into a generator to obtain a synthetic frame image containing a complete face may include the following steps:

[0076] The sample audio is processed by the audio encoder in the generator to obtain an audio feature vector; the audio feature vector and the image feature vector of the sample face image are fused by the facial encoder in the generator to obtain an encoded output result; the encoded output result is decoded and output by the facial decoder in the generator to obtain a synthetic frame image.

[0077] The audio encoder has a one-dimensional convolutional layer and a neural network based on a self-attention mechanism.

[0078] In a specific implementation, the mel feature dimension corresponding to the sample audio input to the Audio Encoder is [80, 20] (it can also be other values, which are not specifically limited in this embodiment), and the combination of the neural network layer in the audio encoder can be 1D Conv + BN + ReLU, which can include a one-dimensional convolution layer, batch normalization (BN) and ReLU activation function. Among them, the number of filters in the one-dimensional convolution layer can be set to 512, the filter size can be set to 10, and the step size is 2, then the feature size of the one-dimensional convolution layer output is [512, 11] (it can also be other values, which are not specifically limited in this embodiment).

[0079] In an optional embodiment, a Transformer encoder (i.e., a neural network based on a self-attention mechanism) may be used, and the Transformer encoder may include multiple self-attention layers and feed-forward neural network layers. The input of the Transformer encoder is a sequence, such as the feature output by the one-dimensional convolutional layer can be regarded as a sequence of length 11, and the dimension of each element is 512. By inputting it into the Transformer encoder and processing it using a self-attention layer and a feed-forward neural network layer, the output of the Transformer encoder can be obtained, wherein the output dimension of the self-attention layer is [512, 11].

[0080] In one example, the output of the Transformer encoder can be globally averaged and pooled. A sequence can be converted into a vector by global average pooling, and each element of the sequence can be averaged, so that the output feature size can be

[512] . Based on the obtained 512-dimensional vector (i.e., audio feature vector), the vector can be used as the output of the audio encoder for subsequent calculations or tasks. Since the audio encoder only uses one one-dimensional convolutional layer and one Transformer encoder, the design has a small amount of computation and can effectively improve processing efficiency.

[0081] In this embodiment, the sample audio is processed by the audio encoder in the generator to obtain an audio feature vector, and then the audio feature vector and the image feature vector of the sample face image are fused by the facial encoder in the generator to obtain an encoding output result, and then the encoding output result is decoded and output by the facial decoder in the generator to obtain a synthetic frame image. The audio feature vector can be obtained based on the audio encoder processing, which provides data support for further fusion of the audio feature vector and the image feature vector.

[0082] In an exemplary embodiment, the facial encoder may include a plurality of convolutional layers connected front and back, and the facial encoder in the generator may fuse the audio feature vector and the image feature vector of the sample face image to obtain the encoding output result, which may include the following steps:

[0083] The sample face image is convolved through the middle convolution layer in the multiple convolution layers of the facial encoder to obtain an intermediate feature vector; the audio feature vector and the middle feature vector are fused based on the cross attention mechanism to obtain a fused feature vector output by the middle convolution layer; the encoding output result is obtained based on the image feature vectors output by each convolution layer except the middle convolution layer in the facial encoder and the fused feature vector output by the middle convolution layer.

[0084] In practical applications, a Face Encoder can be used to process the sample face image. The purpose of the Face Encoder is to extract the feature representation of the image. It can be a CNN (Convolutional Neural Networks) with multiple convolutional layers and pooling layers. The feature dimension of the sample face image input by the Face Encoder is [6, 256, 256] (it can also be other values, which are not specifically limited in this embodiment), which is obtained by splicing two images, such as a real frame image with the mouth area blocked and a fake sample image with a complete face.

[0085] In one example, the facial encoder may include a series of convolutional layers, each of which may be followed by batch normalization (BN) and ReLU activation functions. For example, the output size of the first layer may be [64, 128, 128] (it may also be other values, which are not specifically limited in this embodiment), the output size of the second layer may be [128, 64, 64] (it may also be other values, which are not specifically limited in this embodiment), and so on, with each layer output gradually decreasing in size and increasing in number of channels. Each convolutional layer may be followed by a maximum pooling layer, which may be used to gradually reduce the spatial size of the feature map; after being processed by multiple convolutional layers and pooling layers, a smaller-sized feature map may be obtained, such as [1024, 8, 8] (it may also be other values, which are not specifically limited in this embodiment), which may be used as the output of the facial encoder (i.e., the encoding output result).

[0086] Specifically, in the middle layer of the facial encoder, a feature map can be selected based on the Cross Attention module to perform cross-attention fusion of audio features. For example, the audio feature vector can be a 512-dimensional vector obtained from the audio encoder; the image feature vector can be an intermediate feature map (i.e., intermediate feature vector) in the selected facial encoder, such as [1024, 8, 8] (it can also be other values, which are not specifically limited in this embodiment). In the cross-attention processing process, the Query can be set to each position of the image feature map, such as [8*8, 1024], the Key can be set to map the audio feature vector through a linear layer (fully connected layer) to a space with the same dimension as the image feature, such as [L (fully connected layer), 1024], and the Value can be set to be the same as the Key. Through attention calculation, the similarity with the Key can be calculated for each Query, such as using a dot product, and the softmax function can be used to normalize the similarity score to determine the attention weight, and then the attention weight can be used to weight the Value and add or multiply it with the original image feature to achieve feature fusion.

[0087] In an optional embodiment, for the cross-attention mechanism, it can allow the model to refer to another related sequence when processing an input sequence. This mechanism can be used for natural language processing tasks and computer vision tasks, such as machine translation, text summarization, image description generation, etc. In the cross-attention, it contains a Query (Q), a Key (K) and a Value (V), as shown in Figure 4, the Query comes from the input being processed (such as the latent array in Figure 4), and the Key and Value come from the sequence that needs to be referenced (such as the input array in Figure 4). The goal of the attention mechanism is to calculate the similarity between the Query and each Key, and then use the similarity to perform a weighted summation on the Value to generate a new context vector that combines the information of the input array and the latent array.

[0088] As shown in Figure 4, specifically, the calculation process of the cross-attention mechanism is as follows:

[0089] First, through linear transformation (such as fully connected layer processing), the input array and latent array can be converted into Query, Key and Value. For example, we can have Q = latent array * W q , K = input array * W k , V = input array * W v , where W q , W kand W v is the weight matrix that needs to be learned.

[0090] In the example in Figure 4, the dimension of the input array is [M, C], where M is the length of the input sequence and C is the dimension of the input feature. The dimension of the latent array is [N, D], where N is the length of the latent sequence and D is the dimension of the latent feature. The input array and latent array can be converted into Query, Key, and Value through linear transformation. Assume that the weight matrix W q The dimension is [D, D'], and the weight matrix W k and W v The dimensions of are [C, D'] and [C, C'], then:

[0091] Q = latent array * W q , so the dimension of Q is [N, D'].

[0092] K = input array *W k , so the dimension of K is [M, D'].

[0093] V = input array * W v , so the dimension of V is [N, C'].

[0094] The obtained D' and C' are new feature dimensions, which can be set according to the specific task and model structure. In practical applications, D' and C' can be set to a smaller value to reduce computational complexity and prevent overfitting.

[0095] Then, we can calculate the dot product between Q and each K to get a similarity matrix, which can be [M, N]. By scaling the similarity matrix, such as dividing by the square root of D', applying the softmax function so that the sum of each row is 1, we can get an attention weight matrix.

[0096] Finally, the attention weight matrix [M, N] can be used to multiply the [N, C']-dimensional V to obtain an [M, C']-dimensional output vector, which combines the information of the input array and the latent array.

[0097] In another example, for the audio and video feature fusion process, by extracting audio feature vectors, such as using an audio encoder to extract a 512-dimensional feature vector of audio, and extracting image feature vectors and selecting intermediate layer features, such as using a facial encoder to extract a multi-scale feature representation of an image, and selecting an intermediate layer feature map from the facial encoder for cross-attention fusion. During the fusion process, the audio feature vector can be mapped to the same dimension as the image feature vector through a linear layer, and then the cross-attention between the image feature vector and the mapped audio feature vector can be calculated, and then the obtained attention weight can be used to weight the audio feature vector and fuse it with the image feature vector (that is, the audio feature vector and the intermediate feature vector are fused based on the cross-attention mechanism to obtain the fused feature vector output by the intermediate convolution layer). Among them, since the input of the self-attention mechanism is in sequence form, when processing image data, it is necessary to convert the two-dimensional feature map into a sequence form. When using cross-attention, the Query and Key need to be reshaped into a two-dimensional form to facilitate the calculation of the attention weight.

[0098] For example, the reshaping process and calculation steps of Query and Key in cross attention are as follows:

[0099] 1. Reshaping of Query: For an image feature vector, if the feature map size obtained from the encoder is [C, H, W] (C is the number of channels, H and W are height and width respectively), it needs to be reshaped into [H* W, C] to form a sequence; the dimension of the output Q is [H*W, C];

[0100] 2. Reshaping of Key and Value: For the audio feature vector, its original dimension is

[512] . In cross attention, it needs to be expanded to the same number of channels C as the image feature vector, such as mapping it to [L, C] through a linear layer, so that Key and Value can match the reshaped Query; the dimension of output K is [L, C], and the dimension of output V is [L, C];

[0101] 3. Calculate attention weights: The reshaped Query and Key can be used to calculate the attention score, such as through dot product, and then the softmax function can be applied to obtain the normalized attention weights; the output softmax (Q*K^T) has a dimension of [H * W, L];

[0102] 4. Apply the attention weight. You can use the obtained attention weight to weight the Value, and then reshape the weighted Value back to the original image feature vector dimension as needed to merge with the image feature; the dimension of the output weighted Value is [H*W, C];

[0103] 5. Fusion features: The weighted Value can be fused with the original image features, such as by addition, multiplication or other fusion techniques; the output dimension is [H*W, C];

[0104] 6. Reshape features back to image size: The fused features can be reshaped from the sequence form [H*W, C] back to the two-dimensional feature map [C, H, W] for further processing or as input to the decoder.

[0105] Through the above steps, cross attention allows the model to effectively exchange information between different modalities, audio features can guide the update of image features, and vice versa, thereby enhancing the model's understanding and processing capabilities of multimodal data. The generative network is optimized through a supervised learning framework, enabling it to produce high-definition dental images and enhance visual realism.

[0106] In practical applications, a Face Decoder can be used to gradually upsample the output of the face encoder (i.e., the encoding output result) to restore the spatial size of the original image. For example, the input of the face decoder can be a feature map output by the face encoder, such as [1024, 8, 8] (it can also be other values, which are not specifically limited in this embodiment). When upsampling, a transposed convolution or upsampling technique can be used to gradually increase the spatial size of the feature map. Then, a convolution layer can be connected after each upsampling step. The convolution layer can be used to refine the features, and then the outputs of the corresponding levels in the encoder and decoder can be spliced. The spatial information can be restored through jump connections. The output of the face decoder can be an image (i.e., a synthetic frame image) upsampled to the original image size [3, 256, 256] (it can also be other values, which are not specifically limited in this embodiment). Therefore, through the above processing method, the model can effectively combine audio and image information and enhance feature representation in the middle layer, effectively improving the performance of multimodal tasks.

[0107] In this embodiment, the sample face image is convolved by the middle convolution layer in the multiple convolution layers of the facial encoder to obtain an intermediate feature vector, and then the audio feature vector and the middle feature vector are fused based on the cross-attention mechanism to obtain a fused feature vector output by the middle convolution layer. Then, the encoding output result is obtained according to the image feature vectors output by each convolution layer except the middle convolution layer in the facial encoder and the fused feature vector output by the middle convolution layer, which can effectively combine audio and image information, enhance feature representation, and improve the performance of the lip image generation model.

[0108] In an exemplary embodiment, the following steps may also be included:

[0109] The first mouth area image and the second mouth area image are respectively used as input images and input into the discriminator; the input image is feature extracted through each convolution layer in the discriminator to obtain the mouth features of the input image; according to the mouth features of the input image, the discriminant probability information output by the discriminator is obtained; the discriminant probability information is used to characterize the probability that the input first mouth area image belongs to the real frame image and the probability that the second mouth area image belongs to the real frame image.

[0110] In one example, by using the face recognition algorithm in the data preprocessing stage to determine the range of the mouth area, the mouth area can be cropped from the real frame image and the synthetic frame image respectively, and can be adjusted to a fixed size, such as [64, 128] (it can also be other values, which are not specifically limited in this embodiment); the input of the discriminator is the resized mouth area image (i.e., the first mouth area image and the second mouth area image), and the size is [3, 64, 128] (it can also be other values, which are not specifically limited in this embodiment).

[0111] In another example, with respect to the network structure of the discriminator, the discriminator can extract image features through a series of convolutional layers, and each convolutional layer can have batch normalization (BN) and LeakyReLU activation functions after it to enhance the nonlinear expression ability of the model; for example (other values ​​may be used as follows, which are not specifically limited in this embodiment), the input of the first convolutional layer is [3, 64, 128], and the output is [64, 32, 64], the input of the second convolutional layer is [64, 32, 64], and the output is [128, 16, 32], the input of the third convolutional layer is [128, 16, 32], and the output is [256, 8, 16], and the input of the fourth convolutional layer is [256, 8, 16], and the output is [512, 4, 8]. The output of the fourth convolutional layer can be flattened through a fully connected layer, and then processed through a fully connected layer with an output size of [1]. The output of this layer can represent the discriminator's judgment that the input image is a real image (not a generated image). Through the Sigmoid activation function, the discriminator's output can be converted into a probability value between 0 and 1, which can represent the probability that the discriminator believes that the input image is a real image (i.e., the discriminant probability information).

[0112] In an optional embodiment, a convolutional layer with a stride greater than 1 may be used in the discriminator to replace the pooling layer for downsampling, thereby avoiding possible information loss caused by the pooling operation.

[0113] In this embodiment, the first mouth area image and the second mouth area image are respectively used as input images and input into the discriminator. Then, the features of the input images are extracted through each convolutional layer in the discriminator to obtain the mouth features of the input image. Then, based on the mouth features of the input image, the discrimination probability information output by the discriminator is obtained. Based on the introduction of the discriminator focusing on the mouth features, the clarity of the tooth details can be optimized in the generative network.

[0114] In an exemplary embodiment, in step S104, using the first sample data and the second sample data to perform adversarial training on the generator and the discriminator may include the following steps:

[0115] In the process of training the generator with the first sample data, the parameters of the generator are adjusted based on the adversarial loss value and the content loss value; the training goal of the generator is to enhance the authenticity of the synthetic frame image output by the generator; the adversarial loss value is calculated based on the random noise vector of the synthetic frame image, and the content loss value is calculated based on the spatial metric distance of the real frame image; in the process of training the discriminator with the second sample data, the parameters of the discriminator are adjusted based on the loss value corresponding to the real frame image, the loss value corresponding to the synthetic frame image, and the feature matching loss; the training goal of the discriminator is to reduce the error degree of the discriminant probability information output by the discriminator; the feature matching loss is determined based on the intermediate layer feature representation of the discriminator.

[0116] In the specific implementation, through adversarial training, the goal of the generator G is to maximize the probability that the discriminator D is deceived, and the goal of the discriminator D is to minimize the error rate of classifying true and false images.

[0117] For example, for training the generator, content loss can be used:

[0118]

[0119] Among them, x is the real image (that is, the real frame image), is the Euclidean distance (i.e., spatial metric distance).

[0120] And using Adversarial Loss:

[0121]

[0122] Where G(z) is the fake image generated by the generator G and z is a random noise vector.

[0123] The generator G can be trained as follows:

[0124] 1. Fixed discriminator D;

[0125] 2. Sample a batch of data from random noise z to generate a batch of fake images (i.e. fake sample images);

[0126] 3. Calculate the adversarial loss L of the generator G adv (i.e., adversarial loss value) and content loss L content (i.e., content loss value);

[0127] 4. Update the parameters of generator G to minimize the parameters of generator G, L G = L content +λ adv L adv , where λ adv is the weight against the loss.

[0128] For example, for the training of the discriminator, the real / fake image loss can be used:

[0129]

[0130]

[0131] in, express From real data (i.e. the first mouth area image cropped from the real frame image), Represents the image from the generated data (i.e., the second mouth region image cropped from the synthetic frame image).

[0132] And Feature Matching Loss:

[0133]

[0134] in, It is the intermediate layer feature representation of the discriminator D.

[0135] The discriminator D can be trained as follows:

[0136] 1. Fixed generator G;

[0137] 2. Sample a batch of real images 𝑥 from real data;

[0138] 3. Sample a batch of fake images from the generator G ;

[0139] 4. Calculate the loss L of the discriminator D on real images and fake images real and L fake .

[0140] 5. Update the parameters of the discriminator D to minimize, L total-D =L real +L fake +λ feat L feat , where λ fake is the weight of the feature matching loss.

[0141] In this embodiment, by adjusting the parameters of the generator based on the adversarial loss value and the content loss value in the process of training the generator using the first sample data, and adjusting the parameters of the discriminator based on the loss value corresponding to the real frame image, the loss value corresponding to the synthetic frame image, and the feature matching loss in the process of training the discriminator using the second sample data, the clarity of the optimized tooth details in the generated network can be improved based on the adversarial training of the generator and the discriminator.

[0142] In an exemplary embodiment, as shown in FIG5 , a method for generating a lip shape image is provided, and the method is applied to a server as an example for explanation, and includes the following steps S501 to S502. Among them:

[0143] In step S501, based on the current processing frame in the original video, the audio to be processed and the face image to be processed are obtained;

[0144] Among them, the audio to be processed can be obtained based on the video voice corresponding to the current processing frame in the original video, and the facial image to be processed can include a first facial image and a second facial image. The first facial image can be the facial image in the current processing frame, and the second facial image can be the facial image obtained after blocking the mouth area of ​​the facial image in the current processing frame.

[0145] In step S502, the audio to be processed and the facial image to be processed are input into the trained lip shape image generation model to obtain a target facial image with lip shape; the target facial image is used to replace the facial image in the current processing frame in the original video to generate a target video.

[0146] The trained lip-image generation model may be obtained by training using any of the lip-image generation model training methods.

[0147] In practical applications, the mel spectrum features can be extracted from the input speech (i.e., the audio to be processed) to obtain speech features, and features can be extracted from two groups of face images (i.e., the face images to be processed) with the full face and the lower half of the face removed. The extracted features can then be input into the trained wav2lip network (i.e., the trained lip shape image generation model) to generate a face image with lip shape (i.e., the target face image).

[0148] In one example, the generated facial image can be pasted back to the corresponding position of the original video (i.e., replacing the facial image corresponding to the current processing frame), and a new video can be generated frame by frame. Then, the generated video and the audio to be processed can be synthesized using an audio and video stream conversion tool to obtain a video with lip movements (i.e., the target video).

[0149] In the above-mentioned lip image generation method, the audio to be processed and the facial image to be processed are obtained based on the current processing frame in the original video, and then the audio to be processed and the facial image to be processed are input into the trained lip image generation model to obtain a target facial image with lip shape. The target facial image is used to replace the facial image in the current processing frame in the original video to generate a target video, which can optimize the clarity of dental details and effectively improve the image quality of the mouth image, especially the dental area.

[0150] In one embodiment, as shown in FIG6 , a flow chart of another method for training a lip image generation model is provided. In this embodiment, the method comprises the following steps:

[0151] In step S601, the images in the sample video are sampled in time series to obtain a plurality of real frame images, and the audio in the sample video is sampled in time series to obtain sample audio corresponding to the real frame image. For any real frame image, the sample video is randomly sampled to obtain a frame image different from any real frame image as a fake sample image. In step S602, the first sample data for training the generator in the lip-shaped image generation model is obtained, and the sample audio is processed by the audio encoder in the generator to obtain an audio feature vector. In step S603, the audio feature vector and the image feature vector of the sample face image are fused by the facial encoder in the generator to obtain an encoding output result, and the encoding output result is decoded and output by the facial decoder in the generator to obtain a synthetic frame image. In step S604, the first mouth area image cut out from the real frame image and the second mouth area image cut out from the synthetic frame image are used as the second sample data for training the discriminator in the lip-shaped image generation model. In step S605, in the process of using the first sample data to train the generator, the parameters of the generator are adjusted based on the adversarial loss value and the content loss value. In step S606, in the process of training the discriminator using the second sample data, the parameters of the discriminator are adjusted based on the loss value corresponding to the real frame image, the loss value corresponding to the synthetic frame image, and the feature matching loss until the training end condition is met, and a trained lip image generation model is obtained. It should be noted that the specific definition of the above steps can be referred to the specific definition of a training method for a lip image generation model above, which will not be repeated here.

[0152] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0153] Based on the same inventive concept, the embodiment of the present application also provides a lip image generation model training device for implementing the lip image generation model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more lip image generation models provided below can refer to the limitations of the lip image generation model training method above, and will not be repeated here.

[0154] In one embodiment, as shown in FIG7 , a training device for a lip shape image generation model is provided, comprising:

[0155] A first sample data acquisition module 701 is used to acquire first sample data for training the generator in the lip image generation model; the first sample data includes a plurality of sample pairs, each of which includes a sample face image and a sample audio of a time interval corresponding to the sample face image; the sample face image includes a real frame image with a mouth area blocked and a fake sample image with a complete face, the real frame image is a frame image sampled from a certain frame in a sample video, and the fake sample image is a frame image sampled from a frame other than the certain frame in the sample video;

[0156] A synthetic frame image obtaining module 702 is used to input the sample face image and the sample audio into the generator to obtain a synthetic frame image containing a complete face;

[0157] A second sample data acquisition module 703 is used to use the first mouth area image cropped from the real frame image and the second mouth area image cropped from the synthetic frame image as second sample data for training the discriminator in the lip shape image generation model;

[0158] The adversarial training module 704 is used to use the first sample data and the second sample data to perform adversarial training on the generator and the discriminator until the training end condition is met, thereby obtaining a trained lip image generation model; the lip image generation model is used to generate a facial image with lip shape according to the input speech conversion.

[0159] In one embodiment, the apparatus further comprises:

[0160] A sample acquisition module, configured to perform time-sequential sampling on the images in the sample video to obtain a plurality of the real frame images, and perform time-sequential sampling on the audio in the sample video to obtain sample audio corresponding to the real frame image; the time interval of the sample audio matches the time interval of the real frame image;

[0161] The fake sample image obtaining module is used for obtaining a frame image different from any real frame image as the fake sample image by randomly sampling the sample video.

[0162] In one embodiment, the synthetic frame image obtaining module 702 includes:

[0163] An audio encoding submodule, configured to process the sample audio through an audio encoder in the generator to obtain an audio feature vector; wherein the audio encoder has a one-dimensional convolutional layer and a neural network based on a self-attention mechanism;

[0164] A generator fusion submodule, configured to fuse the audio feature vector and the image feature vector of the sample face image through a facial encoder in the generator to obtain a coded output result;

[0165] The decoding output submodule is used to decode and output the encoded output result through the face decoder in the generator to obtain the synthesized frame image.

[0166] In one embodiment, the facial encoder includes a plurality of convolutional layers connected front and back, and the generator fusion submodule includes:

[0167] an intermediate feature vector obtaining unit, configured to perform convolution processing on the sample face image through an intermediate convolution layer among the plurality of convolution layers of the face encoder to obtain an intermediate feature vector;

[0168] A feature vector fusion unit, used to fuse the audio feature vector and the intermediate feature vector based on a cross attention mechanism to obtain a fused feature vector output by the intermediate convolutional layer;

[0169] The encoding output result obtaining unit is used to obtain the encoding output result according to the image feature vectors output by each convolution layer except the middle convolution layer in the facial encoder and the fusion feature vector output by the middle convolution layer.

[0170] In one embodiment, the apparatus further comprises:

[0171] An image input module, used for inputting the first mouth area image and the second mouth area image as input images to the discriminator;

[0172] A mouth feature extraction module, used for extracting features of the input image through each convolution layer in the discriminator to obtain the mouth features of the input image;

[0173] A discriminator output module is used to obtain the discriminant probability information output by the discriminator according to the mouth features of the input image; the discriminant probability information is used to characterize the probability that the first mouth area image input belongs to the real frame image and the probability that the second mouth area image belongs to the real frame image.

[0174] In one embodiment, the adversarial training module 704 includes:

[0175] A generator training submodule, used for adjusting the parameters of the generator based on the adversarial loss value and the content loss value in the process of training the generator with the first sample data; the training goal of the generator is to enhance the realism of the synthetic frame image output by the generator; the adversarial loss value is calculated based on the random noise vector of the synthetic frame image, and the content loss value is calculated based on the spatial metric distance of the real frame image;

[0176] The discriminator training submodule is used to adjust the parameters of the discriminator based on the loss value corresponding to the real frame image, the loss value corresponding to the synthetic frame image, and the feature matching loss during the process of training the discriminator using the second sample data; the training goal of the discriminator is to reduce the error degree of the discriminant probability information output by the discriminator; the feature matching loss is determined based on the intermediate layer feature representation of the discriminator.

[0177] Each module in the training device for the lip image generation model can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0178] Based on the same inventive concept, the embodiment of the present application also provides a lip image generation device for implementing the lip image generation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more lip image generation device embodiments provided below can refer to the limitations of the lip image generation method above, and will not be repeated here.

[0179] In one embodiment, as shown in FIG8 , a lip shape image generating device is provided, comprising:

[0180] The current processing frame data acquisition module 801 is used to acquire the audio to be processed and the facial image to be processed based on the current processing frame in the original video; the audio to be processed is obtained based on the video voice corresponding to the current processing frame in the original video; the facial image to be processed includes a first facial image and a second facial image, the first facial image is the facial image in the current processing frame, and the second facial image is the facial image obtained by blocking the mouth area of ​​the facial image in the current processing frame;

[0181] The target face image obtaining module 802 is used to input the audio to be processed and the face image to be processed into a trained lip shape image generation model to obtain a target face image with a lip shape that matches the audio to be processed; the target face image is used to replace the face image in the current processing frame in the original video to generate a target video;

[0182] Wherein, the trained lip-image generation model is obtained by training using any one of the lip-image generation model training methods.

[0183] Each module in the above-mentioned lip image generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0184] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a training method for a lip image generation model is implemented.

[0185] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store lip image generation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a lip image generation method is implemented.

[0186] Those skilled in the art will understand that the structures shown in FIGS. 9 and 10 are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figures, or combine certain components, or have a different arrangement of components.

[0187] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0188] In an exemplary embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0189] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0190] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0191] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0192] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training a lip image generation model, characterized in that: The method comprises: Acquire first sample data for training a generator in the lip-image generation model; the first sample data includes a plurality of sample pairs, each of which includes a sample face image and a sample audio of a time interval corresponding to the sample face image; the sample face image includes a real frame image with a mouth area blocked and a fake sample image with a complete face, the real frame image is a frame image obtained by sampling a certain frame in a sample video, and the fake sample image is a frame image obtained by sampling frames in the sample video other than the certain frame; Inputting the sample face image and the sample audio into the generator to obtain a synthetic frame image containing a complete face; Using a first mouth region image cropped from the real frame image and a second mouth region image cropped from the synthetic frame image as second sample data for training a discriminator in the lip shape image generation model; The first sample data and the second sample data are used to perform adversarial training on the generator and the discriminator until the training end condition is met, thereby obtaining a trained lip image generation model; the lip image generation model is used to generate a facial image with lip shape according to input speech conversion.

2. The method according to claim 1, characterized in that The method further comprises: Performing time-sequential sampling on the images in the sample video to obtain a plurality of the real frame images, and performing time-sequential sampling on the audio in the sample video to obtain sample audio corresponding to the real frame image; the time interval of the sample audio matches the time interval of the real frame image; For any real frame image, by randomly sampling the sample video, a frame image different from the real frame image is obtained as the false sample image.

3. The method according to claim 1, characterized in that The step of inputting the sample face image and the sample audio into the generator to obtain a synthetic frame image containing a complete face includes: The sample audio is processed by an audio encoder in the generator to obtain an audio feature vector; wherein the audio encoder has a one-dimensional convolutional layer and a neural network based on a self-attention mechanism; The facial encoder in the generator fuses the audio feature vector and the image feature vector of the sample face image to obtain a coding output result; The encoded output result is decoded and outputted by a facial decoder in the generator to obtain the synthesized frame image.

4. The method according to claim 3, characterized in that The facial encoder includes a plurality of convolutional layers connected front and back; the encoding output result is obtained by fusing the audio feature vector and the image feature vector of the sample face image through the facial encoder in the generator, including: Performing convolution processing on the sample face image through an intermediate convolution layer among the multiple convolution layers of the face encoder to obtain an intermediate feature vector; Fusing the audio feature vector and the intermediate feature vector based on a cross attention mechanism to obtain a fused feature vector output by the intermediate convolutional layer; The encoding output result is obtained according to the image feature vectors output by each convolution layer except the middle convolution layer in the facial encoder and the fusion feature vector output by the middle convolution layer.

5. The method according to claim 1, characterized in that The steps of training the discriminator include: inputting the first mouth region image and the second mouth region image as input images to the discriminator; Extracting features of the input image through each convolutional layer in the discriminator to obtain mouth features of the input image; According to the mouth features of the input image, the discriminant probability information output by the discriminator is obtained; the discriminant probability information is used to characterize the probability that the first mouth area image input belongs to the real frame image and the probability that the second mouth area image belongs to the real frame image.

6. The method according to claim 1, characterized in that The using the first sample data and the second sample data to perform adversarial training on the generator and the discriminator includes: In the process of training the generator using the first sample data, the parameters of the generator are adjusted based on the adversarial loss value and the content loss value; the training goal of the generator is to enhance the authenticity of the synthetic frame image output by the generator; the adversarial loss value is calculated based on the random noise vector of the synthetic frame image, and the content loss value is calculated based on the spatial metric distance of the real frame image; In the process of training the discriminator using the second sample data, the parameters of the discriminator are adjusted based on the loss value corresponding to the real frame image, the loss value corresponding to the synthetic frame image, and the feature matching loss; the training goal of the discriminator is to reduce the error degree of the discriminant probability information output by the discriminator; the feature matching loss is determined based on the intermediate layer feature representation of the discriminator.

7. A method for generating a lip shape image, characterized in that: The method comprises: Based on the current processing frame in the original video, obtain the audio to be processed and the facial image to be processed; the audio to be processed is obtained based on the video voice corresponding to the current processing frame in the original video; the facial image to be processed includes a first facial image and a second facial image, the first facial image is the facial image in the current processing frame, and the second facial image is the facial image obtained after the mouth area of ​​the facial image in the current processing frame is blocked; Inputting the audio to be processed and the facial image to be processed into a trained lip-image generation model to obtain a target facial image with a lip shape that matches the audio to be processed; the target facial image is used to replace the facial image in the current processing frame in the original video to generate a target video; Wherein, the trained lip-image generation model is obtained by training the lip-image generation model training method according to any one of claims 1-6.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images