Speaking face video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a speaking face video generation method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a target audio and a target face image, extracting audio coding features of the target audio through a target audio encoder, predicting an object head posture sequence based on the audio coding features through a target head posture predictor, and performing feature fusion on the audio coding features and the object head posture sequence to obtain a target head posture image; carrying out forward diffusion processing on the target face image to obtain a noise action sequence, carrying out reverse diffusion processing according to the target fusion feature and the noise action sequence through a target action generation model to obtain a target action sequence, and extracting a target appearance hidden variable of the target face image through a target image encoder; the target speaking face video is generated through the target decoder based on the target action sequence and the target appearance hidden variable, and the quality of speaking face video generation can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device, and storage medium for generating a video of a speaking face. Background Art

[0002] Talking face video generation technology is a multimodal deep learning technique that generates synchronized talking face videos based on audio signals. It can be applied to fields such as virtual customer service and virtual hosts. Related technologies use a 3D Morphable Model (3DMM) for talking face video generation. A 3D morphable model is a differentiable, parameterized 3D model that represents facial shape and appearance through a set of adjustable parameters. This results in a lack of naturalness and diversity in the generated talking face videos, leading to lower quality. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for generating a speaking face video, aiming to improve the quality of speaking face video generation.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a method for generating a speaking face video, the method comprising:

[0005] Obtain target audio and target face image;

[0006] Extracting audio coding features of the target audio through a target audio encoder, predicting a head posture sequence of the object based on the audio coding features through a target head posture predictor, and performing feature fusion on the audio coding features and the head posture sequence of the object to obtain a target fusion feature;

[0007] Performing forward diffusion processing on the target face image to obtain a noise action sequence;

[0008] Performing reverse diffusion processing on the target fusion features and the noise action sequence through a target action generation model to obtain a target action sequence;

[0009] Extracting a target appearance latent variable of the target face image through a target image encoder;

[0010] A target speaking face video is generated by a target decoder based on the target action sequence and the target appearance latent variable.

[0011] In some embodiments, the target image encoder and the target decoder are jointly trained according to the following steps:

[0012] Obtain a first sample speaking face video;

[0013] Performing facial key point detection on the first sample speaking face video to obtain a facial key point sequence, and extracting sample action latent variables of the facial key point sequence;

[0014] Randomly selecting a candidate face image from the first sample speaking face video, and extracting a sample appearance latent variable of the candidate face image through an original image encoder; wherein the characteristic scale of the sample appearance latent variable is greater than the characteristic scale of the sample action latent variable;

[0015] Generate a predicted speaking face video based on the sample action latent variables and the sample appearance latent variables through an original decoder;

[0016] Calculating a first target loss based on the sample action latent variable, the sample appearance latent variable, the first sample speaking face video, and the predicted speaking face video;

[0017] The model parameters of the original image encoder are adjusted according to the first target loss to obtain the target image encoder, and the model parameters of the original decoder are adjusted according to the first target loss to obtain the target decoder.

[0018] In some embodiments, calculating the first target loss based on the sample action latent variable, the sample appearance latent variable, the first sample speaking face video, and the predicted speaking face video includes:

[0019] Performing loss calculation based on the sample action latent variable, the sample appearance latent variable, and a preset standard normal distribution to obtain a first sub-loss;

[0020] Performing loss calculation based on the first sample speaking face video and the predicted speaking face video to obtain a second sub-loss;

[0021] The first sub-loss and the second sub-loss are summed to obtain the first target loss.

[0022] In some embodiments, the first sample speaking face video includes multiple sample face images, the predicted speaking face video includes multiple predicted face images, and the loss calculation based on the first sample speaking face video and the predicted speaking face video to obtain the second sub-loss includes:

[0023] Calculating a pixel difference value between each sample face image and the corresponding predicted face image;

[0024] Performing image discrimination on each of the sample face images by a discriminator to obtain a first probability that the sample face image is a real image;

[0025] Performing image discrimination on each of the predicted facial images by the discriminator to obtain a second probability that the predicted facial image is a composite image;

[0026] The second sub-loss is determined according to the pixel difference value, the first probability, and the second probability.

[0027] In some embodiments, the target audio encoder, the target head pose predictor, and the target motion generation model are jointly trained according to the following steps:

[0028] Obtaining a second sample speaking face video, and extracting sample audio, reference head posture, and reference action latent variables from the second sample speaking face video;

[0029] Extracting sample audio coding features of the sample audio through an original audio encoder, predicting a sample head posture based on the sample audio coding features through an original head posture predictor, and performing feature fusion on the sample audio coding features and the reference head posture to obtain a sample fusion feature;

[0030] Performing forward diffusion processing on the reference action latent variable to obtain a noisy action latent variable;

[0031] Performing reverse diffusion processing based on the sample fusion features and the noise-added action latent variables through a diffusion model to obtain a predicted action latent variable;

[0032] Calculating a second target loss based on the reference head posture, the sample head posture, the reference action latent variable, and the predicted action latent variable;

[0033] Adjust the model parameters of the original audio encoder according to the second target loss to obtain the target audio encoder, adjust the model parameters of the original head posture predictor according to the second target loss to obtain the target head posture predictor, and adjust the model parameters of the diffusion model according to the second target loss to obtain the target action generation model.

[0034] In some embodiments, the diffusion model includes multiple conformal network layers, and performing reverse diffusion processing based on the sample fusion features and the noisy action latent variables through the diffusion model to obtain the predicted action latent variables includes:

[0035] Performing back diffusion processing according to the sample fusion feature and the noise-added action latent variable through the first conformal network layer to obtain the intermediate action latent variable output by the first conformal network layer;

[0036] Performing back diffusion processing by the current conformal network layer according to the sample fusion feature and the intermediate action latent variable output by the previous conformal network layer to obtain the intermediate action latent variable output by the current conformal network layer;

[0037] Until the intermediate action latent variable output by the last conformal network layer is obtained, the predicted action latent variable is obtained.

[0038] In some embodiments, calculating the second target loss according to the reference head posture, the sample head posture, the reference action latent variable, and the predicted action latent variable includes:

[0039] Performing loss calculation based on the reference head posture and the sample head posture to obtain a third sub-loss;

[0040] Performing loss calculation based on the reference action latent variable and the predicted action latent variable to obtain a fourth sub-loss;

[0041] The third sub-loss and the fourth sub-loss are summed to obtain the second target loss.

[0042] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a device for generating a video of a speaking face, the device comprising:

[0043] An acquisition module is used to acquire target audio and target face image;

[0044] a first encoding module, configured to extract audio coding features of the target audio through a target audio encoder, predict a subject head posture sequence based on the audio coding features through a target head posture predictor, and perform feature fusion on the audio coding features and the subject head posture sequence to obtain a target fusion feature;

[0045] A forward diffusion module, configured to perform forward diffusion processing on the target face image to obtain a noise action sequence;

[0046] An action generation module is configured to perform reverse diffusion processing based on the target fusion feature and the noise action sequence through a target action generation model to obtain a target action sequence;

[0047] A second encoding module is used to extract the target appearance latent variable of the target face image through a target image encoder;

[0048] The video generation module is used to generate a target speaking face video based on the target action sequence and the target appearance latent variable through a target decoder.

[0049] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method for generating a speaking face video described in the first aspect above.

[0050] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method for generating a speaking face video described in the first aspect above.

[0051] The present invention proposes a method, device, electronic device, and computer-readable storage medium for generating a speaking face video. These methods obtain target audio and a target facial image to generate a speaking face video based on the target audio and the target facial image. A target audio encoder extracts audio coding features from the target audio. A target head pose predictor predicts the subject's head pose sequence based on the audio coding features. This ensures that the speaking face generated based on the target audio matches the actual head movements during speaking, enhancing the naturalness of the speaking face video. Feature fusion is performed on the audio coding features and the subject's head pose sequence to generate target fused features. This target fused features are then used to assist in generating more natural, realistic, and diverse speaking face videos. Forward diffusion is performed on the target facial image to generate a noise motion sequence, which increases the diversity of the generated speaking face based on the noise motion sequence. To more accurately model the speaking face, a target motion generation model performs back diffusion on the target fused features and the noise motion sequence to generate a target motion sequence. The target motion generation model accurately captures the temporal characteristics of facial motions, generating a more natural and coherent facial motion sequence. To create more realistic and diverse talking faces, a target image encoder extracts the target appearance latent variables of the target face image. A target decoder generates a target talking face video based on the target action sequence and the target appearance latent variables. This accurately models the talking face based on facial movements and appearance, improving the quality of the generated talking face video. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flowchart of the method for generating a speaking face video provided by an embodiment of the present application;

[0053] Figure 2 is a flowchart of the training process of the target image encoder and target decoder provided in an embodiment of the present application;

[0054] Figure 3 yes Figure 2 Flowchart of step S250 in FIG.

[0055] Figure 4 yes Figure 3 Flowchart of step S320 in FIG.

[0056] Figure 5 This is a flowchart of the training process of the target audio encoder, target head posture predictor, and target action generation model provided by the embodiment of the present application;

[0057] Figure 6 yes Figure 5 Flowchart of step S540 in FIG.

[0058] Figure 7 yes Figure 5 Flowchart of step S550 in FIG.

[0059] Figure 8 Schematic diagram of the structure of the device for generating a speaking face video provided in an embodiment of the present application;

[0060] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0062] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0064] With the development of deep learning technology and the increase in computing power of hardware devices used to support deep learning model training and inference, artificial intelligence has made significant progress in the field of multimodal generation. The human face, which contains information such as emotion, movement, and identity, is a highly information-intensive medium and has become a research focus in the field of speaking face video generation. Speaking face video generation technology combines audio signals with facial images to generate speaking videos that match the audio signals. This technology has applications in digital characters, online education, film production, and other fields. Related technologies use three-dimensional deformable models for speaking face video generation. Three-dimensional deformable models are differentiable parameterized three-dimensional models that represent facial shape and appearance through a set of adjustable parameters to assist in modeling the three-dimensional face. This results in a lack of naturalness and diversity in the generated speaking face videos, resulting in low quality.

[0065] Based on this, the embodiments of the present application provide a method for generating a speaking face video, a speaking face video generating device, an electronic device, and a computer-readable storage medium, aiming to improve the quality of speaking face video generation.

[0066] The speaking face video generation method, speaking face video generation device, electronic device and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the speaking face video generation method in the embodiments of the present application is described.

[0067] The method for generating a video of a speaking face provided in the embodiment of the present application relates to the field of artificial intelligence technology. The method for generating a video of a speaking face provided in the embodiment of the present application can be applied in a terminal, can be applied in a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for generating a speaking face video, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0069] Figure 1 This is an optional flowchart of the method for generating a speaking face video provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S160.

[0070] Step S110, acquiring target audio and target face image;

[0071] Step S120, extracting audio coding features of the target audio using the target audio encoder, predicting the subject's head posture sequence based on the audio coding features using the target head posture predictor, and performing feature fusion on the audio coding features and the subject's head posture sequence to obtain a target fused feature;

[0072] Step S130, performing forward diffusion processing on the target face image to obtain a noise action sequence;

[0073] Step S140, performing back diffusion processing on the target fusion features and the noise action sequence through the target action generation model to obtain the target action sequence;

[0074] Step S150, extracting target appearance latent variables of the target face image through a target image encoder;

[0075] Step S160 , generating a target speaking face video based on the target action sequence and the target appearance latent variables through a target decoder.

[0076] In step S110 of some embodiments, target audio is captured using an audio capture device such as a microphone, or extracted from an existing video using an audio extraction tool. The target audio is the audio signal used to generate a video of the speaking face. It contains the speaker's speech content and can reflect the speaker's voice rhythm and emotional state. A target facial image is captured using a camera, or extracted from an existing image file. The target facial image is the facial image used to generate the speaking face video. It contains the speaker's facial features, such as the shape and position of the eyes, nose, and mouth.

[0077] In step S120 of some embodiments, when a speaker speaks, he or she may nod, shake his or her head, or slightly turn his or her head to assist in expressing emotions. These head movements are related to the rhythm, intonation, emotion, and other features of the audio signal. In order to generate a speaker video with natural head movements, a target audio encoder is used to extract audio features from the target audio to obtain audio coding features. The target audio encoder may adopt a conformal model, and the audio coding features may be features such as the rhythm, intonation, and emotion of the target audio. There is a close relationship between head movements and audio features. In order to improve the quality and diversity of video generation, a target head posture predictor is introduced between the target audio encoder and the backbone network. The target head posture predictor may adopt a convolutional neural network. The convolutional neural network includes multiple convolutional layers. The number of convolutional layers can be set according to actual conditions, such as the number of convolutional layers is 2. The target head posture predictor performs head posture prediction based on the audio coding features to obtain an object head posture sequence, so as to accurately express the speaker's head movements based on the object head posture sequence. The subject head posture sequence is a sequence of head postures that match the target audio. The head posture can be represented by three dimensions: pitch angle, yaw angle, and roll angle.

[0078] The subject's head posture sequence can be expressed as x p ∈R N×3 , N is the number of video frames of the speaking face video to be generated, and R is the real number space.

[0079] This embodiment of the application uses single-image talking face generation technology to generate a talking face video from a single face image and audio signal. To ensure that the speaker's mouth shape and audio signal are synchronized in time, and that the head posture and facial expression are coherent and natural, the audio encoding features and the subject's head posture sequence are summed to obtain the target fusion features.

[0080] In step S130 of some embodiments, the number of video frames of the speaking face video to be generated is obtained. At each diffusion time step in the forward diffusion process, multiple noise vectors equal to the number of video frames are sampled from a standard Gaussian distribution. The corresponding noise vector is added to each image of the previous diffusion time step to obtain the image of the current diffusion time step until the diffusion time step reaches the time step limit T. The image sequence obtained at the diffusion time step T is used as the noise action sequence to increase the diversity of the speaking face video generation based on the noise distribution and noise intensity reflected by the noise action sequence. The forward diffusion process can be expressed as:

[0081]

[0082] in, is the mth image at diffusion time step t; β t is the noise variance adjustment factor of the diffusion time step t, which is a noise scheduling parameter with a non-negative value; ∈ t is the noise vector of diffusion time step t; the target face image is represented as x0.

[0083] The forward diffusion process can be expressed by the following stochastic differential equation:

[0084]

[0085] Where dt is the small increment of the diffusion time step; express Changes within a small time interval dt; β t is the noise scheduling parameter, which is a non-negative function of the diffusion time step t; w t represents standard Brownian motion; dw t is the increment of the standard Brownian motion.

[0086] In step S140 of some embodiments, the reverse diffusion process can be expressed by an ordinary differential equation:

[0087]

[0088] in, represents the gradient operator; p t For models based on The learned probability density function; log represents the logarithmic operation.

[0089] The target action generation model uses a diffusion model based on the Conformer network architecture as its foundational model and incorporates a diffusion generation approach to model action sequence generation. This approach effectively captures the temporal characteristics of actions and generates more natural and coherent action sequences. The diffusion model models the logarithmic gradient of the probability density function based on the noise action sequence. This logarithmic gradient guides the back-diffusion process to produce the target action sequence. The target action sequence contains object movements that match the target audio, such as lip movements and head movements. The target action generation model consists of multiple cascaded conformal network layers, each employing a Conformer network. A multilayer perceptron is used to encode the diffusion time step T to produce a temporal feature. This temporal feature is concatenated with the target fusion feature and the noise action sequence and fed into the first conformal network layer of the target action generation model for back-diffusion, resulting in the intermediate action sequence output by the first conformal network layer. Starting from the second conformal network layer, a multilayer perceptron is used to encode the current diffusion time step to obtain the current time feature. The current time feature, the target fusion feature, and the intermediate action sequence output by the previous conformal network layer are concatenated and input into the current conformal network layer of the target action generation model for back diffusion, resulting in the intermediate action sequence output by the current conformal network layer. The current diffusion time step is calculated by subtracting 1 from the previous diffusion time step. This step is repeated until the intermediate action sequence output by the last conformal network layer is obtained, and this intermediate action sequence is used as the target action sequence.

[0090] In step S150 of some embodiments, the speaker video generation method of the present invention employs a two-stage generation strategy, dividing the speaker face generation task into audio-based action generation and action-based video generation. The first stage uses the Conformer model as the base model to generate action sequences. The second stage uses a variational autoencoder (VAE) to model the generation of the speaker face video based on the action sequence. This two-stage generation framework generates action latent variable sequences in the first stage rather than directly generating facial key point sequences. This produces more stable and realistic generation results, avoiding the local jitter problem inherent in key point sequences. The variational autoencoder includes a target image encoder and a target decoder. To ensure that the speaker identity features of the generated video are consistent with the facial identity features of the target facial image and that the generated video's visual content is consistent with the target facial image, the target image encoder extracts target appearance latent variables of the target facial image to avoid loss of facial details and blurring of the video content. The target appearance latent variables indicate facial appearance features of the target facial image, such as face shape, facial features, skin features, and hair features.

[0091] In step S160 of some embodiments, the target action sequence and the target appearance latent variables are spliced ​​and input into the target decoder for video generation to obtain the target speaking face video, so that the speaking face video achieves a balance between lip synchronization, natural movements, consistent identity features and visual fidelity, solves problems such as blurred visual content, incoherence between frames or loss of facial details of the character, and enhances the realism of the video.

[0092] See also Figure 2 In some embodiments, the training process of the target image encoder and the target decoder may include but is not limited to steps S210 to S260:

[0093] Step S210, obtaining a first sample speaking face video;

[0094] Step S220: performing facial key point detection on the first sample speaking face video to obtain a facial key point sequence, and extracting sample action latent variables of the facial key point sequence;

[0095] Step S230: randomly selecting a candidate face image from the first sample speaking face video, and extracting a sample appearance latent variable of the candidate face image through an original image encoder; wherein the characteristic scale of the sample appearance latent variable is greater than the characteristic scale of the sample action latent variable;

[0096] Step S240, generating a predicted speaking face video based on the sample action latent variables and the sample appearance latent variables through the original decoder;

[0097] Step S250, calculating a first target loss based on the sample action latent variable, the sample appearance latent variable, the first sample speaking face video, and the predicted speaking face video;

[0098] Step S260: Adjust the model parameters of the original image encoder according to the first target loss to obtain a target image encoder, and adjust the model parameters of the original decoder according to the first target loss to obtain a target decoder.

[0099] In step S210 of some embodiments, to improve the model's performance in generating speaking face videos, a joint training approach is employed to train the target image encoder and target decoder. Specifically, a first sample speaking face video is obtained. The first sample speaking face video is a sample video used to train the target image encoder and target decoder. The first sample speaking face video includes a plurality of sample face images arranged in chronological order, each sample face image containing a speaking face.

[0100] In step S220 of some embodiments, facial key point detection is performed on each sample face image using a pre-trained facial key point extractor to obtain a facial key point image for each sample face image. The facial key point image is an image composed of key feature points used to describe the shape and feature positions of the face. The facial key point image of each sample face image is used as a facial key point sequence. To better decouple action and appearance information, the embodiments of the present application use a dense facial key point sequence to represent facial action information in the video. However, the facial key point sequence and the sample face image are not completely decoupled action and appearance information. Therefore, the embodiments of the present application use a variational autoencoder comprising two encoders and a decoder to obtain completely decoupled action and appearance. The untrained variational autoencoder includes an action encoder, a raw image encoder, and a raw decoder. The action encoder is used to encode the action information of the facial key point sequence, the raw image encoder is used to encode the appearance information of the face image, and the raw decoder is used to generate a speaking face video based on the encoded action features and appearance features. The facial action features of each facial key point image in the facial key point sequence are extracted through the action encoder to obtain the facial action of each facial key point image, and these facial actions are used as sample action latent variables. The sample action latent variables are a sequence of facial actions of each facial key point image.

[0101] In step S230 of some embodiments, a sample face image is randomly selected from the first sample speaking face video as a candidate face image, and appearance features are extracted from the candidate face image using an original image encoder to obtain a sample appearance latent variable.

[0102] Use x∈R N×H×W×3 represents a video of a talking face, R represents the real number space, N is the number of face images in the talking face video, H×W×3 is the size of the face image, and the sequence of facial key points extracted from the video is x m ∈R N ×H×W×3 , the jth face key point image x in the face key point sequence m (j) as the action encoder E m The input of , the face action of the j-th face key point image is obtained, which is expressed as:

[0103] z m (j) = E m (x m (j)),

[0104] Among them, z m (j) is the facial motion of the jth facial key point image.

[0105] The randomly selected i-th face image x(i) is used as the appearance encoder Ea The input of , we get the appearance latent variable, expressed as:

[0106] z a (i) = E a (x(i)),

[0107] Among them, z a (i) is the appearance latent variable.

[0108] R represents the real number space, h m ×w m ×3 is the characteristic scale of facial motion, h a ×w a ×3 is the characteristic scale of the appearance latent variable. It is worth noting that z m (j) contains less information than z a (i) To facilitate the encoding and decoding of information, set z m The characteristic scale is smaller than z a The characteristic scale of the sample appearance latent variable is greater than the characteristic scale of the sample action latent variable.

[0109] In step S240 of some embodiments, the sample action latent variables and the sample appearance latent variables are mapped to the same dimension and concatenated using a linear layer. The concatenated features are used as input to the original decoder to generate a predicted speaking face video. The predicted speaking face video includes multiple predicted facial images, where the number of predicted facial images is equal to the number of sample facial images in the first sample speaking face video. The variational autoencoder reconstructs the current video frame based on the facial key point image of the current frame and a random video frame. During training, the variational autoencoder can use the facial key point image of the jth frame and the video frame of the ith frame to generate the jth frame of the predicted speaking face video.

[0110] In some embodiments, in step S250, to improve the quality of generated speaking face videos, the variational autoencoder needs to be optimized. To guide the update of the variational autoencoder's model parameters, a loss is calculated based on the sample action latent variables, the sample appearance latent variables, the first sample speaking face video, and the predicted speaking face video to obtain a first target loss.

[0111] In step S260 of some embodiments, the first target loss is minimized, the model parameters of the motion encoder are adjusted, the model parameters of the original image encoder are adjusted to obtain the target image encoder, and the model parameters of the original decoder are adjusted to obtain the target decoder.

[0112] Through the above steps S210 to S260, the variational autoencoder can obtain decoupled facial action sequences and appearances from the video, so as to generate smooth, natural, diverse, controllable, high visual quality and fidelity speaking face videos based on the facial action sequences and appearances.

[0113] See also Figure 3 In some embodiments, step S250 may include but is not limited to steps S310 to S330:

[0114] Step S310, performing loss calculation based on the sample action latent variable, the sample appearance latent variable, and a preset standard normal distribution to obtain a first sub-loss;

[0115] Step S320, performing loss calculation based on the first sample speaking face video and the predicted speaking face video to obtain a second sub-loss;

[0116] Step S330 , summing the first sub-loss and the second sub-loss to obtain a first target loss.

[0117] In step S310 of some embodiments, KL divergence is used to measure the difference between the latent variable and the preset standard normal distribution. Based on the KL divergence calculation formula, loss calculation is performed based on the sample action latent variable, the sample appearance latent variable, and the preset standard normal distribution to obtain a first sub-loss, so that the sample action latent variable and the sample appearance latent variable both conform to the standard normal distribution. The preset standard normal distribution has a mean of 0 and a standard deviation of 1. The first sub-loss is expressed as:

[0118]

[0119] Among them, L kl is the first sub-loss; x is the speaking face video; P is the set of model parameters of the action encoder and the image encoder; i represents the i-th latent variable, which can be a sample action latent variable or a sample appearance latent variable; x(i) is the true probability distribution of the i-th latent variable, which can be a standard normal distribution; P(i) is the predicted probability distribution of the i-th latent variable, which is the probability distribution of the sample action latent variable or the probability distribution of the sample appearance latent variable.

[0120] In step S320 of some embodiments, in order to measure the difference between the generated speaking face video and the real speaking face video, a loss is calculated based on the first sample speaking face video and the predicted speaking face video to obtain a second sub-loss.

[0121] In some embodiments, in step S330, the first sub-loss and the second sub-loss are added to obtain a first target loss. A first weight for the first sub-loss and a second weight for the second sub-loss may also be obtained, where the sum of the first weight and the second weight is 1. The first weight and the second weight may be set based on actual conditions, such as a first weight of 0.6 and a second weight of 0.4. The first weight and the first sub-loss are multiplied, and the second weight and the second sub-loss are multiplied. The multiplication results are then added to obtain the first target loss.

[0122] Through the above steps S310 to S330, a first target loss can be obtained to guide the training process of the variational autoencoder based on the first target loss, and the parameters of the variational autoencoder are updated through backpropagation to obtain a variational autoencoder with better video generation performance, thereby obtaining a high-quality speaking face video.

[0123] See also Figure 4 In some embodiments, step S320 may include but is not limited to steps S410 to S440:

[0124] Step S410, calculating the pixel difference value between each sample face image and the corresponding predicted face image;

[0125] Step S420, performing image discrimination on each sample face image through a discriminator to obtain a first probability that the sample face image is a real image;

[0126] Step S430, performing image discrimination on each predicted face image through a discriminator to obtain a second probability that the predicted face image is a composite image;

[0127] Step S440 : determining a second sub-loss according to the pixel difference value, the first probability, and the second probability.

[0128] In step S410 of some embodiments, for each sample facial image, the sample facial image in frame j of the original video is used as a training target. Based on the L1 loss function, the pixel difference between the sample facial image in frame j and the predicted facial image in frame j of the generated video is calculated to obtain a pixel difference value. It should be noted that the pixel difference between the images is the sum of the absolute values ​​of the pixel differences between each aligned pixel point.

[0129] In step S420 of some embodiments, when the variational autoencoder generates video samples, the video may be blurry or of low quality. To fully capture the complex structure of the video data, the embodiment of the present application uses an adversarial generation method to train the variational autoencoder. During the training process, a discriminator is introduced to determine whether the image is generated by the model, thereby generating more realistic video samples. The discriminator performs image discrimination on each sample facial image to determine the probability that the sample facial image is a real image, thereby obtaining a first probability.

[0130] In step S430 of some embodiments, a discriminator performs image discrimination on each predicted facial image to determine the probability that the predicted facial image is a composite image, thereby obtaining a second probability.

[0131] In step S440 of some embodiments, a mean is calculated for each first probability to obtain a first loss value, a mean is calculated for each second probability to obtain a second loss value, and the first loss value and the second loss value are added together to obtain a generative adversarial loss. A mean is calculated for each pixel difference value to obtain an average pixel difference, and the generative adversarial loss and the average pixel difference are added together to obtain a second sub-loss. The generative adversarial loss is expressed as:

[0132] L dis =E[logD(x)]+E[log(1-D(z))],

[0133] Among them, L dis is the generative adversarial loss; x is the sample face image; D is the discriminator; E is the mean calculation; E[logD(x)] is the first loss value, D(x) is the first probability, and log is the logarithmic calculation; z is the predicted face image; E[log(1-D(z))] is the second loss value; 1-D(z) is the second probability.

[0134] The discriminator model parameters can be adjusted by maximizing the first objective loss to improve the discriminator's ability to distinguish between real images and synthetic images.

[0135] Through the above steps S410 to S440, based on the second sub-loss, the generated speaking face video can be made close to the real video, thereby improving the quality of the generated speaking face video.

[0136] Existing speaking face generation models usually use a 3DMM prior model as a modeling aid. This method uses a differentiable parameterized three-dimensional face model to decouple and parameterize facial features, and controls the three-dimensional face model by adjusting parameters. Although the 3DMM prior model is helpful for modeling speaking face generation tasks, it can also easily lead to a lack of naturalness and diversity in the generated results, and the ultimate performance is limited by the decoupling ability of the 3DMM. To this end, the embodiment of the present application proposes a speaking face generation framework that does not rely on a priori models. The framework can learn the generation model directly from the data, and is expected to generate more natural and diverse speaking face videos. This end-to-end learning method can better capture the complex distribution characteristics of real data and overcome the limitations of the 3DMM model in generating naturalness and diversity. The training process of the generation model is described in detail below.

[0137] See also Figure 5 In some embodiments, the training process of the target audio encoder, the target head pose predictor, and the target action generation model may include, but is not limited to, steps S510 to S560:

[0138] Step S510, obtaining a second sample speaking face video, and extracting sample audio, reference head posture, and reference action latent variables from the second sample speaking face video;

[0139] Step S520: extracting sample audio coding features of the sample audio using the original audio encoder, predicting the sample head posture based on the sample audio coding features using the original head posture predictor, and performing feature fusion on the sample audio coding features and the reference head posture to obtain a sample fusion feature;

[0140] Step S530, performing forward diffusion processing on the reference action latent variable to obtain a noisy action latent variable;

[0141] Step S540, performing reverse diffusion processing based on the sample fusion features and the noise-added action latent variables through the diffusion model to obtain the predicted action latent variables;

[0142] Step S550, calculating a second target loss according to the reference head posture, the sample head posture, the reference action latent variable, and the predicted action latent variable;

[0143] Step S560: Adjust the model parameters of the original audio encoder according to the second target loss to obtain a target audio encoder, adjust the model parameters of the original head posture predictor according to the second target loss to obtain a target head posture predictor, and adjust the model parameters of the diffusion model according to the second target loss to obtain a target action generation model.

[0144] In step S510 of some embodiments, in order to improve model training efficiency and video generation quality, embodiments of the present application use a joint training method to train the target audio encoder, target head pose predictor, and target action generation model. Specifically, a second sample speaking face video is obtained. The second sample speaking face video is a sample video used to train the target audio encoder, target head pose predictor, and target action generation model. The second sample speaking face video can be the same as or different from the first sample speaking face video. The second sample speaking face video also includes multiple sample face images arranged in chronological order, each sample face image containing a speaking face. Audio is extracted from the second sample speaking face video by an audio extractor to obtain sample audio. A head pose extractor extracts a head pose sequence from the second sample speaking face video to obtain a reference head pose. The reference head pose is the real head pose of the speaker. After the variational autoencoder training is completed, a real facial action sequence is extracted from the second sample speaking face video by an action encoder to obtain a reference action latent variable.

[0145] In step S520 of some embodiments, audio features are extracted from the sample audio through the original audio encoder to obtain sample audio coding features. The original audio encoder may adopt a Conformer model, and the sample audio coding features are used to describe and characterize the characteristics of the audio signal, such as emotional features, timbre features, pitch features, etc. The sample audio coding features are input into the original head posture predictor to predict the sample head posture, which is a predicted sequence of speaker head postures. The reference head posture is mapped to the model dimension through a linear layer so that the feature dimension of the reference head posture is the same as the feature dimension of the sample audio coding features. The mapped reference head posture and the sample audio coding features are feature-added to obtain a sample fusion feature.

[0146] In step S530 of some embodiments, referring to step S130, forward diffusion processing is performed on the reference action latent variable to obtain the noise-added action latent variable. Different degrees of noise will be gradually added. As the time step becomes large enough, the noisy action latent variable It can be regarded as pure Gaussian noise, and the pure Gaussian noise is used as the noise action hidden variable.

[0147] In step S540 of some embodiments, the Conformer model is used as the skeleton of the action latent variable generation model, and then the diffusion generation method is combined to model the generation process of the action latent variable. The Conformer model restores the clean action latent variable according to the sample audio coding features, time step, noisy action latent variable and reference action latent variable. The diffusion model is a Conformer model, which includes multiple conformal network layers cascaded in sequence. In the reverse diffusion process, the model will gradually learn to restore the noisy data into clean data samples based on the conditions. Through multiple iterations, the model can generate samples that conform to the original data distribution from Gaussian noise. Specifically, referring to step S140, the sample fusion features, the noisy action latent variables and the time features obtained after time step encoding are spliced ​​together, and the diffusion model is used to perform reverse diffusion processing based on the spliced ​​features to obtain the predicted action latent variables, which are the facial action sequences generated by the diffusion model.

[0148] In step S550 of some embodiments, in order to generate a better facial action sequence, a loss is calculated based on the reference head posture, the sample head posture, the reference action latent variable and the predicted action latent variable to obtain a second target loss, so as to guide the generation of the facial action sequence based on the second target loss.

[0149] In step S560 of some embodiments, the second target loss is minimized, the model parameters of the original audio encoder are adjusted to obtain the target audio encoder, the model parameters of the original head pose predictor are adjusted to obtain the target head pose predictor, and the model parameters of the diffusion model are adjusted to obtain the target action generation model.

[0150] The above steps S510 to S560 model the generation of facial action sequences through the diffusion model and combine it with the audio-based head posture predictor to generate natural and coherent facial action sequences, thereby improving the quality and diversity of video generation.

[0151] See also Figure 6 In some embodiments, step S540 may include but is not limited to steps S610 to S630:

[0152] Step S610, performing back diffusion processing on the sample fusion features and the noise-added action latent variable through the first conformal network layer to obtain the intermediate action latent variable output by the first conformal network layer;

[0153] Step S620, performing back diffusion processing on the current conformal network layer according to the sample fusion features and the intermediate action latent variables output by the previous conformal network layer to obtain the intermediate action latent variables output by the current conformal network layer;

[0154] Step S630: until the intermediate action latent variable output by the last conformal network layer is obtained, the predicted action latent variable is obtained.

[0155] In step S610 of some embodiments, a first time step, i.e., time step T, is encoded using a multilayer perceptron to obtain a first time feature. The sample fusion feature, the noisy action latent variable, and the first time feature are concatenated, and the concatenated feature is back-diffused through a first conformal network layer to obtain an intermediate action latent variable output by the first conformal network layer.

[0156] In step S620 of some embodiments, starting from the second conformal network layer, the current time step is encoded through a multilayer perceptron to obtain the current time feature, where the current time step is the previous time step minus 1. The sample fusion feature, the intermediate action latent variable output by the previous conformal network layer, and the current time feature are concatenated, and the concatenated current feature is back-diffused through the current conformal network layer to obtain the intermediate action latent variable output by the current conformal network layer.

[0157] In step S630 of some embodiments, step S620 is repeated until the intermediate action latent variable output by the last conformal network layer is obtained, and the last intermediate action latent variable is used as the predicted action latent variable.

[0158] The above steps S610 to S630 can capture the complex structure and details of facial movements through back diffusion, thereby generating a high-quality facial movement sequence.

[0159] See also Figure 7 In some embodiments, step S550 may include but is not limited to steps S710 to S730:

[0160] Step S710, performing loss calculation based on the reference head posture and the sample head posture to obtain a third sub-loss;

[0161] Step S720, performing loss calculation based on the reference action latent variable and the predicted action latent variable to obtain a fourth sub-loss;

[0162] Step S730: sum the third sub-loss and the fourth sub-loss to obtain a second target loss.

[0163] In step S710 of some embodiments, in order to make the predicted sample head pose close to the true reference head pose, the difference between the reference head pose and the sample head pose is calculated based on the L1 loss function to obtain a third sub-loss.

[0164] In step S720 of some embodiments, in order to make the predicted facial action sequence close to the real facial action sequence, the difference between the reference action latent variable and the predicted action latent variable is calculated to obtain a fourth sub-loss.

[0165] In step S730 of some embodiments, the third sub-loss and the fourth sub-loss are added to obtain a second target loss. The second target loss is expressed as:

[0166]

[0167] Among them, L dif Indicates the loss of the second target; To predict the action latent variable; is the reference action hidden variable; L1 represents the L1 loss function; x p is the reference head pose; is the sample head pose.

[0168] Through the above steps S710 to S730, a second target loss can be obtained to guide the optimization process of the facial action sequence based on the second target loss.

[0169] The model's performance in generating speaking face videos was tested on a comprehensive dataset constructed from multiple public datasets. This embodiment of the application selected the High-Definition Talking Face Dataset (HDTF) and the Casual Conversation datasets v1 & v2 (CC v1 & v2) for model testing. These datasets contain thousands of speaking face videos of different people, with identities covering different genders, ages, skin types, and languages. The specific collection distribution is shown in Table 1.

[0170]

[0171] Table 1

[0172] The original data refers to the data in the dataset that has not been preprocessed, and the filtered data refers to the data filtered out from the dataset for model testing.

[0173] In order to evaluate the visual quality of talking face videos, Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS) are used to measure the similarity between the generated video distribution and the real distribution, and Peak Signal-to-Noise Ratio (PSNR) is used to evaluate the pixel-level mean square error between the real video and the generated video. In addition, the average distance (Average Keypoint Distance, AKD) between the facial key points of the real video frame and the reconstructed video frame and the motion stability index (MSI) are also detected, and the quality of the talking face video reconstructed by the variational autoencoder is evaluated using this index. In this embodiment of the application, the FOMM and face-vid2vid models are used as baseline models for performance testing. These models all use the feature warping method and are video-driven baseline models. This method is usually used as a priori technology in the talking face video generation process. The variational autoencoder of the embodiment of the present application is quantitatively compared with the baseline model, and the performance test results are shown in Table 2.

[0174]

[0175] Table 2

[0176] In Table 2, ↑ indicates that performance improvement or increase in indicators is positive and expected, and higher values ​​generally indicate better model performance; ↓ indicates that performance degradation or decrease in indicators is positive and expected, and lower values ​​generally indicate better model performance.

[0177] As shown in Table 2, the variational autoencoder of the present embodiment achieved optimal performance during self-reconstruction compared to the baseline method. Cross-generation refers to the process of video generation using a variational autoencoder and a diffusion model. In this cross-generation task, the method of the present embodiment showed a significant performance difference from the baseline method. This result demonstrates that the method of the present embodiment better decouples appearance and motion representations in speaking face videos.

[0178] See also Figure 8 The present application also provides a device for generating a video of a speaking face, which can implement the above-mentioned database query method. The device for generating a video of a speaking face:

[0179] An acquisition module 810 is configured to acquire target audio and target facial image;

[0180] A first encoding module 820 is configured to extract audio coding features of a target audio using a target audio encoder, predict a subject head posture sequence based on the audio coding features using a target head posture predictor, and perform feature fusion on the audio coding features and the subject head posture sequence to obtain a target fused feature;

[0181] A forward diffusion module 830 is used to perform forward diffusion processing on the target face image to obtain a noise action sequence;

[0182] An action generation module 840 is configured to perform back diffusion processing based on the target fusion features and the noise action sequence using a target action generation model to obtain a target action sequence;

[0183] The second encoding module 850 is used to extract the target appearance latent variable of the target face image through the target image encoder;

[0184] The video generation module 860 is used to generate a target speaking face video based on the target action sequence and the target appearance latent variables through a target decoder.

[0185] The specific implementation of the speaking face video generation device is basically the same as the specific embodiment of the speaking face video generation method described above, and will not be repeated here.

[0186] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the aforementioned method for generating a speaking face video. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0187] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0188] The processor 910 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0189] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called by the processor 910 to execute the speaking face video generation method of the embodiments of this application.

[0190] Input / output interface 930, used to implement information input and output;

[0191] Communication interface 940, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0192] bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0193] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0194] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for generating a speaking face video.

[0195] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0196] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0197] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0198] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0199] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0200] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0201] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0203] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0206] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for generating a speaking face video, characterized in that: The method comprises: Obtain target audio and target face image; Extracting audio coding features of the target audio through a target audio encoder, predicting a head posture sequence of the object based on the audio coding features through a target head posture predictor, and performing feature fusion on the audio coding features and the head posture sequence of the object to obtain a target fusion feature; Performing forward diffusion processing on the target face image to obtain a noise action sequence; Performing reverse diffusion processing on the target fusion features and the noise action sequence through a target action generation model to obtain a target action sequence; Extracting a target appearance latent variable of the target face image through a target image encoder; A target speaking face video is generated by a target decoder based on the target action sequence and the target appearance latent variable.

2. The method according to claim 1, characterized in that The target image encoder and the target decoder are jointly trained according to the following steps: Obtain a first sample speaking face video; Performing facial key point detection on the first sample speaking face video to obtain a facial key point sequence, and extracting sample action latent variables of the facial key point sequence; Randomly selecting a candidate face image from the first sample speaking face video, and extracting a sample appearance latent variable of the candidate face image through an original image encoder; wherein the characteristic scale of the sample appearance latent variable is greater than the characteristic scale of the sample action latent variable; Generate a predicted speaking face video based on the sample action latent variables and the sample appearance latent variables through an original decoder; Calculating a first target loss based on the sample action latent variable, the sample appearance latent variable, the first sample speaking face video, and the predicted speaking face video; The model parameters of the original image encoder are adjusted according to the first target loss to obtain the target image encoder, and the model parameters of the original decoder are adjusted according to the first target loss to obtain the target decoder.

3. The method according to claim 2, characterized in that The calculating a first target loss according to the sample action latent variable, the sample appearance latent variable, the first sample speaking face video, and the predicted speaking face video includes: Performing loss calculation based on the sample action latent variable, the sample appearance latent variable, and a preset standard normal distribution to obtain a first sub-loss; Performing loss calculation based on the first sample speaking face video and the predicted speaking face video to obtain a second sub-loss; The first sub-loss and the second sub-loss are summed to obtain the first target loss.

4. The method according to claim 3, characterized in that The first sample speaking face video includes a plurality of sample face images, the predicted speaking face video includes a plurality of predicted face images, and the loss calculation is performed based on the first sample speaking face video and the predicted speaking face video to obtain a second sub-loss, including: Calculating a pixel difference value between each sample face image and the corresponding predicted face image; Performing image discrimination on each of the sample face images by a discriminator to obtain a first probability that the sample face image is a real image; Performing image discrimination on each of the predicted facial images by the discriminator to obtain a second probability that the predicted facial image is a composite image; The second sub-loss is determined according to the pixel difference value, the first probability, and the second probability.

5. The method according to any one of claims 1 to 4, characterized in that The target audio encoder, the target head pose predictor, and the target action generation model are jointly trained according to the following steps: Obtaining a second sample speaking face video, and extracting sample audio, reference head posture, and reference action latent variables from the second sample speaking face video; Extracting sample audio coding features of the sample audio through an original audio encoder, predicting a sample head posture based on the sample audio coding features through an original head posture predictor, and performing feature fusion on the sample audio coding features and the reference head posture to obtain a sample fusion feature; Performing forward diffusion processing on the reference action latent variable to obtain a noisy action latent variable; Performing reverse diffusion processing based on the sample fusion features and the noise-added action latent variables through a diffusion model to obtain a predicted action latent variable; Calculating a second target loss based on the reference head posture, the sample head posture, the reference action latent variable, and the predicted action latent variable; Adjust the model parameters of the original audio encoder according to the second target loss to obtain the target audio encoder, adjust the model parameters of the original head posture predictor according to the second target loss to obtain the target head posture predictor, and adjust the model parameters of the diffusion model according to the second target loss to obtain the target action generation model.

6. The method according to claim 5, characterized in that The diffusion model includes a plurality of conformal network layers, and the back diffusion processing is performed by the diffusion model according to the sample fusion features and the noise-added action latent variables to obtain the predicted action latent variables, including: Performing back diffusion processing according to the sample fusion feature and the noise-added action latent variable through the first conformal network layer to obtain the intermediate action latent variable output by the first conformal network layer; Performing back diffusion processing by the current conformal network layer according to the sample fusion feature and the intermediate action latent variable output by the previous conformal network layer to obtain the intermediate action latent variable output by the current conformal network layer; Until the intermediate action latent variable output by the last conformal network layer is obtained, the predicted action latent variable is obtained.

7. The method according to claim 5, characterized in that The calculating a second target loss according to the reference head posture, the sample head posture, the reference action latent variable, and the predicted action latent variable includes: Performing loss calculation based on the reference head posture and the sample head posture to obtain a third sub-loss; Performing loss calculation based on the reference action latent variable and the predicted action latent variable to obtain a fourth sub-loss; The third sub-loss and the fourth sub-loss are summed to obtain the second target loss.

8. A device for generating a video of a speaking face, characterized in that: The device comprises: An acquisition module is used to acquire target audio and target face image; a first encoding module, configured to extract audio coding features of the target audio through a target audio encoder, predict a subject head posture sequence based on the audio coding features through a target head posture predictor, and perform feature fusion on the audio coding features and the subject head posture sequence to obtain a target fusion feature; A forward diffusion module, configured to perform forward diffusion processing on the target face image to obtain a noise action sequence; An action generation module is configured to perform reverse diffusion processing based on the target fusion feature and the noise action sequence through a target action generation model to obtain a target action sequence; A second encoding module is used to extract the target appearance latent variable of the target face image through a target image encoder; The video generation module is used to generate a target speaking face video based on the target action sequence and the target appearance latent variable through a target decoder.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images