Lip shape driving model generation method and device, electronic equipment, and storage medium

Abstract

The present disclosure provides a method and device for generating a lip driving model, electronic equipment and storage medium, relating to the technical field of artificial intelligence, in particular to the technical field of computer vision, augmented reality, virtual reality, deep learning and the like, and can be applied to the scene of meta universe, virtual digital person and the like. It comprises: inputting audio data, a mask image and a reference face image into an initial lip driving model to obtain a lip image; determining a first loss according to the difference between the lip image and a sample face image; inputting the audio data and the lip image into a plurality of synchronization networks generated based on different types of speech respectively to obtain a second loss output by each synchronization network, and correcting the initial lip driving model according to the first loss and the minimum value of the plurality of second losses to obtain a lip driving model. Thus, the generated lip driving model can have high accuracy in different types of speech scenarios.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, augmented reality, virtual reality, deep learning and the like, which can be applied to the scene of meta universe, virtual digital person and the like, and specifically relates to a method and device for generating a lip driving model, an electronic device and a storage medium. BACKGROUND

[0002] With the vigorous development of artificial intelligence (AI) and big data technology, AI has penetrated into all aspects of life, and virtual object technology is a relatively important subfield in AI technology, which can construct a virtual object through AI technology such as deep learning technology, and drive the facial expression of the virtual object to simulate human speech.

[0003] The main application of facial expression driving is to realize the lip driving of the virtual object through the voice to achieve the purpose of voice and lip synchronization. Therefore, how to realize the accurate lip driving of the virtual object by different types of voice has become a key research direction. SUMMARY

[0004] The present disclosure provides a method and device for generating a lip driving model, an electronic device and a storage medium.

[0005] According to a first aspect of the present disclosure, a method for generating a lip driving model is provided, comprising:

[0006] obtaining a training data set, wherein the training data set comprises audio data, sample facial images aligned with the audio data, mask images corresponding to the sample facial images and not containing lip regions, and reference facial images;

[0007] inputting the audio data, the mask images and the reference facial images into an initial lip driving model to obtain a lip image;

[0008] determining a first loss according to the difference between the lip image and the sample facial image;

[0009] inputting the audio data and the lip image into a plurality of synchronization networks respectively to obtain a second loss output by each of the synchronization networks, wherein each of the synchronization networks is generated based on different types of voice training;

[0010] correcting the initial lip driving model according to the first loss and the minimum value of the plurality of second losses to obtain a lip driving model.

[0011] According to a second aspect of the present disclosure, a lip driving method is provided, comprising:

[0012] obtaining to-be-driven face image data, mask image data corresponding to the to-be-driven face image data and not containing a lip region, and target speech data;

[0013] inputting the to-be-driven face image data, the mask image data and the target speech data into a lip shape driving model to obtain target lip shape image data driven by the target speech data; wherein the lip shape driving model is obtained based on the generation method of the lip shape driving model provided in the first aspect of the present disclosure.

[0014] According to a third aspect of the present disclosure, a lip shape driving model generation device is provided, comprising:

[0015] A first obtaining module is configured to obtain a training data set, wherein the training data set comprises audio data, sample face images aligned with the audio data, mask images corresponding to the sample face images and not containing a lip region, and reference face images.

[0016] A second obtaining module is configured to input the audio data, the mask images and the reference face images into an initial lip shape driving model to obtain a lip shape image.

[0017] A determining module is configured to determine a first loss according to the difference between the lip shape image and the sample face image.

[0018] A third obtaining module is configured to input the audio data and the lip shape image into a plurality of synchronization networks respectively to obtain a second loss output by each of the synchronization networks, wherein each of the synchronization networks is generated based on different types of speech training.

[0019] A fourth obtaining module is configured to correct the initial lip shape driving model according to the first loss and the minimum value of the plurality of second losses to obtain a lip shape driving model.

[0020] According to a fourth aspect of the present disclosure, a lip shape driving device is provided, comprising:

[0021] A first obtaining module is configured to obtain to-be-driven face image data, mask image data corresponding to the to-be-driven face image data and not containing a lip region, and target speech data.

[0022] A second obtaining module is configured to input the to-be-driven face image data, the mask image data and the target speech data into a lip shape driving model to obtain target lip shape image data driven by the target speech data; wherein the lip shape driving model is obtained by the lip shape driving model generation device provided in the first aspect of the present disclosure.

[0023] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0024] at least one processor; and

[0025] a memory connected with the at least one processor in communication; wherein

[0026] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for generating a lip-driven model according to the first aspect, or perform the method for lip driving according to the second aspect.

[0027] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to cause the computer to perform the method for generating a lip-driven model according to the first aspect, or perform the method for lip driving according to the second aspect.

[0028] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method for generating a lip-driven model according to the first aspect, or perform the steps of the method for lip driving according to the second aspect.

[0029] The method for generating a lip-driven model, the device, the electronic device and the storage medium provided by the present disclosure have the following advantages

[0030] Advantages:

[0031] In the embodiments of the present disclosure, first, the audio data, the mask image and the reference face image are input into an initial lip-driven model to obtain a lip image, then a first loss is determined according to the difference between the lip image and a sample face image, and the audio data and the lip image are respectively input into a plurality of synchronization networks to obtain a second loss output by each synchronization network, wherein each synchronization network is generated based on different types of speech training. Finally, the initial lip-driven model is corrected according to the first loss and the minimum value of the plurality of second losses to obtain the lip-driven model. Thus, the second loss generated by the best synchronization network is selected from the synchronization networks generated by training of different types of speech to supervise the training of the lip-driven network, so that the generated lip-driven model can have high accuracy in different types of speech scenarios.

[0032] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them:

[0034] Figure 1 is a flowchart of a method for generating a lip shape driving model according to an embodiment of the present disclosure;

[0035] Figure 2 is a flowchart of a method for generating a lip shape driving model according to another embodiment of the present disclosure;

[0036] Figure 3 is a flowchart of a method for generating a lip shape driving model according to another embodiment of the present disclosure;

[0037] Figure 4 is a flowchart of a method for generating a lip shape driving model according to another embodiment of the present disclosure;

[0038] Figure 5 is a flowchart of a method for generating a lip shape driving model according to another embodiment of the present disclosure;

[0039] Figure 6 is a structural diagram of a device for generating a lip shape driving model according to an embodiment of the present disclosure;

[0040] Figure 7 is a structural diagram of a device for generating a lip shape driving model according to another embodiment of the present disclosure;

[0041] Figure 8 is a block diagram of an electronic device for implementing a method for generating a lip shape driving model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, descriptions of well-known functions and structures are omitted in the following description.

[0043] Embodiments of the present disclosure relate to the technical field of artificial intelligence, such as computer vision and deep learning.

[0044] Artificial intelligence (AI) is an English abbreviation. It is a new technical science that studies, develops, simulates, extends and expands human intelligence.

[0045] Deep learning is learning the internal rules and representation levels of sample data, and the information obtained in the learning process is helpful for the interpretation of data such as text, images and sound. The ultimate goal of deep learning is to enable machines to have analytical learning ability like humans, and to recognize data such as text, images and sound.

[0046] Computer vision refers to machine vision that uses cameras and computers to replace human eyes to identify, track and measure targets, and further performs image processing to make the computer processing more suitable for human observation or transmission to instrument detection.

[0047] Augmented Reality (AR) is a technology that skillfully combines virtual information with the real world, and widely uses multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and other technical means. After simulating computer-generated text, images, three-dimensional models, music, video and other virtual information, it is applied to the real world, and the two kinds of information complement each other, so as to realize the "enhancement" of the real world.

[0048] Virtual Reality (VR) technology is a computer simulation system that can create and experience virtual worlds. It uses computer-generated simulation environments to immerse users in the environment. Virtual reality technology is to use real-life data, through computer technology to generate electronic signals, and combine them with various output devices to convert them into phenomena that people can experience. These phenomena can be real objects in reality, or substances that we cannot see with our eyes, which are represented by three-dimensional models.

[0049] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0050] The generation method and device of the lip shape driving model, the electronic equipment and the storage medium of the embodiment of the present disclosure will be described below with reference to the accompanying drawings.

[0051] It should be noted that the execution subject of the lip shape driving model generation method of the present embodiment is a lip shape driving model generation device, which can be realized by software and / or hardware. The device can be configured in an electronic equipment, which can include but is not limited to a terminal, a server end, etc.

[0052] Figure 1 is a flowchart of a lip shape driving model generation method according to an embodiment of the present disclosure.

[0053] As Figure 1 shown, the lip shape driving model generation method comprises:

[0054] S101: Obtain a training data set, wherein the training data set contains audio data, a sample face image aligned with the audio data, a mask image corresponding to the sample face image and not containing a lip region, and a reference face image.

[0055] The audio data and the sample face image aligned with the audio data can be obtained from a video containing sound. When each frame of audio is obtained from the video, the sample face image corresponding to each frame of audio is obtained.

[0056] Optionally, the video can be obtained in various ways, and one or more ways can be used to obtain the video. For example, the video can be recorded in real time, pre-stored video can be obtained, video sent by other electronic devices can be received, or video downloaded from the network can be obtained. The present disclosure does not limit this.

[0057] In the embodiments of the present disclosure, in order to improve the generalization ability of the lip shape driving model, i.e., to process various types of speech, the type of audio in the video can include English, Chinese, and minority languages. Each set of training data in the training data set can be a set of data obtained by dividing the video by words, for example, each set of training data can contain 29 frames of video. Alternatively, a set of data obtained by dividing the video by sentences can also be used. Alternatively, the resolution of the video can also be different. The present disclosure does not limit this.

[0058] Optionally, after obtaining each frame of audio from the video, the mel-frequency spectrum feature corresponding to each frame of audio can be further obtained, and then the mel-frequency spectrum feature corresponding to each frame of audio can be used as the audio data.

[0059] It should be noted that the purpose of speech-driven face is to tamper with the lip shape and the face region dominated by the chin, and therefore, the mask image corresponding to the sample face image and not containing the lip region can be an image obtained by masking the lip region, face region, and chin region of the face in the sample face image. Thus, the region without masking can be kept consistent.

[0060] In addition, since the masking region in the mask image is relatively large, the training difficulty for the generation task is also increased. The generated result is unstable due to insufficient reference information, and the interference is large. Therefore, the face can be randomly sampled from the same video as the reference face image, and the reference face image is randomly sampled. Only the lip shape information is learned by the model from the reference face image (such as the teeth, lip color, and face color should be what), and the lip movement information is not introduced into the result.

[0061] S102: input the audio data, the mask image and the reference face image into the initial lip shape driving model to obtain a lip shape image.

[0062] In the embodiments of the present disclosure, the continuous frame video can be used as a driving template, for example, 0.2 seconds of common 5 frames of audio are used as input units, that is, 5 frames of continuous audio data, 5 frames of continuous mask images, 5 frames of reference face images and 5 frames of continuous sample face images are input into the lip shape driving model each time. The generated lip shape image is also a 5-frame continuous video.

[0063] The initial lip shape driving model can be a lip shape driving model that has not been trained. Optionally, the initial lip shape driving model can include an audio encoder, an image encoder and a generation network. The audio encoder is configured to encode the audio data, the image encoder is configured to encode the mask image and the reference face image, and the generation network is configured to generate the lip shape image based on the outputs of the audio encoder and the image encoder.

[0064] Optionally, the audio data can be first input into the audio encoder in the initial lip shape driving model to obtain audio features. The mask image and the reference face image can be input into the image encoder in the initial lip shape driving model to obtain first image features corresponding to the mask image and second image features corresponding to the reference sample image. Then, the audio features, the first image features and the second image features can be input into the generation network in the initial lip shape driving model to obtain the lip shape image.

[0065] The audio features output by the audio encoder can be high-order semantic features corresponding to the audio data, and the audio features include lip movement features. For example, the dimension of the audio features can be 512.

[0066] The first image features include high-order features of the eyes, eyebrows and contours of the face. The second image features include high-order features of the teeth, lip color and face color of the face.

[0067] Optionally, the audio features, the first image features and the second image features can also be input into the generation network in the initial lip shape driving model after being spliced.

[0068] Thus, the audio encoder in the initial lip-driven model is input with the audio data to obtain audio features, then the image encoder in the initial lip-driven model is input with the mask image and the reference face image to obtain first image features corresponding to the mask image and second image features corresponding to the reference sample image, and then the audio features, the first image features and the second image features are input into the generation network in the initial lip-driven model to obtain the lip image. Thus, different network structures in the initial lip-driven model can process different data, so that the lip image can be accurately obtained.

[0069] S103: Determine the first loss according to the difference between the lip image and the sample face image.

[0070] Optionally, the mean absolute error formula (i.e., the L1 loss function) can be used to calculate the difference between the lip image and the sample face image to obtain the first loss.

[0071] Alternatively, other applicable loss functions can be used to calculate the difference between the lip image and the sample face image to obtain the first loss. The present disclosure does not limit this.

[0072] S104: Input the audio data and the lip image into a plurality of synchronization networks respectively to obtain a second loss output by each synchronization network, wherein each synchronization network is generated based on different types of speech.

[0073] In the embodiments of the present disclosure, integrated learning can be performed for the plurality of synchronization networks (SyncNet) during training. In this way, the lip-driven model generated by training can maintain a high-precision lip movement state in different scenarios.

[0074] Specifically, K SyncNets are trained for different scene data (i.e., different types of speech, such as English, Chinese, French, etc.), and each SyncNet is trained in a self-supervised manner, i.e., the cosine similarity of the calculated features is calculated, and a binary cross-entropy loss function (BCEloss) is used to determine whether the speech and the lip are synchronized or not. Synchronization is 1 and asynchronization is 0. The training data of the SyncNet includes positive samples and negative samples. Each positive sample is the synchronized speech and lip in the same video, and the negative sample is the asynchronized speech and lip in the same video.

[0075] In the embodiments of the present disclosure, after obtaining the lip image output by the generation network, the audio data and the lip image can be input into the K trained synchronization networks to obtain K second losses.

[0076] S105: correct the initial lip-driven model according to the first loss and the minimum value of the plurality of second losses to obtain the lip-driven model.

[0077] Specifically, the training loss can be determined based on a certain rule according to the first loss and the minimum value of the plurality of second losses. For example, the first loss and the minimum value of the plurality of second losses can be added, and the result of the addition is taken as the training loss. For another example, the first loss and the minimum value of the plurality of second losses can be averaged, and the obtained average value is taken as the training loss.

[0078] After the minimum value of the first loss and the plurality of second losses, the audio encoder, the image encoder and the generation network in the initial lip-driven model can be corrected to obtain the lip-driven model.

[0079] Optionally, the correction stop condition can be that the number of iterations reaches a preset number, or the sum of the first loss and the minimum value of the plurality of second losses is less than a threshold value. The preset number and the threshold value can be set by those skilled in the art according to actual needs.

[0080] It should be noted that the smaller the second loss is, the better the performance of the training data in the synchronization network is; the larger the second loss is, the less good the synchronization network itself is in discriminating the training data. If the second loss with a larger value is used to supervise the training of the lip-driven model, it may cause overtraining and result in poor lip movement. In the embodiment of the present disclosure, the best synchronization network lip-driven model is selected for supervised training, so that the generated lip-driven model has high accuracy in different types of speech scenes, and does not need to worry about the "yin and yang strange" face-to-face conversation (face-talking) caused by the difference between scenes.

[0081] Moreover, the method for generating the lip-driven model in the present disclosure has good expansibility. For other special scenes, only one scene corresponding synchronization network needs to be trained and integrated, and the lip-driven model needs to be fine-tuned, which has very low training cost.

[0082] In the embodiment of the present disclosure, first, the audio data, the mask image and the reference face image are input into the initial lip shape driving model to obtain a lip shape image, then a first loss is determined according to the difference between the lip shape image and the sample face image, and the audio data and the lip shape image are input into a plurality of synchronization networks respectively to obtain a second loss output by each synchronization network, wherein each synchronization network is generated based on different types of speech training; finally, the initial lip shape driving model is corrected according to the first loss and the minimum value of the plurality of second losses to obtain the lip shape driving model. Thus, the second loss generated by the best synchronization network is selected from the synchronization networks generated by the plurality of different types of speech training to supervise the training of the lip shape driving network, so that the generated lip shape driving model has high accuracy in different types of speech scenes.

[0083] Figure 2 is a flowchart of a method for generating a lip shape driving model according to another embodiment of the present disclosure; as shown in Figure 2 , the method for generating a lip shape driving model comprises:

[0084] S201: obtaining a training data set, wherein the training data set contains audio data, sample face images aligned with the audio data, mask images corresponding to the sample face images and not containing lip regions, and reference face images.

[0085] S202: inputting the audio data, the mask image and the reference face image into an initial lip shape driving model to obtain a lip shape image.

[0086] Figure 3 is a flowchart of a method for generating a lip shape driving model according to an embodiment of the present disclosure; as shown in Figure 3 , the audio encoder of the initial lip shape driving model can contain a plurality of audio encoding layers, the dimensions of the audio features output by each audio encoding layer are inconsistent, the audio features output by the audio encoding layers closer to the input layer of the audio encoder have lower dimensions, and the audio features output by the last audio encoding layer have the highest dimension, i.e. high-order semantic features.

[0087] The image encoder of the initial lip shape driving model can contain a plurality of image encoding layers, the dimensions of the image features output by each image encoding layer are inconsistent, the image features output by the image encoding layers closer to the input layer of the image encoder have lower dimensions, and the image features output by the last image encoding layer have the highest dimension, i.e. high-order image features.

[0088] The generation network of the initial lip driving model can also include multiple network layers. In addition, in order to generate high-quality faces, low-order features can also be introduced in the process of generating the lip image by the generation network in the form of a skip-connection, so that the lip driving model is trained faster, and the generated face is clearer and more vivid.

[0089] As shown in Figure 3 In the embodiment of the present disclosure, the audio data is 5 frames, the mask image is 5 frames, the sample face image is 5 frames, the reference image is 5 frames, the 5 frames of audio data are input into the audio encoder in the initial lip driving model, and the multiple audio encoding layers contained in the audio encoder are used to sequentially encode the audio data to obtain audio features; the 5 frames of mask images and the 5 frames of reference images are input into the image encoder in the initial lip driving model, and the multiple image encoding layers in the image encoder are used to sequentially encode the mask images and the reference face images to obtain first image features corresponding to the mask images and second image features corresponding to the reference sample images, and then the audio features, the first image features and the second image features are input into the generation network in the initial lip driving model, and the multiple network layers in the generation network are used to analyze the features after the fusion of the audio features, the first image features and the second image features to obtain a lip image.

[0090] S203: Determine the face reconstruction loss between the lip image and the sample face image based on the mean absolute error formula.

[0091] As shown in Figure 3 Based on the mean absolute error formula, the L1 loss of the 5 frames of sample face images and the 5 frames of lip images is calculated, and then the average value of the loss of the 5 frames of lip images is determined as the face reconstruction loss.

[0092] S204: Input the lip image and the sample face image into the convolutional neural network VGG to obtain the content-aware loss between the lip image and the sample face image.

[0093] The convolutional neural network VGG can be pre-trained and used to determine the content-aware loss between the lip image and the sample face image.

[0094] As shown in Figure 3 The 5 frames of lip images and the 5 frames of sample face images are input into the convolutional neural network VGG, so that the convolutional neural network VGG calculates the content-aware loss between each frame of lip image and the corresponding sample face image, and then the average value of the content-aware loss corresponding to the 5 frames of lip images is determined as the content-aware loss corresponding to the 5 frames of lip images.

[0095] S205: input the lip shape image and the sample face image into the adversarial network to obtain an adversarial loss between the lip shape image and the sample face image.

[0096] The adversarial network (GAN) can be pre-trained, and is a network for determining the adversarial loss between the lip shape image and the sample face image.

[0097] As shown in Figure 3 The 5 lip shape images and the 5 sample face images are input into the adversarial network GAN, so that the adversarial network GAN calculates the adversarial loss between each lip shape image and the corresponding sample face image, and then the average of the adversarial losses corresponding to the 5 lip shape images is determined as the adversarial loss corresponding to the 5 lip shape images.

[0098] S206: fuse the face reconstruction loss, the content-aware loss, and the adversarial loss to determine a first loss.

[0099] In the embodiments of the present disclosure, the face reconstruction loss, the content-aware loss, and the adversarial loss are fused to determine the first loss, so that the first loss can be determined from multiple dimensions, and the determined first loss is more comprehensive and accurate.

[0100] Optionally, a first weight corresponding to the face reconstruction loss, a second weight corresponding to the content-aware loss, and a third weight corresponding to the adversarial loss can be obtained, and then the face reconstruction loss, the content-aware loss, and the adversarial loss are fused based on the first weight, the second weight, and the third weight to obtain the first loss. Thus, the first loss can be more accurately determined, thereby increasing the accuracy of the lip-driven model training.

[0101] The sum of the first weight, the second weight, and the third weight can not be 1, and the greater the influence of a loss on the lip-driven model, the greater the corresponding weight value.

[0102] For example, the first weight can be 10, the second weight can be 0.3, and the third weight can be 1. The present disclosure does not limit this.

[0103] Alternatively, the face reconstruction loss, the content-aware loss, and the adversarial loss can be added to obtain the first loss.

[0104] S207: input the audio data and the lip shape image into a plurality of synchronization networks respectively to obtain a second loss output by each synchronization network, wherein each synchronization network is generated based on a different type of speech.

[0105] As shown in Figure 3As shown, after 5 frames of lip shape images are acquired, 5 frames of audio data and 5 frames of lip shape images are simultaneously input into multiple synchronization networks (such as SyncNet-1, …, SyncNet-k, etc.) in Figure 3 , and then a synchronization loss (Sync loss) output by the multiple synchronization networks, i.e., a second loss, is acquired.

[0106] S208: A fourth weight corresponding to a minimum value in the multiple second losses is acquired.

[0107] The fourth weight can be a preset proportion of the minimum value in the multiple second losses and the first loss. For example, the fourth loss can be 0.3, and the present disclosure is not limited thereto.

[0108] S209: The minimum value in the second loss and the first loss are fused based on the fourth weight to acquire a target loss.

[0109] In the embodiments of the present disclosure, the target loss is determined according to the face reconstruction loss, the content perception loss, the adversarial loss, and the minimum value in the multiple second losses, which further increases the accuracy and efficiency of training the lip shape driven model.

[0110] S210: The audio encoder, the image encoder, and the generation network in the initial lip shape driven model are corrected based on the target loss to acquire the lip shape driven model.

[0111] In the embodiments of the present disclosure, the audio data, the mask image and the reference face image are input into an initial lip driving model to obtain a lip image, then a face reconstruction loss between the lip image and a sample face image is determined based on a mean absolute error formula, the lip image and the sample face image are input into a convolutional neural network VGG to obtain a content-aware loss between the lip image and the sample face image, the lip image and the sample face image are input into a generative adversarial network to obtain an adversarial loss between the lip image and the sample face image, then the face reconstruction loss, the content-aware loss and the adversarial loss are fused to determine a first loss, the audio data and the lip image are input into a plurality of synchronization networks respectively to obtain a second loss output by each synchronization network, wherein each synchronization network is generated based on different types of speech, finally, based on a fourth weight corresponding to a minimum value in the plurality of second losses, the first loss and the minimum value in the second losses are fused to obtain a target loss, and finally, based on the target loss, the audio encoder, the image encoder and the generative network in the initial lip driving model are modified to obtain a lip driving model. Therefore, the first loss can be determined from multiple dimensions of the face reconstruction loss, the content-aware loss and the adversarial loss, so that the determined first loss is more accurate, and the target loss obtained by fusing the determined first loss and the minimum value in the plurality of second losses is more accurate, thereby improving the efficiency and accuracy of the lip driving model training, and further improving the accuracy of the generated lip driving model.

[0112] Figure 4 is a flowchart of a lip driving method according to yet another embodiment of the present disclosure; as shown in Figure 4 the lip driving method comprises:

[0113] S401: obtaining to-be-driven face image data, mask image data corresponding to the to-be-driven face image data and not containing a lip region, and target speech data.

[0114] The target speech data can refer to a piece of speech, which is used to drive the lips of a virtual object, so that the lips of the virtual object can be opened and closed according to the target speech data, that is, the lip shape of the virtual object is similar to the lip shape of a real person when the real person speaks the target speech data, and the process of the virtual object speaking is simulated through lip driving.

[0115] The target speech data can be obtained in various ways, such as real-time recording of a piece of speech, obtaining a pre-stored piece of speech, receiving a piece of speech sent by another electronic device, or downloading a piece of speech from the network.

[0116] The to-be-driven face image data can refer to image data including virtual object face content. In the case of a virtual character as the virtual object, the to-be-driven face image data can be human face data. The to-be-driven face image data can include only one human face image, or can include multiple human face images, which are not specifically limited here. The multiple human face images can be referred to as a human face series, which refers to multiple human face images of the same virtual character. The poses, expressions, lip shapes, etc. of the human faces in the multiple human face images can be different.

[0117] The lips in the to-be-driven face image data can be in an open state (i.e., the virtual object is in a speaking state) in whole or in part, or can be in a closed state in whole or in part, which are not specifically limited here. When the to-be-driven face image data is in a closed state in whole, the to-be-driven face image data can be face image data with lip shapes eliminated, i.e., the virtual object is in a silent state without speaking.

[0118] The to-be-driven face image data can be in the form of a video or an image, which are not specifically limited here.

[0119] The to-be-driven face image data can be obtained in various ways. For example, a video can be recorded in real time or images can be taken in real time as the to-be-driven face image data. A pre-stored video or images can be obtained as the target face image data. A video or images sent by another electronic device can be received as the target face image data. Or a video or images can be downloaded from the network as the to-be-driven face image data. The obtained video can include face images, and the obtained images can include face image content.

[0120] The mask image data corresponding to the to-be-driven face image data and not containing the lip region can be image data obtained by masking the lips, face, and chin region of the human face in the to-be-driven face image data.

[0121] S402: input the to-be-driven face image data, the mask image data, and the target speech data into the lip shape driving model to obtain target lip shape image data driven by the target speech data.

[0122] In this embodiment, the lip shape driving model is trained based on the training method of the lip shape driving model provided in any embodiment of the present disclosure.

[0123] Optionally, the target speech data can be input into an audio encoder in the lip driving model to obtain speech features corresponding to each frame of speech data in the target speech data, the to-be-driven face image data and the mask image data can be input into an image encoder in the lip driving model to obtain first target image features corresponding to the to-be-driven face image data and second target image features corresponding to the mask image data, and then the first target image features, the second target image features and the target speech features can be input into a generation network in the lip driving model to obtain target lip shape image data driven by the target speech data.

[0124] In the embodiments of the present disclosure, first, to-be-driven face image data, mask image data corresponding to the to-be-driven face image data and not containing a lip region, and target speech data are obtained, and then the to-be-driven face image data, the mask image data and the target speech data are input into a lip driving model to obtain target lip shape image data driven by the target speech data. In this way, the target lip shape image data driven by the target speech data is determined through the lip driving model, and the synchronization of speech and lip shape of the to-be-driven face in different types of speech scenes can be ensured.

[0125] Figure 5 is a flowchart of a lip driving method according to another embodiment of the present disclosure; as Figure 5 shown, the lip driving method comprises:

[0126] S501: obtaining to-be-driven face image data, mask image data corresponding to the to-be-driven face image data and not containing a lip region, and target speech data.

[0127] S502: inputting the target speech data into an audio encoder in the lip driving model to obtain speech features corresponding to each frame of speech data in the target speech data.

[0128] S503: fusing the speech features corresponding to each frame of speech data with target speech features corresponding to a previous frame of speech data to obtain target speech features corresponding to each frame of speech data.

[0129] In the embodiments of the present disclosure, in order to avoid the problem of jitter when generating continuous target lip shape image data, and further to avoid the problem of instability of lip movement contingency, a set of lip movement smoothing strategy is designed, that is, the speech features corresponding to the current frame of speech data are fused with the target speech features corresponding to the previous frame of speech data to obtain the target speech features corresponding to each frame of speech data.

[0130] Optionally, the speech features corresponding to each frame of speech data can be fused with the target speech features corresponding to the previous frame of speech data in a weighted manner.

[0131] Wherein, the weight corresponding to the speech feature corresponding to each frame of speech data is 1 plus the weight of the target speech feature corresponding to the previous frame of speech data. For example, the weight of the speech feature corresponding to each frame of speech data is 0.7, and the weight of the target speech feature corresponding to the previous frame of speech data is 0.3.

[0132] S504: input the to-be-driven face image data and the mask image data into an image encoder in the lip shape driving model to obtain first target image features corresponding to the to-be-driven face image data and second target image features corresponding to the mask image data.

[0133] S505: input the first target image features, the second target image features and the target speech features into a generation network in the lip shape driving model to obtain target lip shape image data driven by the target speech data.

[0134] In the embodiment of the present disclosure, first, the to-be-driven face image data, the mask image data corresponding to the to-be-driven face image data and not containing the lip region, and the target speech data are obtained, then the target speech data is input into an audio encoder in the lip shape driving model to obtain speech features corresponding to each frame of speech data in the target speech data, the speech features corresponding to each frame of speech data are fused with the target speech features corresponding to the previous frame of speech data to obtain target speech features corresponding to each frame of speech data, the to-be-driven face image data and the mask image data are input into an image encoder in the lip shape driving model to obtain first target image features corresponding to the to-be-driven face image data and second target image features corresponding to the mask image data, finally, the first target image features, the second target image features and the target speech features are input into a generation network in the lip shape driving model to obtain target lip shape image data driven by the target speech data. Thus, by fusing the speech features corresponding to each frame of speech data with the target speech features corresponding to the previous frame of speech data to obtain the target speech features corresponding to each frame of speech data, the problem of jitter of the generated continuous target lip shape image data can be avoided, and the problem of instability of lip movement incidentally caused by the jitter can be further avoided, thereby improving the lip movement smoothness of the generated target lip shape image data.

[0135] Figure 6 is a structural schematic diagram of a lip shape driving model generation device according to an embodiment of the present disclosure, as Figure 6 shown, the lip shape driving model generation device 600 comprises:

[0136] The first acquisition module 610 is configured to acquire a training data set, wherein the training data set comprises audio data, sample face images aligned with the audio data, mask images corresponding to the sample face images and not containing lip regions, and reference face images.

[0137] The second obtaining module 620 is configured to input the audio data, the mask image, and the reference face image into an initial lip shape driving model to obtain a lip shape image.

[0138] The determining module 630 is configured to determine a first loss according to a difference between the lip shape image and a sample face image.

[0139] The third obtaining module 640 is configured to input the audio data and the lip shape image into a plurality of synchronization networks respectively to obtain a second loss output by each of the synchronization networks, wherein each of the synchronization networks is generated based on a different type of speech.

[0140] The fourth obtaining module 650 is configured to correct the initial lip shape driving model according to the first loss and a minimum value of the plurality of second losses to obtain a lip shape driving model.

[0141] Optionally, the second obtaining module 620 is specifically configured to:

[0142] input the audio data into an audio encoder in the initial lip shape driving model to obtain audio features;

[0143] input the mask image and the reference face image into an image encoder in the initial lip shape driving model to obtain first image features corresponding to the mask image and second image features corresponding to the reference sample image;

[0144] input the audio features, the first image features, and the second image features into a generation network in the initial lip shape driving model to obtain the lip shape image.

[0145] Optionally, the determining module 630 is specifically configured to:

[0146] determine a face reconstruction loss between the lip shape image and the sample face image based on a mean absolute error formula;

[0147] input the lip shape image and the sample face image into a convolutional neural network VGG to obtain a content perception loss between the lip shape image and the sample face image;

[0148] input the lip shape image and the sample face image into an adversarial network to obtain an adversarial loss between the lip shape image and the sample face image;

[0149] fuse the face reconstruction loss, the content perception loss, and the adversarial loss to determine the first loss.

[0150] Optionally, the determining module 630 is specifically configured to:

[0151] obtain a first weight corresponding to the face reconstruction loss, a second weight corresponding to the content perception loss, and a third weight corresponding to the adversarial loss;

[0152] The face reconstruction loss, the content perception loss, and the adversarial loss are fused based on the first weight, the second weight, and the third weight to obtain a first loss.

[0153] Optionally, the fourth obtaining module 650 is specific for:

[0154] obtaining a fourth weight corresponding to the minimum value in the plurality of second losses;

[0155] fusing the minimum value in the second losses and the first loss based on the fourth weight to obtain a target loss;

[0156] correcting the audio encoder, the image encoder, and the generation network in the initial lip-driven model based on the target loss to obtain the lip-driven model.

[0157] It should be noted that the foregoing explanation and description of the method for generating the lip-driven model also apply to the device for generating the lip-driven model of the present embodiment, which will not be described here again.

[0158] In the present embodiment, the audio data, the mask image, and the reference face image are first input into the initial lip-driven model to obtain a lip image, then the first loss is determined according to the difference between the lip image and the sample face image, and the audio data and the lip image are respectively input into a plurality of synchronous networks to obtain the second loss output by each synchronous network, wherein each synchronous network is generated based on different types of speech training; finally, the initial lip-driven model is corrected according to the first loss and the minimum value in the plurality of second losses to obtain the lip-driven model. Thus, the second loss generated by the best synchronous network is selected from the synchronous networks generated by training of different types of speech to supervise the training of the lip-driven network, so that the generated lip-driven model can have high accuracy in different types of speech scenarios.

[0159] Figure 7 is a structural schematic diagram of a device for generating a lip-driven model according to an embodiment of the present disclosure, as Figure 7 shown, the lip-driven device 700 comprises:

[0160] The first obtaining module 710 is configured to obtain face image data to be driven, mask image data corresponding to the face image data to be driven and not containing a lip region, and target speech data.

[0161] The second obtaining module 720 is configured to input the face image data to be driven, the mask image data, and the target speech data into a lip-driven model to obtain target lip image data driven by the target speech data; wherein the lip-driven model is obtained based on the device for generating the lip-driven model of any one of claims 8-12.

[0162] Optionally, the second acquisition module 720 is specific for:

[0163] inputting the target voice data into an audio encoder in the lip driving model to acquire voice features corresponding to each frame of voice data in the target voice data;

[0164] fusing the voice features corresponding to each frame of voice data and target voice features corresponding to a previous frame of voice data to acquire target voice features corresponding to each frame of voice data;

[0165] inputting the to-be-driven face image data and the mask image data into an image encoder in the lip driving model to acquire first target image features corresponding to the to-be-driven face image data and second target image features corresponding to the mask image data;

[0166] inputting the first target image features, the second target image features, and the target voice features into a generation network in the lip driving model to acquire target lip shape image data driven by the target voice data.

[0167] It should be noted that the foregoing explanation and description of the generation method of the lip driving model also apply to the generation apparatus of the lip driving model of this embodiment, which will not be described here again.

[0168] In the embodiments of the present disclosure, first, the to-be-driven face image data, the mask image data corresponding to the to-be-driven face image data and not containing the lip region, and the target voice data are acquired, and then the to-be-driven face image data, the mask image data, and the target voice data are input into the lip driving model to acquire target lip shape image data driven by the target voice data. Thus, the target lip shape image data driven by the target voice data is determined through the lip driving model, which can ensure that the voice and the lip shape of the to-be-driven face are synchronized in different types of voice scenes.

[0169] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0170] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0171] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0172] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the method for generating a lip-driven model, or a lip-driven method. For example, in some embodiments, the method for generating a lip-driven model, or a lip-driven method, can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the method for generating a lip-driven model, or a lip-driven method, described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform a method for generating a lip-driven model, or a lip-driven method.

[0174] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0175] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0176] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0178] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0179] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service ("Virtual Private Server", or simply "VPS"). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0180] In this embodiment, first, the audio data, the mask image and the reference face image are input into the initial lip shape driving model to obtain a lip shape image, then a first loss is determined according to the difference between the lip shape image and a sample face image, and the audio data and the lip shape image are respectively input into a plurality of synchronization networks to obtain a second loss output by each synchronization network, wherein each synchronization network is generated based on different types of speech training; finally, the initial lip shape driving model is corrected according to the first loss and the minimum value of the plurality of second losses to obtain the lip shape driving model. Thus, the second loss generated by the best synchronization network selected from the synchronization networks generated by training of different types of speech is used to supervise the training of the lip shape driving network, so that the generated lip shape driving model has high accuracy in different types of speech scenes.

[0181] It should be understood that the steps can be reordered, added, or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0182] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified. In the description of the present disclosure, the word "if" and "if" can be interpreted as "when" or "when" or "in response to determining" or "in the case of".

[0183] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for generating a lip-driven model, comprising: obtaining a training data set, wherein the training data set comprises audio data, sample facial images aligned with the audio data, mask images corresponding to the sample facial images and not containing lip regions, and reference facial images; inputting the audio data, the mask images and the reference facial images into an initial lip-driven model to obtain lip shape images; determining a first loss according to a difference between the lip shape images and the sample facial images; inputting the audio data and the lip shape images into a plurality of synchronization networks respectively to obtain a second loss output by each of the synchronization networks, wherein each of the synchronization networks is generated based on different types of speech training; correcting the initial lip-driven model according to the first loss and a minimum value of the plurality of second losses to obtain a lip-driven model.

2. The method of claim 1, wherein, The inputting the audio data, the mask images and the reference facial images into the initial lip-driven model to obtain the lip shape images comprises: inputting the audio data into an audio encoder in the initial lip-driven model to obtain audio features; inputting the mask images and the reference facial images into an image encoder in the initial lip-driven model to obtain first image features corresponding to the mask images and second image features corresponding to the reference sample images; inputting the audio features, the first image features and the second image features into a generation network in the initial lip-driven model to obtain the lip shape images.

3. The method of claim 2, wherein, The determining the first loss according to the difference between the lip shape images and the sample facial images comprises: determining a face reconstruction loss between the lip shape images and the sample facial images based on a mean absolute error formula; inputting the lip shape images and the sample facial images into a convolutional neural network (VGG) to obtain a content-aware loss between the lip shape images and the sample facial images; inputting the lip shape images and the sample facial images into a generative adversarial network (GAN) to obtain an adversarial loss between the lip shape images and the sample facial images; fusing the face reconstruction loss, the content-aware loss and the adversarial loss to determine the first loss.

4. The method of claim 3, wherein, The fusing the face reconstruction loss, the content-aware loss and the adversarial loss to determine the first loss comprises: obtaining a first weight corresponding to the face reconstruction loss, a second weight corresponding to the content-aware loss, and a third weight corresponding to the adversarial loss; fusing the face reconstruction loss, the content-aware loss and the adversarial loss based on the first weight, the second weight and the third weight to obtain the first loss.

5. The method of claim 4, wherein, The correcting the initial lip-driven model according to the first loss and the minimum value of the plurality of second losses to obtain the lip-driven model comprises: obtaining a fourth weight corresponding to the minimum value of the plurality of second losses; fusing the minimum value of the second losses and the first loss based on the fourth weight to obtain a target loss; Based on the target loss, the audio encoder, the image encoder and the generation network in the initial lip-driven model are corrected to obtain the lip-driven model.

6. A lip-driven method, comprising: obtaining to-be-driven face image data, mask image data corresponding to the to-be-driven face image data and not containing a lip region, and target speech data; inputting the to-be-driven face image data, the mask image data and the target speech data into a lip-driven model to obtain target lip image data driven by the target speech data; wherein the lip-driven model is obtained based on the generation method of the lip-driven model in any one of claims 1-5.

7. The method of claim 6, wherein, The inputting the to-be-driven face image data into the lip-driven model to obtain the target lip image data driven by the target speech data comprises: inputting the target speech data into an audio encoder in the lip-driven model to obtain speech features corresponding to each frame of speech data in the target speech data; fusing the speech features corresponding to each frame of speech data with target speech features corresponding to a previous frame of speech data to obtain target speech features corresponding to each frame of speech data; inputting the to-be-driven face image data and the mask image data into an image encoder in the lip-driven model to obtain first target image features corresponding to the to-be-driven face image data and second target image features corresponding to the mask image data; inputting the first target image features, the second target image features and the target speech features into a generation network in the lip-driven model to obtain the target lip image data driven by the target speech data.

8. A generation device of a lip-driven model, comprising: a first obtaining module configured to obtain a training data set, wherein the training data set contains audio data, sample face images aligned with the audio data, mask images corresponding to the sample face images and not containing a lip region, and reference face images; a second obtaining module configured to input the audio data, the mask images and the reference face images into an initial lip-driven model to obtain lip images; a determining module configured to determine a first loss according to a difference between the lip images and the sample face images; a third obtaining module configured to input the audio data and the lip images into a plurality of synchronization networks respectively to obtain second losses output by each of the synchronization networks, wherein each of the synchronization networks is generated based on different types of speech; a fourth obtaining module configured to correct the initial lip-driven model according to the first loss and a minimum value of the plurality of second losses to obtain a lip-driven model.

9. The apparatus of claim 8, wherein, The second obtaining module is specifically configured to: input the audio data into an audio encoder in the initial lip-driven model to obtain audio features; input the mask image and the reference face image into an image encoder in the initial lip shape driving model to obtain first image features corresponding to the mask image and second image features corresponding to a reference sample image; input the audio features, the first image features and the second image features into a generation network in the initial lip shape driving model to obtain the lip shape image.

10. The apparatus of claim 9, wherein, The determination module is specifically configured to: determine a face reconstruction loss between the lip shape image and the sample face image based on a mean absolute error formula; input the lip shape image and the sample face image into a convolutional neural network VGG to obtain a content perception loss between the lip shape image and the sample face image; input the lip shape image and the sample face image into an adversarial network to obtain an adversarial loss between the lip shape image and the sample face image; fuse the face reconstruction loss, the content perception loss and the adversarial loss to determine the first loss.

11. The apparatus of claim 10, wherein, The determination module is specifically configured to: obtain a first weight corresponding to the face reconstruction loss, a second weight corresponding to the content perception loss and a third weight corresponding to the adversarial loss; fuse the face reconstruction loss, the content perception loss and the adversarial loss based on the first weight, the second weight and the third weight to obtain the first loss.

12. The apparatus of claim 11, wherein, The fourth obtaining module is specifically configured to: obtain a fourth weight corresponding to a minimum value in the plurality of second losses; fuse the minimum value in the second losses and the first loss based on the fourth weight to obtain a target loss; correct the audio encoder, the image encoder and the generation network in the initial lip shape driving model based on the target loss to obtain the lip shape driving model.

13. A lip shape driving apparatus, comprising: a first obtaining module configured to obtain face image data to be driven, mask image data not containing a lip region corresponding to the face image data to be driven and target speech data; a second obtaining module configured to input the face image data to be driven, the mask image data and the target speech data into a lip shape driving model to obtain target lip shape image data driven by the target speech data, wherein the lip shape driving model is obtained based on the generation apparatus of the lip shape driving model in any one of claims 8-12.

14. The apparatus of claim 13, wherein, The second obtaining module is specifically configured to: input the target speech data into an audio encoder in the lip shape driving model to obtain speech features corresponding to each frame of speech data in the target speech data; fuse the speech features corresponding to each frame of speech data and target speech features corresponding to a previous frame of speech data to obtain target speech features corresponding to each frame of speech data; input the face image data to be driven and the mask image data into an image encoder in the lip shape driving model to obtain first target image features corresponding to the face image data to be driven and second target image features corresponding to the mask image data; The first target image feature, the second target image feature and the target voice feature are input into a generation network in the lip shape driving model to obtain the target lip shape image data driven by the target voice data. 15.An electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5, or perform the method of any one of claims 6-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-5, or perform the method of any one of claims 6-7. 17.A computer program product comprising computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1-5, or implement the steps of the method of any one of claims 6-7.

Images