Lip shape synthesis model design method and device, storage medium and program product

Abstract

The invention provides a lip shape synthesis model design method and device, a storage medium and a program product, and relates to the technical field of computer vision. The lip shape synthesis model design method comprises the following steps: collecting and preprocessing data to obtain an original facial image, an original emotion label and an audio feature; using a generator to obtain a recovered face image and a recovered emotion label; performing confrontation training on the original facial image and the recovered facial image by using a discriminator; performing reverse iteration by using the picture reconstruction loss function, the perception loss function, the adversarial loss function and the emotional loss function; and performing model fine tuning training on the to-be-generated video. According to the method, the overall accuracy and the image quality of lip-shaped composite image generation are improved, and related generation flaws and errors are avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lip shape synthesis model design method, device, storage medium and program product. Background Art

[0002] Digital human video synthesis technology refers to the technology of using digital humans to reproduce the target person's video. Specifically, it is the process of synthesizing the target person's video synchronized with the audio based on a piece of audio and the target person's visual information (such as images or videos). To judge the quality of the synthesized digital human, the most basic indicator is the matching accuracy of the synthesized digital human's lip shape with the audio.

[0003] Traditional digital human image generation models often convert the input audio code into acoustic features first, and then use the acoustic features to restore the facial image, such as the Wav2Lip model. The Chinese patent (CN 113192161A) published on July 30, 2021 matches the character's movements through text content or voice, and then obtains a virtual portrait video. This synthesis method is completely driven by voice, but the accuracy of obtaining emotional information from voice is often not as high as the accuracy of facial expression recognition. Therefore, it is difficult for the synthesized video to accurately reflect the emotional expression of the character.

[0004] The Chinese patent (CN 114245215 A) published on March 25, 2022, uses a neural rendering model to directly reconstruct a face model based on emotion tags and audio features. This method directly presets the emotion type, and the synthesis result is not realistic or natural enough. It is also difficult to handle more complex facial emotion features. For example, a news anchor smiles due to professional habit, but his voice does not contain a smile. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a lip shape synthesis model design method, device, storage medium and program product. By adding an emotion coding module, the emotion information of facial recognition is introduced into the training, and the emotion information is used as an independent control factor to control the synthesis effect. While better integrating the emotion characteristics, the accuracy of lip shape synthesis is improved.

[0006] In a first aspect, the present invention provides a method for designing a lip shape synthesis model, comprising the following steps:

[0007] Data collection and preprocessing to obtain original facial images, original emotion labels and audio features;

[0008] Use the generator to obtain restored facial images and restored emotion labels;

[0009] Use the discriminator to conduct adversarial training on the original face image and the restored face image;

[0010] Use image reconstruction loss function, perception loss function, adversarial loss function, and emotion loss function for reverse iteration.

[0011] Fine-tune the model for the video to be generated.

[0012] As a further improvement of the present invention, the data collection and preprocessing includes:

[0013] Collect video samples and classify them by speaker;

[0014] Extract pictures from the video frame by frame, and obtain the original facial image fig through automatic recognition and cropping;

[0015] Performing emotion recognition on the original facial image to obtain an original emotion label emo;

[0016] Through the ASR model, the audio hidden features in the sample video are extracted to obtain the audio features audio corresponding to each original facial image.

[0017] As a further improvement of the present invention, the method of using a generator to obtain a restored facial image and restore an emotion label includes:

[0018] Mask the face part below the nose of the original face image to obtain the face masked image fig_masked;

[0019] The facial mask image fig_masked is converted into encoding information, and then the restored facial image fig_generated with the masked part completed and the restored emotion label emo_generated are obtained according to the encoding information.

[0020] As a further improvement of the present invention, the facial mask image is converted into coding information, and then a restored facial image with the masked part completed and a restored emotion label are obtained according to the coding information, including:

[0021] Use a deep neural network to process the facial mask image fig_masked and obtain the image encoding F1.

[0022] Use the emotion encoding module to process the original emotion label emo and obtain the emotion code F2.

[0023] The audio feature audio is processed using the audio encoding module to obtain the audio code F3.

[0024] The feature fusion module and the feature decoding module are used to process the above F1, F2, and F3 in turn to obtain the restored facial image fig_generated with the masked part completed.

[0025] The restored emotion label emo_generated is obtained according to the restored facial image fig_generated.

[0026] As a further improvement of the present invention, the reverse iteration using the image reconstruction loss function, the perception loss function, the adversarial loss function, and the emotion loss function includes:

[0027] Use the image reconstruction loss function to measure the difference in pixel values ​​between the original face image and the restored face image;

[0028] The perceptual loss function is used to measure the difference in feature representation between the original face image and the restored face image;

[0029] Use adversarial loss function as convergence constraint for adversarial training;

[0030] The emotion loss function is used to measure the difference in emotional feature distribution between the original face image and the restored face image.

[0031] As a further improvement of the present invention, the model fine-tuning training for the video to be generated includes:

[0032] Collect the speaking videos of the characters in the video to be generated as fine-tuning training videos;

[0033] Performing the preprocessing on the fine-tuning training video, and then using the generator and the discriminator to perform adversarial training;

[0034] In adversarial training, the number of training steps is set and the step size is adjusted.

[0035] In a second aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0036] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0037] In a fourth aspect, the present invention provides a computer program product, which implements the steps of the method described in the first aspect when the computer program is executed by a processor.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention proposes a lip shape synthesis method, device, storage medium and program product using Gaussian blur and deep learning, which extracts mask images of various limbs of the human body, blurs them, and then obtains corresponding limb images through calculation. On this basis, the lip shape synthesis task is completed, thereby improving the overall accuracy and image quality of the generated image and avoiding the occurrence of related generation defects and errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention discloses a flow chart of a method for designing a lip shape synthesis model. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S1, S2... in the embodiments described in the present invention do not limit the only execution steps of the present invention; the various models, simulation environments, and software described in the present invention are not the only limiting methods of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] In the present invention, computer devices / equipment / systems refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, a server can also be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0044] In a first aspect, the present invention provides an embodiment of a lip shape synthesis model design method, such as Figure 1 As shown, the specific process can be as follows:

[0045] S1: Data collection and preprocessing.

[0046] Preferably, the method specifically comprises the following steps:

[0047] S11: Video sample collection and classification. Collect 50,000 to 100,000 hours of multi-person video samples. The number of people involved in the video should be ≥ 500,000, and the video duration of each person should be 5 to 10 minutes. The video types cover various scenarios such as live broadcast, news, short videos, lectures, etc., and the content discussed in the video should cover various fields such as education, finance, industry, etc. Classify the videos by speaker.

[0048] S12: Image extraction and cropping. Extract images from the video frame by frame, and then crop the facial area of ​​each frame through mediepipe to obtain the original facial image fig in each frame.

[0049] S13: Image emotion recognition. Use deepface to perform emotion recognition on the original facial image and obtain the corresponding original emotion label emo. The original emotions include: neutral, happy, angry, sad, etc.

[0050] S14: Audio feature extraction. Audio is extracted from the video data through ASR models such as whisper, and the latent features extracted from its encoder are used as acoustic features. The audio features corresponding to each original facial image are obtained.

[0051] After processing from S12 to S14, a data pair is obtained for each frame of a complete video sample: (fig, emo, audio).

[0052] S2: Use the generator G to process the data pair (fig, emo, audio) obtained in step S1 to obtain the restored facial image fig_generated and the restored emotion label emo_generated.

[0053] Preferably, the processing of the generator comprises the following steps:

[0054] S21: Lip masking: Using the coordinates calculated by mediapipe for the original facial image, the facial part below the nose is appropriately masked (i.e., filled with all black pixels), to obtain a facial mask image fig_masked.

[0055] S22: converting the facial mask image fig_masked into encoding information, and then obtaining the restored facial image fig_generated and the restored emotion label emo_generated according to the encoding information.

[0056] Preferably, this step specifically includes:

[0057] S221: Use a deep neural network to process fig_masked and obtain the image code F1.

[0058] S222: Use the emotion encoding module to process the original emotion label emo to obtain the emotion code F2.

[0059] S223: Use the audio encoding module to process the audio feature audio to obtain the audio code F3.

[0060] S224: Use the feature fusion module and the feature decoding module to process the above F1, F2, and F3 in sequence to obtain a restored facial image fig_generated with the masked part completed.

[0061] S225: Use deepface or other emotion recognition models to process the restored facial image fig_generated to obtain the restored emotion label emo_generated.

[0062] S3: Use the discriminator D to perform adversarial training on the original facial image fig and the restored facial image fig_generated.

[0063] S4: Backward iteration using loss function.

[0064] The total loss function is loss_all:

[0065] loss_all=loss_recon+loss_perceptual+loss_gan+loss_emo;

[0066] in,

[0067] 1. loss_recon is the image reconstruction loss function;

[0068] Preferably, the reconstruction loss function uses the MAE (mean Absolute Error) function:

[0069] loss_recon(fig,fig_generated)=mean(|x i -y i |),

[0070] Among them, mean represents the average value, x i Represents the RGB value of a single pixel in fig, y i Represents the RGB value of a single pixel in fig_generated.

[0071] 2. loss_perceptual is the perceptual loss function; preferably, the following function is used:

[0072]

[0073] Among them, h represents the height of the convolution layer, w represents the width of the convolution layer, and c represents the number of channels of the convolution layer. represents the perception layer of the deep neural network, l represents The first layer of the perception layer. represents the restored facial image fig_generated, and I represents the original facial image fig.

[0074] The perceptual loss function performs comparative calculations on the high-dimensional feature representation of the perceptual layer, and its constraint goal is to make the features of the restored facial image fig_generated and the original facial image fig in the high-dimensional perceptual layer similar.

[0075] 3. loss_gan is the adversarial loss function. Preferably, the following function is used:

[0076] loss_gan=L G +L D ,

[0077]

[0078] Among them, E represents expectation, D represents discriminator, G represents generator, and x represents the original facial image fig. Indicates the restored facial image fig_generated, P data represents the data distribution of the original facial image fig, P g Represents the data distribution of the restored facial image fig_generated.

[0079] 4. loss_emo is the emotion loss function, preferably, the cross entropy function is used.

[0080] S5: Perform model fine-tuning training on the video to be generated.

[0081] Preferably, the specific steps include:

[0082] S51: Collect a 5-minute speaking video of the person in the video to be generated as a fine-tuning training video;

[0083] S52: preprocess the fine-tuning training video using the method described in S1;

[0084] S53: When performing adversarial training using the method described in S2 to S4, the number of training steps is set and the step size is adjusted.

[0085] In one embodiment of the present invention, the number of fine-tuning training steps is set to 5000-10000 (one generation, discrimination and reverse iteration using steps S2-S4 is counted as one step).

[0086] In one embodiment of the present invention, the step size of fine-tuning training is controlled by setting a smaller learning rate: during model pre-training, the learning rate lr is set to 1.0×e -4 , set the learning rate lr=1.0×e during fine-tuning training -5 .

[0087] Fine-tuning training is essentially transfer learning of the pre-trained model. By setting the number of steps and adjusting the step size in fine-tuning training, the fine-tuning process only makes slight changes to the parameters of the pre-trained model, that is, based on the pre-trained model, it adapts to the facial data of the person to be generated.

[0088] S54: Use the fine-tuned model to perform digital human video synthesis. During synthesis, provide the base video to be synthesized, the audio of the corresponding duration, and specify when to insert what emotion (by default, the emotion of each frame is preset to be neutral), and then synthesize the digital human video of the corresponding duration.

[0089] In a second aspect, the present invention provides an embodiment of a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0090] In a third aspect, the present invention provides an embodiment of a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0091] In a fourth aspect, the present invention provides an embodiment of a computer program product, which implements the steps of the method described in the first aspect when the computer program is executed by a processor.

[0092] The present invention proposes a lip shape synthesis model design method, device, storage medium and program product. By adding an emotion coding module, the emotion information of facial recognition is introduced into the training, and the emotion information is used as an independent control factor to control the synthesis effect. While better integrating the emotion characteristics, the accuracy of lip shape synthesis is improved.

Claims

1. A lip shape synthesis model design method, characterized in that: The following steps are involved: Data collection and preprocessing to obtain original facial images, original emotion labels and audio features; Use the generator to obtain restored facial images and restored emotion labels; Use the discriminator to conduct adversarial training on the original face image and the restored face image; Use image reconstruction loss function, perception loss function, adversarial loss function, and emotion loss function for reverse iteration; Fine-tune the model for the video to be generated.

2. The method according to claim 1, characterized in that: The data collection and preprocessing include: Collect video samples and classify them by speaker; Extract pictures from the video frame by frame, and obtain the original facial image through automatic recognition and cropping; Performing emotion recognition on the original facial image to obtain an original emotion label; Through the ASR model, the audio latent features in the sample video are extracted to obtain the audio features corresponding to each original facial image.

3. The method according to claim 1, characterized in that: The method of using the generator to obtain a restored facial image and a restored emotion label includes: The facial part below the nose of the original facial image is covered to obtain a facial cover image; The facial mask image is converted into encoding information, and then the restored facial image with the masked part completed and the restored emotion label are obtained based on the encoding information.

4. The method according to claim 3, characterized in that The step of converting the facial mask image into coding information and obtaining a restored facial image with the masked portion completed and a restored emotion label according to the coding information includes: Use a deep neural network to process the facial mask image and obtain the image encoding; Use the emotion encoding module to process the original emotion label to obtain the emotion encoding; Use the audio encoding module to process audio features and obtain audio encoding; The image encoding, the emotion encoding, and the audio encoding are processed in sequence using a feature fusion module and a feature decoding module to obtain a restored facial image that completes the covered portion; The restored emotion label is obtained based on the restored facial image.

5. The method according to claim 1, characterized in that The reverse iteration using the image reconstruction loss function, the perception loss function, the adversarial loss function, and the emotion loss function includes: Use the image reconstruction loss function to measure the difference in pixel values ​​between the original face image and the restored face image; The perceptual loss function is used to measure the difference in feature representation between the original face image and the restored face image; Use adversarial loss function as convergence constraint for adversarial training; The emotion loss function is used to measure the difference in emotional feature distribution between the original face image and the restored face image.

6. The method according to claim 1, characterized in that The model fine-tuning training for the video to be generated includes: Collect the speaking videos of the characters in the video to be generated as fine-tuning training videos; Performing the preprocessing on the fine-tuning training video, and then using the generator and the discriminator to perform adversarial training; In adversarial training, the number of training steps is set and the step size is adjusted.

7. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images