Intelligent generation method for voice-driven anchor mouth shape and expression synchronization

Abstract

The invention discloses an intelligent generation method for voice-driven anchor mouth shape and expression synchronization, and the method comprises the steps: carrying out the processing of an anchor broadcast video, pre-training an audio-image synchronization model and an image-expression synchronization model, taking the pre-trained audio-image synchronization model and image-expression synchronization model as a generation model loss function, and carrying out the recognition of the generated model loss function. And training the mapping relationship between the voice and the image by using the generative adversarial network to obtain a generative model, inputting the audio and the reference image / video to the generative model, and generating an anchor broadcast video with mouth shape and expression synchronized with the audio. An anchor broadcast video is marked with happy, enthusiasm, excitement, sadness, anger, emergency and neutral emotions, and the mouth shape and facial expressions can be adjusted according to the emotions of audios when a generative model of an adversarial generative network is used for synchronously generating images by pre-training an image-expression synchronization model; therefore, the mouth shape synchronous video which is clear in mouth shape and capable of expressing emotions through expressions can be quickly generated.

Description

Technical Field

[0001] The present invention relates to the generation and application of digital human, and in particular to an intelligent generation method for synchronizing the lip shape and expression of a speech-driven anchor. Background Art

[0002] With the rapid development of artificial intelligence technology, speech-driven speaking digital human video generation technology has become the research frontier in the field of virtual digital humans. Speaking digital humans use input audio information and an image or video clip containing the characteristics of the target person to generate a video of the target person expressing this information naturally through information extraction, semantic expansion, fusion and alignment. The core of this technology lies in the fusion and presentation of multimodal data, aiming to present the speech content of the target person in an intuitive visual form.

[0003] At the same time, with the popularity of various social media such as Douyin, Xiaohongshu, and Bilibili, the application field of talking digital humans has been greatly expanded and deepened. These platforms provide a new stage for talking digital humans to display and interact. You only need to prepare a piece of content to be broadcast, an image or video clip containing a person, and use the digital human synthesis function to obtain a digital human broadcast video, which greatly reduces the cost of manual broadcasting and editing.

[0004] In the technology of digital human generation, voice-driven lip synchronization technology is crucial to improving the realism and naturalness of digital humans; for knowledge dissemination, using video to disseminate knowledge is more attractive to learners; among them, using professional virtual anchor images to broadcast content can shape a professional image, enhance personal brand, enhance the authority and reliability of content, and improve the dissemination effect of information. When using professional virtual anchor images to broadcast content, the existing virtual anchor's voice and lip synchronization technology is not very professional.

[0005] In order to make the pronunciation of each word clear and distinct, the anchor will pay more attention to the clarity of the mouth shape, which will make the mouth shape look more exaggerated or consciously controlled.

[0006] Because the anchor needs to communicate with the audience visually, when expressing different emotions, they will use different lip shapes and facial expressions to express happiness, enthusiasm, excitement, sadness, anger, urgency, neutrality and other emotions.

[0007] Early technologies for generating digital human speakers mainly relied on computer graphics and traditional machine learning techniques. However, the quality of videos generated by such technologies was very poor, with obvious synthetic traces.

[0008] With the development of deep learning networks, the existing voice-driven speaking digital human technologies are mainly divided into three categories: generative adversarial models, diffusion models, and neural radiation fields.

[0009] (1) Generative adversarial models, due to their unique adversarial training mechanism, have achieved remarkable results in the authenticity and diversity of digital human images, improved the realism of digital humans, and endowed them with rich facial expressions and posture changes. Representative technical implementations include styleTalker, video_retalking, and sadtalker.

[0010] (2) The diffusion model generates data by simulating the diffusion and reverse diffusion processes. It can generate high-quality and diverse digital human images. It also has good robustness and controllability. In terms of customization, it can precisely control the digital human to meet the needs of different users. Representative technical implementations include diffTalker, dreamTalker, EMO, and VASA-1.

[0011] (3) The neural radiation field method encodes and renders three-dimensional scenes through neural networks, providing a three-dimensional solution for digital human generation. Through the precise reconstruction and rendering of the three-dimensional scene of the digital human, the generated digital human has a more realistic three-dimensional feeling and stereoscopic sense. In the field of virtual reality and augmented reality, the digital human can better integrate into the virtual world and bring an immersive experience to users. In the fashion industry, virtual fitting and virtual makeup are also well applied. Representative technical implementations include AD-NeRF, GeneFace, ER-NeRF, and GaussianTalker.

[0012] Benefiting from the rapid development of deep learning-based artificial intelligence technology, current speaking digital video generation technology has made significant progress, but still faces many challenges.

[0013] 1. Improve the quality of details in the generated video; currently, the mouth shape and head movements are good, but the movements, expressions, and eyes of the characters are dull and lack of liveliness. Lip movements with audio-visual consistency are very important for speaking digital humans, but current research pays less attention to implicit features, such as eyes and emotional features.

[0014] 2. Reduce computational complexity; most current models are generated too slowly, and the diffusion model is based on frame-by-frame video generation, resulting in higher temporal complexity. Human-computer interaction virtual humans require digital humans to have multimodal information, such as natural language, facial expressions, and natural human-like gestures. At the same time, high-quality video feedback must be provided quickly when a voice request is received, so timely response is crucial to improving user experience.

[0015] 3. Build high-quality data sets. The data collected in the laboratory are limited in scale due to cost constraints, while the data sets on social media are of varying quality and lack fine-grained annotations.

[0016] 4. Add comprehensive evaluation indicators; subjective evaluation by humans will result in significant individual differences. Current objective indicators, such as PSNR and SSIM, cannot well explain human perception, and are more limited to video quality evaluation at the image level, lacking effective evaluation methods for the coherence of video content and the naturalness of human expressions and movements. Summary of the invention

[0017] In order to solve the above problems, the present invention provides an intelligent generation method for voice-driven synchronization of anchor's lip shape and expression. Starting from the perspective of data set, the method uses anchor data set, and after fine-grained screening and annotation, enriches the data set in the field of digital mouth shape synchronization, and optimizes the problem of unclear lip shape of videos generated in the anchor field; at the same time, emotional factors such as happiness, enthusiasm, excitement, sadness, anger, urgency, and neutrality are added to the broadcast video, so that when the image is synchronously generated, the lip shape and facial expression are adjusted according to the emotional phonemes of the audio; by pre-training the image-expression synchronization model, the lip shape and facial expression are adjusted according to the emotion of the audio when the image is synchronously generated using the generative model of the adversarial generative network; thereby, it is possible to quickly generate lip-synced videos with clear lip shape and the ability to express emotions through expressions.

[0018] The present invention proposes a method for intelligently generating a voice-driven anchor's lip shape and expression synchronization, comprising the following steps:

[0019] Step 1: Collect anchor broadcast videos, and pre-process the anchor broadcast videos to obtain pre-processed videos;

[0020] Step 2: Extract audio files and video frame image sequences with emotion labels from the preprocessed video obtained in step 1;

[0021] Step 3: Obtain audio features from the audio file obtained in step 2, obtain image features with emotion labels from the video frame image sequence with emotion labels obtained in step 2, and pair the audio features with the image features with emotion labels to form a synchronized audio-image pair;

[0022] Step 4: Obtain audio features from the audio file obtained in step 2, remove the part below the nose of the face from the video frame image sequence with emotion labels obtained in step 2, obtain the portrait features with emotion labels, and pair the audio features with the portrait features with emotion labels to form a training audio-image pair;

[0023] Step 5: Input the synchronized audio-image pair obtained in step 3 into the audio-image synchronization model, output the audio coding features and the image coding features, input the audio coding features and the image coding features into the audio-image synchronization loss module, calculate the output loss value through binary cross entropy loss, and iteratively optimize the weights of the audio-image synchronization model to obtain the optimized audio-image synchronization model;

[0024] Step 6: Input the synchronized audio-image pair obtained in step 3 into the image-emotion synchronization model, output the image emotion coding features, input the image emotion coding features and the corresponding emotion labels into the image-emotion synchronization loss module, calculate the output loss value through the normalization function and the multi-classification cross entropy loss function, and obtain the optimized image-emotion synchronization model by iteratively optimizing the weights of the image-emotion synchronization model;

[0025] Step 7: The audio-image synchronization model and the audio-image synchronization loss module optimized in step 5 are used as components of the loss module of the generation model, and the image-emotion synchronization model and the image-emotion synchronization loss module optimized in step 6 are used as components of the generation loss module; construct an adversarial generation network, the adversarial generation network includes a generation model and a discriminant model, the generation model includes: an audio encoder, an image encoder, an image decoder and a generation loss module, the output ends of the audio encoder and the image encoder are connected in parallel to the input end of the image decoder, and the input end of the generation loss module is connected to the output end of the image decoder; the discriminant model includes: a facial decoder and a discriminant loss module; the facial decoder and the discriminant loss module connected in sequence; the generation loss module and the discriminant loss module constitute the loss module of the adversarial generation network, the training audio-image pair obtained in step 4 is input into the adversarial generation network, and the generation model and the discriminant model in the generation network are optimized by alternating iterations to obtain the optimized generation model;

[0026] Step 8: Input audio and reference image / video to the generative model optimized in step 7 to generate a host’s announcement video with synchronized audio lip movements and facial expressions.

[0027] In step 1, the anchor's broadcast video is preprocessed, including:

[0028] First, the anchor's broadcast video is sliced, and then the frame number and audio sampling rate are adjusted. The adjusted video is labeled with emotional labels to obtain the preprocessed video.

[0029] In step 2, audio files and video frame image sequences with emotion labels are extracted from the preprocessed video obtained in step 1, specifically including:

[0030] Use the ffmpeg tool to batch extract audio files and video frame image sequences from the video obtained in the first step, and the video frame images in the video frame sequence have emotion labels.

[0031] In step 3, the image features with emotion labels are obtained from the video frame image sequence with emotion labels obtained in step 2, and the audio features are paired with the image features with emotion labels to form a synchronized audio-image pair, which specifically includes:

[0032] 3.1) Obtain 5 consecutive video frame images from the video frame image sequence with emotion labels as image features, and the image features are denoted as i s ;

[0033] 3.2) Use the Mel filter feature extraction tool to convert the audio into a feature representation in the frequency domain to obtain the audio Melp feature; obtain the Melp feature fragments corresponding to 5 consecutive video frame images as audio features, denoted as a s ;

[0034] 3.3) Change i s and a s compose synchronized audio-image pairs;

[0035] The data in this step is used to train the audio-image synchronization model and the image-emotion synchronization model;

[0036] In step 4, audio features are obtained from the audio file obtained in step 2, the part below the nose of the face is removed from the video frame image sequence with emotion labels obtained in step 2, and the portrait features with emotion labels are obtained. The audio features are paired with the portrait features with emotion labels to form a training audio-image pair, which specifically includes:

[0037] 4.1) Obtain 5 consecutive video frame images from the video frame image sequence with emotion labels, remove the part below the nose of the face, and use the 5 consecutive video frame images without the part below the nose of the face as the portrait feature, denoted as i t ;

[0038] 4.2) Use the Mel filter feature extraction tool to convert the audio into a feature representation in the frequency domain to obtain the audio Melp feature; obtain the Melp feature fragments corresponding to 5 consecutive video frame images as audio features, denoted as a t ;

[0039] 4.3) Change i t and a t compose training audio-image pairs;

[0040] The data in this step is used to train the adversarial generation network;

[0041] In step 5, the synchronized audio-image pair obtained in step 3 is input into the audio-image synchronization model, and the audio coding features and the image coding features are output. The audio coding features and the image coding features are input into the audio-image synchronization loss module, and the output loss value is calculated by binary cross entropy loss. The weights of the audio-image synchronization model are iteratively optimized to obtain the optimized audio-image synchronization model, which specifically includes:

[0042] 5.1) The audio-image synchronization model includes an audio encoder and an image encoder; s The audio coding feature is obtained through the audio encoder, denoted as a s_emb , change i s The image encoding feature is obtained through the image encoder, denoted as i s_emb ;

[0043] 5.2) For a s_emb and i s_emb The loss value is calculated by using the cosine similarity function and the binary cross entropy loss function in the audio-image synchronization loss module, denoted as L1 s ;

[0044] 5.3) By minimizing L1 s The weights of the audio-image synchronization model are iteratively optimized to obtain an optimized audio-image synchronization model.

[0045] In step 6, the synchronized audio-image pair obtained in step 3 is input into the image-emotion synchronization model, and the image emotion coding features are output. The image emotion coding features and the corresponding emotion labels are input into the image-emotion synchronization loss module, and the output loss value is calculated by the normalization function and the multi-classification cross entropy loss function. The weights of the image-emotion synchronization model are iteratively optimized to obtain the optimized image-emotion synchronization model, which specifically includes:

[0046] 6.1) The image-emotion synchronization model consists of a multi-layer convolutional network. s The image emotion encoding features are obtained through the image-emotion synchronization model;

[0047] 6.2) The output loss value of the image emotion encoding feature is calculated by the normalized multi-classification cross entropy loss function in the audio-image synchronization loss module, denoted as L2 s .

[0048] 6.3) By minimizing L2 s The weights of the audio-image synchronization model are iteratively optimized to obtain an optimized image-emotion synchronization model.

[0049] In step 7, the audio-image synchronization model and the audio-image synchronization loss module optimized in step 5 are used as components of the loss module of the generation model, and the image-emotion synchronization model and the image-emotion synchronization loss module optimized in step 6 are used as components of the generation loss module; a generative adversarial network is constructed, wherein the generative adversarial network comprises a generative model and a discriminative model, wherein the generative model comprises: an audio encoder, an image encoder, an image decoder and a generative loss module, wherein the output ends of the audio encoder and the image encoder are connected in parallel to the input end of the image decoder, and the input end of the generative loss module is connected to the output end of the image decoder; the discriminative model comprises: a facial decoder and a discriminative loss module; a facial decoder and a discriminative loss module connected in sequence; the generative loss module and the discriminative loss module constitute the loss module of the generative adversarial network, and the training audio-image pair obtained in step 4 is input into the generative adversarial network, and the generative model and the discriminative model in the generative network are optimized by alternate iteration to obtain the optimized generative model, which specifically comprises:

[0050] 7.1) Generative model construction, denoted as G, specifically:

[0051] The audio encoder and image encoder in the generative model are respectively down-sampled by convolutional neural networks to obtain audio coding features and image coding features; the image decoder converts the audio coding features and image coding features into image features through up-sampling by convolutional neural networks. The image features are activated by the Sigmoid function to obtain an array value with the output value limited to the range of [0,1]. The value is scaled to a pixel value suitable for the image by multiplying it by 255, and the image composed of pixel values ​​is used as the final generated image.

[0052] 7.1.1) Change i t Input into the audio encoder in the adversarial generation network to obtain the audio coding features, denoted as i t_emb ; t Input into the image encoder in the adversarial generation network to obtain the image encoding feature, denoted as a t_em b ;

[0053] 7.1.2) for i t_emb and a t_emb The concatenation is performed and the generated image is obtained by passing it through the image decoder, denoted as g;

[0054] 7.1.3) For g and a t The audio-image synchronization loss value is obtained through the audio-image synchronization model and the audio-image synchronization loss module, which is recorded as L1;

[0055] 7.1.4) Obtain the image-emotion synchronization loss value of g and the emotion label through the image-emotion synchronization model and the image-emotion synchronization loss module, denoted as L2;

[0056] 7.1.5) Apply L1 norm function to g and video frame image to obtain L1 norm loss, denoted as L3;

[0057] 7.2) Discriminant model construction, denoted as D, specifically:

[0058] 7.2.1) The facial feature encoding of g and the real video frame image is output by the facial decoder in the discriminative model, and the probability value of the image being true / false is obtained by the sigmoid function, which is recorded as p i ;

[0059] 7.2.2) For p i The discriminant loss value is obtained by calculating the difference between the probability value and the true label through the binary cross entropy loss in the discriminant loss module, which is recorded as L4;

[0060]

[0061] y i is the true label of the i-th sample, which takes a value of 0 or 1. It is 0 when g is input and 1 when a real video frame image is input;

[0062] log represents the logarithmic function;

[0063] N represents the number of training data;

[0064] 7.3) The final loss function is denoted as L, and the formula is as follows:

[0065]

[0066] It means optimizing the discriminant network by maximizing the loss value. Represents optimizing the generated network by minimizing the loss value.

[0067] 7.4) By alternately training the generator and the discriminator, the final optimized generation model is obtained.

[0068] In step 8, the optimized generative model from step 7 is input with audio and reference image / video to generate a host’s announcement video with audio-synchronized lip movements and facial expressions, including:

[0069] 8.1) Input the audio and reference video or image into the generative model to generate images with lip movements and expressions;

[0070] 8.2) Paste the images with lip shapes and expressions back to the input video or image frame by frame, and write them into a video frame by frame;

[0071] 8.3) Use ffmpeg to fuse the generated video and input speech to generate the final video with speech;

[0072] Compared with the prior art, the present invention has the following advantages:

[0073] The present invention realizes a method for synchronously generating speech-driven lip shape and expression for a host, and the difference from the existing method is that:

[0074] 1. Use fine-grained screened anchor broadcast videos as training data to make the digital human mouth shape in the generated video clearer; however, the existing data lacks fine-grained annotations, resulting in unclear digital human mouth shape.

[0075] 2. Annotate the emotions expressed in the anchor’s broadcast video, such as happiness, enthusiasm, excitement, sadness, anger, urgency, and neutrality. Add an image-emotion loss module to the loss module of the generative adversarial network, so that the generative model can adjust the lip shape and facial expressions according to the emotional characteristics of the audio so that the generated image can express emotions through facial expressions while synchronizing the lip shape.

[0076] 3. Use voice to drive lip sync and expression synchronization, while most current models can only drive lip sync. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of an audio-image synchronization model training process of an intelligent generation method for voice-driven host lip shape and expression synchronization of the present invention;

[0078] Figure 2 A schematic diagram of an image-emotion synchronization model training process of an intelligent generation method for speech-driven anchor lip shape and expression synchronization of the present invention;

[0079] Figure 3 A schematic diagram of a training process of a generative adversarial network model of an intelligent generation method for synchronizing the lip shape and facial expression of a voice-driven anchor in the present invention;

[0080] Figure 4 A schematic diagram of a loss module of a generative adversarial network of an intelligent generation method for synchronizing the lip shape and facial expression of a voice-driven anchor in the present invention;

[0081] Figure 5 A schematic diagram showing a comparison between an image generated by a method for intelligently generating a speech-driven anchor's lip shape and facial expression synchronization according to the present invention and an image generated on the market; DETAILED DESCRIPTION

[0082] The following is a further explanation of a method for intelligently generating a voice-driven synchronization of a host's lip shape and facial expressions in conjunction with the accompanying drawings.

[0083] A method for intelligently generating speech-driven anchor lip synchronization comprises: preprocessing anchor broadcasting videos; pre-training audio-image synchronization models and image-emotion synchronization models; constructing a generative adversarial network, defining a loss function of a generative model in the generative adversarial network, and obtaining a final generative model by alternately training a generative model and a discriminative model in the generative adversarial network; and inputting audio and reference images / videos into the generative model to obtain an anchor broadcasting video with audio-synchronized lip sync and facial expressions.

[0084] The anchor's broadcast video is preprocessed to prepare synchronized audio-image pairs as input data for the audio-image synchronization model and the image-emotion synchronization model. The training audio-image pairs are prepared as input data for the adversarial generation network.

[0085] The trained audio-image synchronization model and image-emotion synchronization model are used as components of the generative model loss module.

[0086] The loss module of the generative model includes: using the pre-trained audio-image synchronization model and the audio-image synchronization loss module to calculate the loss value between the original audio features and the generated image features; using the pre-trained image-emotion synchronization model and the image-emotion synchronization loss module to calculate the loss value between the emotion encoding features in the generated image and the real emotion label; using the fixed-weight discriminant model and the discriminant model loss module to calculate the loss value of the generated image being discriminated as a real image; using the L1 norm function to calculate the loss value of the generated image and the real image.

[0087] The generation process of the host's voice-driven lip shape and expression synchronization includes:

[0088] Step 1: Collect anchor broadcast videos, and pre-process the anchor broadcast videos to obtain pre-processed videos;

[0089] Step 2: Extract audio files and video frame image sequences with emotion labels from the preprocessed video obtained in step 1;

[0090] Step 3: Obtain audio features from the audio file obtained in step 2, obtain image features with emotion labels from the video frame image sequence with emotion labels obtained in step 2, and pair the audio features with the image features with emotion labels to form a synchronized audio-image pair;

[0091] Step 4: Obtain audio features from the audio file obtained in step 2, remove the part below the nose of the face from the video frame image sequence with emotion labels obtained in step 2, obtain the portrait features with emotion labels, and pair the audio features with the portrait features with emotion labels to form a training audio-image pair;

[0092] Step 5: Input the synchronized audio-image pair obtained in step 3 into the audio-image synchronization model, output the audio coding features and the image coding features, input the audio coding features and the image coding features into the audio-image synchronization loss module, calculate the output loss value through binary cross entropy loss, and iteratively optimize the weights of the audio-image synchronization model to obtain the optimized audio-image synchronization model;

[0093] Step 6: Input the synchronized audio-image pair obtained in step 3 into the image-emotion synchronization model, output the image emotion coding features, input the image emotion coding features and the corresponding emotion labels into the image-emotion synchronization loss module, calculate the output loss value through the normalization function and the multi-classification cross entropy loss function, and obtain the optimized image-emotion synchronization model by iteratively optimizing the weights of the image-emotion synchronization model;

[0094] Step 7: The audio-image synchronization model and the audio-image synchronization loss module optimized in step 5 are used as components of the loss module of the generation model, and the image-emotion synchronization model and the image-emotion synchronization loss module optimized in step 6 are used as components of the generation loss module; construct an adversarial generation network, the adversarial generation network includes a generation model and a discriminant model, the generation model includes: an audio encoder, an image encoder, an image decoder and a generation loss module, the output ends of the audio encoder and the image encoder are connected in parallel to the input end of the image decoder, and the input end of the generation loss module is connected to the output end of the image decoder; the discriminant model includes: a facial decoder and a discriminant loss module; the facial decoder and the discriminant loss module connected in sequence; the generation loss module and the discriminant loss module constitute the loss module of the adversarial generation network, the training audio-image pair obtained in step 4 is input into the adversarial generation network, and the generation model and the discriminant model in the generation network are optimized by alternating iterations to obtain the optimized generation model;

[0095] Step 8: Input audio and reference image / video to the generative model optimized in step 7 to generate a host’s announcement video with synchronized audio lip movements and facial expressions.

[0096] In step 1, the anchor's broadcast video is preprocessed, including:

[0097] First, the anchor's broadcast video is sliced, and then the frame number and audio sampling rate are adjusted. The adjusted video is labeled with emotional labels to obtain the preprocessed video.

[0098] In step 2, audio files and video frame image sequences with emotion labels are extracted from the preprocessed video obtained in step 1, specifically including:

[0099] Use the ffmpeg tool to batch extract audio files and video frame image sequences from the video obtained in the first step, and the video frame images in the video frame sequence have emotion labels.

[0100] In step 3, the image features with emotion labels are obtained from the video frame image sequence with emotion labels obtained in step 2, and the audio features are paired with the image features with emotion labels to form a synchronized audio-image pair, which specifically includes:

[0101] 3.1) Obtain 5 consecutive video frame images from the video frame image sequence with emotion labels as image features, and the image features are denoted as i s ;

[0102] 3.2) Use the Mel filter feature extraction tool to convert the audio into a feature representation in the frequency domain to obtain the audio Melp feature; obtain the Melp feature fragments corresponding to 5 consecutive video frame images as audio features, denoted as a s ;

[0103] 3.3) Change i s and a s compose synchronized audio-image pairs;

[0104] The data in this step is used to train the audio-image synchronization model and the image-emotion synchronization model;

[0105] In step 4, audio features are obtained from the audio file obtained in step 2, the part below the nose of the face is removed from the video frame image sequence with emotion labels obtained in step 2, and the portrait features with emotion labels are obtained. The audio features are paired with the portrait features with emotion labels to form a training audio-image pair, which specifically includes:

[0106] 4.1) Obtain 5 consecutive video frame images from the video frame image sequence with emotion labels, remove the part below the nose of the face, and use the 5 consecutive video frame images without the part below the nose of the face as the portrait feature, denoted as i t ;

[0107] 4.2) Use the Mel filter feature extraction tool to convert the audio into a feature representation in the frequency domain to obtain the audio Melp feature; obtain the Melp feature fragments corresponding to 5 consecutive video frame images as audio features, denoted as a t ;

[0108] 4.3) Change i t and a t compose training audio-image pairs;

[0109] The data in this step is used to train the adversarial generation network;

[0110] In step 5, if Figure 1 As shown in the diagram of the training process of the audio-image synchronization model, the training inputs the synchronized audio-image pair obtained in step 3 into the audio-image synchronization model, outputs the audio coding features and the image coding features, inputs the audio coding features and the image coding features into the audio-image synchronization loss module, calculates the output loss value through the binary cross entropy loss, and obtains the optimized audio-image synchronization model by iteratively optimizing the weights of the audio-image synchronization model, which specifically includes:

[0111] 5.1) The audio-image synchronization model includes an audio encoder and an image encoder; s The audio coding feature is obtained through the audio encoder, denoted as a s_emb , change i s The image encoding feature is obtained through the image encoder, denoted as i s_emb ;

[0112] 5.2) For a s_emb and i s_emb The loss value is calculated by using the cosine similarity function and the binary cross entropy loss function in the audio-image synchronization loss module, denoted as L1 s ;

[0113] 5.3) By minimizing L1 s The weights of the audio-image synchronization model are iteratively optimized to obtain an optimized audio-image synchronization model.

[0114] In step 6, if Figure 2 As shown in the diagram of the image-emotion synchronization model training process, the image features in the synchronized audio-image pair obtained in step 3 are input into the image-emotion synchronization model, and the image emotion coding features are output. The image emotion coding features and the corresponding emotion labels are input into the image-emotion synchronization loss module, and the output loss value is calculated by the normalization function and the multi-classification cross entropy loss function. The weights of the image-emotion synchronization model are iteratively optimized to obtain the optimized image-emotion synchronization model, which specifically includes:

[0115] 6.1) The image-emotion synchronization model consists of a multi-layer convolutional network. s The image emotion encoding features are obtained through the image-emotion synchronization model;

[0116] 6.2) The output loss value of the image emotion encoding feature is calculated by the normalized multi-classification cross entropy loss function in the audio-image synchronization loss module, denoted as L2 s .

[0117] 6.3) By minimizing L2 s The weights of the audio-image synchronization model are iteratively optimized to obtain an optimized image-emotion synchronization model.

[0118] In step 7, the audio-image synchronization model and the audio-image synchronization loss module optimized in step 5 are used as components of the loss module of the generation model, and the image-emotion synchronization model and the image-emotion synchronization loss module optimized in step 6 are used as components of the generation loss module; a generative adversarial network is constructed, wherein the generative adversarial network comprises a generative model and a discriminative model, wherein the generative model comprises: an audio encoder, an image encoder, an image decoder and a generative loss module, wherein the output ends of the audio encoder and the image encoder are connected in parallel to the input end of the image decoder, and the input end of the generative loss module is connected to the output end of the image decoder; the discriminative model comprises: a facial decoder and a discriminative loss module; a facial decoder and a discriminative loss module connected in sequence; the generative loss module and the discriminative loss module constitute the loss module of the generative adversarial network, and the training audio-image pair obtained in step 4 is input into the generative adversarial network, and the generative model and the discriminative model in the generative network are optimized by alternate iteration to obtain the optimized generative model, which specifically comprises:

[0119] 7.1) Generative model construction, denoted as G, specifically:

[0120] The audio encoder and image encoder in the generative model are respectively down-sampled by convolutional neural networks to obtain audio coding features and image coding features; the image decoder converts the audio coding features and image coding features into image features through up-sampling by convolutional neural networks. The image features are activated by the Sigmoid function to obtain an array value with the output value limited to the range of [0,1]. The value is scaled to a pixel value suitable for the image by multiplying it by 255, and the image composed of pixel values ​​is used as the final generated image.

[0121] 7.1.1) Change i t Input into the audio encoder in the adversarial generation network to obtain the audio coding features, denoted as i t_emb ; t Input into the image encoder in the adversarial generation network to obtain the image encoding feature, denoted as a t_em b ;

[0122] 7.1.2) for i t_emb and a t_emb The concatenation is performed and the generated image is obtained by passing it through the image decoder, denoted as g;

[0123] 7.1.3) If Figure 4 As shown in the generative model loss module in the adversarial generative network, for g and a t The audio-image synchronization loss value is obtained through the audio-image synchronization model and the audio-image synchronization loss module, which is recorded as L1;

[0124] 7.1.4) Obtain the image-emotion synchronization loss value of g and the emotion label through the image-emotion synchronization model and the image-emotion synchronization loss module, denoted as L2;

[0125] 7.1.5) Apply L1 norm function to g and video frame image to obtain L1 norm loss, denoted as L3;

[0126] 7.2) Discriminant model construction, denoted as D, specifically:

[0127] 7.2.1) The facial feature encoding of g and the real video frame image is output by the facial decoder in the discriminative model, and the probability value of the image being true / false is obtained by the sigmoid function, which is recorded as p i ;

[0128] 7.2.2) For p i The discriminant loss value is obtained by calculating the difference between the probability value and the true label through the binary cross entropy loss in the discriminant loss module, which is recorded as L4;

[0129]

[0130] y i is the true label of the i-th sample, which takes a value of 0 or 1. It is 0 when g is input and 1 when a real video frame image is input;

[0131] log represents the logarithmic function;

[0132] N represents the number of training data;

[0133] 7.3) The final loss function is denoted as L, and the formula is as follows:

[0134]

[0135] It means optimizing the discriminant network by maximizing the loss value. Represents optimizing the generated network by minimizing the loss value.

[0136] 7.4) By alternately training the generator and the discriminator, the final optimized generation model is obtained.

[0137] In step 8, the optimized generative model from step 7 is input with audio and reference image / video to generate a host’s announcement video with audio-synchronized lip movements and facial expressions, including:

[0138] 8.1) Input the audio and reference video or image into the generative model to generate images with lip movements and expressions;

[0139] 8.2) Paste the images with lip shapes and expressions back to the input video or image frame by frame, and write them into a video frame by frame;

[0140] 8.3) Use ffmpeg to fuse the generated video and input speech to generate the final video with speech;

[0141] Specific as Figure 5 Comparison between the generated images of the present invention and the generated images on the market shows that the generated images of the present invention have obvious expressions;

[0142] The above is only an illustration of the present invention, and is not intended to limit the present invention. A person skilled in the art should recognize that any changes or modifications made to the present invention will fall within the protection scope of the present invention.

Claims

1. A method for intelligently generating a voice-driven anchor's lip shape and facial expression synchronization, characterized in that: The following steps are involved: Step 1: Collect anchor broadcast videos, and pre-process the anchor broadcast videos to obtain pre-processed videos; Step 2: Extract audio files and video frame image sequences with emotion labels from the preprocessed video obtained in step 1; Step 3: Obtain audio features from the audio file obtained in step 2, obtain image features with emotion labels from the video frame image sequence with emotion labels obtained in step 2, and pair the audio features with the image features with emotion labels to form a synchronized audio-image pair; Step 4: Obtain audio features from the audio file obtained in step 2, remove the part below the nose of the face from the video frame image sequence with emotion labels obtained in step 2, obtain the portrait features with emotion labels, and pair the audio features with the portrait features with emotion labels to form a training audio-image pair; Step 5: Input the synchronized audio-image pair obtained in step 3 into the audio-image synchronization model, output the audio coding features and image coding features, calculate the output loss value of the audio coding features and image coding features through the cosine similarity function and the binary cross entropy loss function, and obtain the optimized audio-image synchronization model by iteratively optimizing the weights of the audio-image synchronization model; Step 6: Input the synchronized audio-image pair obtained in step 3 into the image-emotion synchronization model, output the image emotion coding features, calculate the output loss value of the image emotion coding features and emotion labels through the normalization function and the multi-classification cross entropy loss function, and obtain the optimized image-emotion synchronization model by iteratively optimizing the weights of the image-emotion synchronization model; Step 7: constructing a generative adversarial network using the optimized audio-image synchronization model obtained in step 5 and the optimized image-emotion synchronization model obtained in step 6, wherein the generative adversarial network includes a generative model and a discriminative model, inputting the training audio-image pairs obtained in step 4 into the generative adversarial network, and optimizing the generative model and the discriminative model in the generative network by alternating iterations to obtain an optimized generative model; Step 8: Input audio and reference image / video into the optimized generative model obtained in step 7 to generate a host broadcast video with synchronized audio lip movements and facial expressions.

2. The method for intelligently generating the synchronization of the host's lip shape and facial expression driven by voice according to claim 1, characterized in that: In step 1, the anchor's broadcast video is preprocessed, including: First, the anchor's broadcast video is sliced, and then the frame number and audio sampling rate are adjusted. The adjusted video is labeled with emotional labels to obtain the preprocessed video.

3. The method for intelligently generating the synchronization of the host's lip shape and facial expression driven by voice according to claim 1, characterized in that: In step 2, audio files and video frame image sequences with emotion labels are extracted from the preprocessed video obtained in step 1, specifically including: Use the ffmpeg tool to batch extract audio files and video frame image sequences from the video obtained in step 1, use digital coding to add emotion labels to the video frame image sequences, and obtain video frame image sequences with emotion labels.

4. The method for intelligently generating the synchronization of the host's lip shape and facial expression driven by voice according to claim 1, characterized in that: In step 7, the generation model includes: an audio encoder, an image encoder, an image decoder and a generation loss module. The output ends of the audio encoder and the image encoder are connected in parallel to the input end of the image decoder, the input end of the generation loss module is connected to the output end of the image decoder, and the generation loss module adopts the optimized audio-image synchronization model obtained in step 5, the optimized image-emotion synchronization model obtained in step 6, and the L1 norm function.

5. The method for intelligently generating speech-driven anchor lip shape and expression synchronization according to claim 1, characterized in that: In step 7, the discriminant model includes: a facial decoder and a discriminant loss module connected in sequence, and the discriminant loss module adopts a binary cross entropy loss function.

Images