Speaking face video generation method and device for standardizing mouth shape, equipment and medium

Abstract

The invention belongs to the technical field of audio-video generation, and discloses a mouth shape-standardized speaking face video generation method, device and equipment and a medium, and the method comprises the steps: generating a heat map feature of an audio according to a text feature, an amplitude feature and a first speaking mouth shape key point feature; sampling the heat map features to obtain a first feature vector; extracting an audio clip of the audio, and acquiring a second feature vector by capturing a time sequence feature of the audio clip; inputting the image sequence of the speaking face into a video encoder to obtain a third feature vector of the image; superposing the third feature vector by using the second feature vector to obtain a fourth feature vector; and decoding the first feature vector and the fourth feature vector by using a video decoder to generate a face video corresponding to the audio. By implementing the method and the device, a more standard speaking mouth shape face video is generated, the ornamental value of the speaking mouth shape face video is improved, and the acceptability of a user to the video is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of audio-video generation, and in particular to a method, device, equipment and medium for generating a speaking face video with standardized lip shape. Background Art

[0002] At present, both in the field of academic research and industrial application, audio-driven methods for generating talking faces or head videos assume that the input audio is pure, that is, free of various noises and has a neutral tone or timbre. However, speech contains rich personalized information and is varied in style. The same word spoken by different people in different contexts may have different durations, amplitudes, and tones.

[0003] The performance of most existing speech face or head video generation technology solutions is easily affected by noise or emotional timbre, pitch, voiceprint and other factors in the input audio signal, which in turn causes unnatural or jittery speech mouth shape, and even facial jitter. The jitter will interfere with the effect of audio-visual synchronization, resulting in the speech face or head video generation methods having irregular pronunciation mouth shape, which affects the viewing experience of video playback and the audience's acceptance, and is not conducive to some application fields with strict requirements for online customer service. At present, most of the post-processing solutions for generating videos are used to solve the above problems, but they are inefficient, affect the process, and are not easy to promote.

[0004] Most of the methods for generating talking face or head video have irregular pronunciation mouth shape. Therefore, standardized pronunciation mouth shape is required for talking face or head video in application, which is conducive to improving the acceptance of digital human image and enhancing the fidelity of the image. Moreover, the more standardized the pronunciation mouth shape in the talking face video is, the more conducive it is to the promotion of application, which helps to improve service quality and customer satisfaction.

[0005] Therefore, how to generate a speaking face or head video with standard pronunciation and mouth shape is a problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the embodiments of the present application is to propose a method, device, equipment and medium for generating a speaking face video with standardized lip shape to solve the problem of non-standard pronunciation lip shape in the speaking face or head video generated in the prior art.

[0007] In order to solve the above technical problems, the embodiment of the present application provides a method for generating a speaking face video with standardized lip shape, which adopts the following technical solution, including:

[0008] Obtaining the text features, amplitude features and the first lip shape key points of the previous frame of the audio;

[0009] Generate a heat map feature of the audio according to the text feature, the amplitude feature and the key point feature of the first speaking mouth shape;

[0010] Sampling the heat map feature to obtain a first feature vector;

[0011] Extracting an audio segment of the audio, and acquiring a second feature vector by capturing a time sequence feature of the audio segment;

[0012] Input the image sequence of the speaking person's face into the video encoder to obtain the third eigenvector of the image;

[0013] Using the second eigenvector to superimpose the third eigenvector to obtain a fourth eigenvector;

[0014] The first feature vector and the fourth feature vector are decoded by using a video decoder to generate a face video corresponding to the audio.

[0015] Furthermore, the step of obtaining the text features, amplitude features and the first lip shape key points of the previous frame of the audio includes:

[0016] Preprocess the input audio to obtain the target audio;

[0017] Extracting content features of the target audio using an automatic speech recognition model to obtain the text features and amplitude features;

[0018] The facial features are extracted, and the lip shape key points in the facial features are detected using a deep neural network to obtain the first speaking lip shape key points.

[0019] Furthermore, generating the heat map feature of the audio according to the text feature, the amplitude feature and the key point feature of the first speaking mouth shape includes:

[0020] Acquire the amplitude characteristics of the audio;

[0021] Combining the amplitude feature with the text feature to obtain the audio feature set;

[0022] Encode the audio feature set to generate a second speech lip key point feature of the audio

[0023] The first lip shape key point feature is used as a reference point feature, and the second lip shape key point feature is converted into a heat map feature of the audio using an implicit model or a heat map regression technique.

[0024] Furthermore, sampling the heat map feature to obtain a first feature vector includes:

[0025] Using a multi-branch attention module to enhance the heat map features;

[0026] The enhanced heat map features are compressed and sampled to obtain the first feature vector.

[0027] Furthermore, the extracting the audio segment of the audio and acquiring the second feature vector by capturing the time sequence feature of the audio segment includes:

[0028] Acquire an audio segment of a preset length from the audio;

[0029] Obtaining a latent feature vector of the audio clip;

[0030] The latent feature vector is used to construct a second feature vector of the spectral features of the audio segment.

[0031] Furthermore, the step of inputting the image sequence of the speaking person's face into a video encoder to obtain a third feature vector of the image includes:

[0032] Obtain a target image sequence and a reference image sequence of a speaking person's face;

[0033] The target image sequence and the reference image sequence are input into a video encoder to obtain a third eigenvector of the speaking person's face image.

[0034] In order to solve the above problem, a device for generating a speaking face with a standardized lip shape is also provided, which adopts the method for generating a speaking face video with a standardized lip shape described in the first aspect, and comprises:

[0035] A first acquisition module is used to acquire text features, amplitude features and the first lip shape key points of the previous frame of the audio;

[0036] A heat map generation module, used for generating a heat map feature of the audio according to the text feature, the amplitude feature and the key point feature of the first speaking mouth shape;

[0037] A sampling module, used for sampling the heat map feature to obtain a first feature vector;

[0038] A second acquisition module is used to extract an audio segment of the audio, and acquire a second feature vector by capturing a time sequence feature of the audio segment;

[0039] An encoding module, used for inputting the image sequence of the speaking person's face into a video encoder to obtain a third eigenvector of the image;

[0040] an enhancement module, configured to superimpose the third eigenvector using the second eigenvector to obtain a fourth eigenvector;

[0041] A generating module is used to decode the first feature vector and the fourth feature vector by using a video decoder to generate a face video corresponding to the audio.

[0042] Furthermore, the heat map generation module includes:

[0043] An amplitude acquisition unit, used to acquire the amplitude characteristics of the audio;

[0044] A combining unit, configured to combine the amplitude feature with the content feature to obtain the audio feature set;

[0045] An encoding unit, used for encoding the audio feature set to generate a second lip shape key point feature of the audio;

[0046] A conversion unit is used to use the first speaking lip shape key point feature as a reference point feature, and use an implicit model or a heat map regression technology to convert the second speaking lip shape key point feature into a heat map feature of the audio.

[0047] In order to solve the above problems, an embodiment of the present application also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a method for generating a speaking face video with standardized lip shape.

[0048] In order to solve the above problems, an embodiment of the present application further proposes a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, steps such as a method for generating a speaking face video with standardized lip shape are implemented.

[0049] Compared with the prior art, the embodiments of the present application mainly have the following technical effects: by converting the speaking lip shape viewpoint features of the audio into heat map features and sampling to obtain the corresponding feature vectors which are input into the video decoder, and using the feature vectors of the audio clips to enhance the image feature vectors and then perform video decoding, a more standardized speaking lip shape face video is generated, which improves the viewing experience of the speaking lip shape face video, increases users' acceptance, recognition and satisfaction with the video, and improves the practical application value of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of an embodiment of a method for generating a speaking face video with standardized lip shape according to the present application;

[0052] Figure 2 yes Figure 1 A flowchart of a specific implementation of S100;

[0053] Figure 3 yes Figure 1 A flowchart of a specific implementation of S200;

[0054] Figure 4 yes Figure 1 A flowchart of a specific implementation of S300;

[0055] Figure 5 yes Figure 1 A flowchart of a specific implementation of S400;

[0056] Figure 6 yes Figure 1 A flowchart of a specific implementation of S500;

[0057] Figure 7 It is a schematic diagram of a block diagram of an embodiment of a method for generating a speaking face video with standardized lip shape according to the present application;

[0058] Figure 8 It is a schematic diagram of the module structure of a device for generating a speaking face video with standardized lip shape according to the present application;

[0059] Fig. 9 It is a schematic diagram of the module structure of a computer device according to the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the accompanying drawings in the invention to clearly and completely describe the technical solutions in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0063] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0064] The purpose of the embodiments of the present application is to propose a method, device, equipment and medium for generating a speaking face video with standardized lip shape to solve the problem of non-standard pronunciation lip shape in the speaking face or head video generated in the prior art.

[0065] In order to solve the above technical problems, the present application embodiment provides a method for generating a speaking face video with standardized lip shape, which adopts the following technical solutions: Figure 1 , Figure 1 A flowchart of an embodiment of a method for generating a speaking face video with standardized lip shape according to the present application; comprising: S100, obtaining text features, amplitude features and the first lip shape key points (Lip Landmarks) of the previous frame of the audio;

[0066] Specifically, the recognition of text content from audio and the generalization and robustness of various audio signals can meet the requirements of practical applications. Therefore, transfer learning can be combined with self-supervised learning methods to use a large amount of audio-video data from open news media and self-media to pre-train the phoneme to lip landmark mapping network.

[0067] Most of the methods for generating talking faces or head videos have irregular pronunciation and mouth shapes. Therefore, standardized pronunciation and mouth shapes are required for talking faces or head videos in applications, which is conducive to improving the acceptance of digital human images and enhancing the fidelity of the images. Therefore, it is necessary to further use the filmed standardized pronunciation-video dataset. Teachers or graduate students related to news media can be invited to read and produce audio-video data based on the content of the corpus (THCHS-30corpus originated from the Speech and Language Technology Center of Tsinghua University), fine-tune the training network, and then use vanilla heatmap regression or implicit function representation method to construct the key points of pronunciation and mouth shape into a feature-equal heatmap of the corresponding block of the generator (Generator) and introduce it into the talking face video generation network to guide the standardized pronunciation and mouth shape generation process.

[0068] The communication module of the computer device on which the method for generating a speaking face video with standardized lip shape in the embodiment of the present application runs can adopt a wired connection method or a wireless connection method. It should be pointed out that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0069] The generation of most talking face or head videos is easily affected by noise or emotional timbre, pitch, voiceprint and other factors in the input audio signal, which in turn causes unnatural or jittery speech patterns in the generated mouth shape, or even facial jitter. The jitter will interfere with the audio-visual synchronization effect, resulting in irregular pronunciation and lip shape in all talking face or head video generation methods.

[0070] For example, the person in the audio is saying "apple", but the person's mouth shape in the video seems to be saying "banana". The change in mouth shape has nothing to do with the actual pronunciation. This situation will make the audience feel very uncoordinated and unnatural, which is an obvious manifestation of irregular lip shape generation. The changes in the mouth shape of the person in the video when speaking are too exaggerated or too sluggish. For example, when speaking a sentence normally, the opening and closing of the mouth shape is far more than what is actually needed, or the mouth shape hardly changes, which cannot accurately reflect the characteristics of the mouth shape when speaking. This is also not in line with the natural state of people when speaking, and is an irregular situation. The mouth shape is inconsistent with the rhythm of the audio. For example, the audio speed is fast, but the mouth shape in the video changes very slowly, resulting in the sound has been finished, but the mouth shape continues; or the audio speed is slow, but the mouth shape changes quickly, so that there is no good synchronization between the two.

[0071] In some optional embodiments, such as Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific implementation method of S100; S100 includes S110, preprocessing the input audio to obtain the target audio; S120, using the automatic speech recognition model to extract the content features of the target audio to obtain text features and amplitude features; S130, extracting facial features, using a deep neural network to detect the lip shape key points in the facial features, and obtaining the first speaking lip shape key points.

[0072] The background noise of the audio is filtered out through a filtering algorithm to remove noise in a specific frequency range, and the audio is pre-processed by framing and other operations. ASR is a technology that converts human speech into text. This technology is designed to enable computers to "dictate" continuous speech spoken by different people, that is, to convert "sound" into "text". This application uses ASR for audio feature extraction, which is more accurate and more efficient for text conversion. It can adapt to multiple languages ​​and accents, breaking the limitations of language and region.

[0073] The present application has a certain degree of robustness against complex acoustic environments such as background noise and reverberation to a certain extent, and can maintain good recognition performance under less than ideal speech conditions. For example, relatively accurate speech recognition can be achieved in noisy factory workshops or indoor environments with a certain degree of echo.

[0074] S200: Generate a heat map feature of the audio according to the text feature, the amplitude feature and the key point feature of the first utterance.

[0075] The heat map mentioned above is a special map with the heat of different areas marked on it. Among them, the heat represents how different facial key points (such as lips, eyes, eyebrows) should move when speaking. Each point in the heat map corresponds to a specific position on the face, and the heat value of this position indicates how the facial muscles at this position should move to match the sound in the audio at a certain moment. By analyzing each syllable and intonation in the audio, the corresponding key points are found on the heat map. For example, when the sound "ba" appears in the audio, the points in the lip area will become very "hot" on the heat map because the lips need to make a circular shape. Therefore, the speaker's mouth shape and expression can be generated or adjusted according to this map to match the dialogue in the audio.

[0076] In some optional embodiments, such as Figure 3 , Figure 3 yes Figure 1A flowchart of a specific implementation method of S200; S200 includes: S210, obtaining the amplitude feature of the audio; S220, combining the amplitude feature with the text feature to obtain an audio feature set; S230, encoding the audio feature set to generate a second lip landmark feature of the audio; S240, using the first lip landmark feature as a reference point feature, and using an implicit model or a heat map regression technique to convert the second lip landmark feature into a heat map feature of the audio.

[0077] In this embodiment, the principle of using implicit model or heat map regression technology to obtain heat map features of lip key point features is as follows:

[0078] Use a dedicated annotation tool to annotate each frame of video in the training data, determine the position of the lip key points, and then generate a two-dimensional Gaussian distribution heat map on the image with each annotated lip key point as the center. Finally, combine the heat map features of each lip key point feature.

[0079] The above audio features include spectral features, rhythmic features and phonetic features. The amplitude information of the audio reflects the intensity or volume of the sound. The amplitude information can be used to control the degree of opening and closing of the mouth shape or the amplitude of the movement. A stronger amplitude may correspond to a larger opening and closing of the mouth shape, while a weaker amplitude corresponds to a smaller movement.

[0080] By establishing a speech pronunciation model, the extracted audio features (including amplitude information) are mapped to corresponding lip landmark parameters, thereby obtaining lip landmark features.

[0081] In this embodiment, the speech pronunciation model can select a Markov model (HMM) or a deep neural network (DNN), and use the speech pronunciation model to encode the extracted audio features to obtain a coding sequence of the audio features, and map the coding sequence to corresponding lip landmark parameters to obtain lip landmark features.

[0082] The present application can utilize audio features (including amplitude information) to generate relatively reasonable and natural lip key point features by performing accurate feature mapping and time synchronization.

[0083] In this embodiment, the first lip landmark features are: ensuring the smoothness of the generated video, preventing the jitter of the mouth area in the generated video, and adding the lip landmark features of the reference image.

[0084] In this embodiment, the second lip landmark features are lip landmark features of the input audio.

[0085] In this embodiment, the lip shape key point features of the reference image are used as the reference point features for driving, and the second lip shape key point features are input into the implicit module or the heat map regression module to generate the heat map features of the audio. This application uses the first lip shape key point (Lip Landmarks) feature as the reference point feature to obtain the second lip shape key point (Lip Landmarks) feature of the corresponding audio, which is conducive to ensuring the smoothness of the generated video and preventing the jitter of the mouth area of ​​the generated video.

[0086] S300: Sample the heat map features to obtain a first feature vector.

[0087] In this embodiment, characteristic values ​​of each sampling point, such as pixel values, color values ​​or gradient values, are extracted, and the extracted characteristic values ​​constitute characteristic vectors of each sampling point. The first characteristic vector is the characteristic vector of all sampling points.

[0088] In some optional embodiments, such as Figure 4 , Figure 4 yes Figure 1 A flowchart of a specific implementation method of S300; S300 includes: S310, using a multi-branch attention module to enhance the heat map features; S320, compressing and sampling the enhanced heat map features to obtain a first feature vector.

[0089] In this embodiment, the multi-branch attention module calculates the self-attention weights of different positions of the heat map features, and uses the self-attention weights to perform weighted summation of the feature values ​​at each position of the heat map to perform feature enhancement.

[0090] The first eigenvector is a compressed representation of the heat map feature, which can not only retain the important feature information in the heat map as much as possible, but also reduce the dimension and the amount of calculation.

[0091] By using multi-head attention modules to enhance the heatmap features, it is helpful to enhance the effectiveness of heat maps.

[0092] S400: extracting an audio segment of the audio, and acquiring a second feature vector by capturing a time sequence feature of the audio segment.

[0093] The second feature vector is constructed by using the latent feature vector to obtain the audio segment spectrum feature vector.

[0094] In some optional embodiments, such as Figure 5 , Figure 5 yes Figure 1 A flowchart of a specific implementation of S400; S400 includes: S410, obtaining an audio segment of a preset length from the audio; S420, obtaining a latent feature vector of the audio segment; S430, using the latent feature vector to construct a second feature vector of the spectral features of the audio segment.

[0095] For example, in order to ensure the smoothness of the generated video, an audio clip with a preset duration of 200ms is used. A latent feature vector is generated based on the audio clip to construct a spectral feature matrix, which is then passed through a 2D convolutional neural network (CNN) or a recurrent neural network (RNN) deep learning encoder to further refine and output a 512-dimensional feature vector. By using the audio features obtained from the audio clip to fit the features of the speaking lip shape image, the detailed information of the ROI (Region of Interest) regional features of the speaking face image can be enhanced, so that the generated face lip shape video can be made more standardized and smooth. S500, input the image sequence of the speaking face into the video encoder to obtain the third feature vector of the image.

[0096] In this embodiment, a video encoder is used to perform intra-frame prediction based on the correlation between pixels in a frame, and inter-frame prediction based on the correlation between facial images between adjacent frames. The motion vector of the current frame is determined based on the reference image frame, thereby determining the displacement of the corresponding area, and then the third eigenvector is obtained through discrete cosine transform (DCT) and quantization coding.

[0097] In some optional embodiments, such as Figure 6 , Figure 6 yes Figure 1 A flowchart of a specific implementation of S500; S500 includes: S510, obtaining a target image sequence and a reference image sequence of a speaking person's face; S520, inputting the target image sequence and the reference image sequence into a video encoder to obtain a third feature vector of the speaking person's face image.

[0098] In this embodiment, the third feature vector is a feature vector corresponding to the ROI (Region of Interest) image of the speaker's face. By using the target image sequence and the reference image sequence to generate the third feature vector corresponding to the speaker's face image, it is helpful to improve the accuracy of the ROI (Region of Interest) regional image.

[0099] S600: Use the second eigenvector to superimpose the third eigenvector to obtain a fourth eigenvector.

[0100] In this embodiment, the feature vector of the spectral feature of the audio segment (the second feature vector) is superimposed with the image feature vector of the spectral feature of the corresponding audio segment extracted from the image sequence (the third feature vector) to obtain the fourth feature vector mentioned above.

[0101] S700: Decode the first feature vector and the fourth feature vector using a video decoder to generate a face video corresponding to the audio.

[0102] Most of the methods for generating talking face or head videos have irregular pronunciation lip shapes. Therefore, standardized pronunciation lip shapes are required in the application of talking face or head videos, which is conducive to improving the acceptance of digital human images and enhancing the fidelity of the image.

[0103] Audio-visual generation is a cross-modal application field. The network training process can easily cause feature drift or even mode collapse, and the generated audio may have related problems such as picture jitter. In particular, this is likely to occur during the network training process of mapping phonemes and lip landmarks. Therefore, the loss function of the speaking face video generation model in the embodiment of the present application uses a weighted linear combination of L2 and speed loss.

[0104] In the embodiment of the present application, the training of generating a video of a speaking person with standard lip shape is divided into two stages:

[0105] like Figure 7 , is a schematic diagram of an embodiment of a method for generating a speaking face video with standardized lip shape according to the present application. In terms of audio processing, the audio and the first lip shape key point features are respectively passed through an automatic speech recognition module and a position encoder / position cloud encoder, and then through an audio encoder to obtain the second lip shape key point features, and then through an implicit module or a heat map regression module to obtain heat map features and obtain a first feature vector through sampling;

[0106] In terms of video processing, the target image sequence and the reference image sequence are processed through a video encoder to obtain the third eigenvector, the fourth eigenvector is obtained by superimposing and enhancing the second eigenvector, and the first eigenvector and the fourth eigenvector are decoded through a video decoder to generate a face video.

[0107] When training a generative model for talking face videos:

[0108] In the first stage, the loss function used for the generation of lip landmarks from audio drive to speech is a weighted linear combination of L2 and speed loss, as shown in formula (1):

[0109] T a-v =w1·T L +w2·T V (1)

[0110] in,

[0111] T a-v is the loss function of the speech lip key point generation model, w1,w is the weighting factor of the L2 loss function, w2, is the weighting factor of the speed loss function, T L is the L2 loss function, T V is the speed loss function.

[0112] Using the training method based on self-supervised learning of audio-video datasets, a frame of face image at a certain time point can be randomly sampled to extract the key points of speaking lip shape (Lip Landmarks) as the benchmark data GT (Ground Truth), and a frame of face image is randomly obtained with the index to extract the key points of speaking lip shape (Lip Landmarks) as the base point input to the position encoder or point cloud encoder module; at the same time, the starting timestamp of the audio corresponding to the benchmark data GT (Ground Truth) is obtained according to the FPS (Frames per Second) of the audio-video, and a 200ms continuous audio segment is extracted from this time point (for example, a 20ms voice segment is input into ASR (Automatic Speech Recognition: DeepSpeech) to obtain an n=29-dimensional vector, and then the amplitude value Vp of the segment is linked to reconstruct an n+1=30-dimensional vector). In this way, five consecutive frames of face images are extracted from the benchmark data GT (Ground Truth) as the starting point and input into the network. The audio input is also input into the audio encoding module by the corresponding five continuous segments, and its size is [10,30].

[0113] The second stage is for speech-driven speaking face generation. The training in this stage needs to freeze the training parameters of the first stage. The loss function used includes the reconstruction loss L1 loss function, lip synchronization discrimination loss, perceptual loss, identity recognition loss and balanced weighted linear combination of adversarial loss, as shown in formula (2):

[0114] T all =w1·T L1 +w2·T sync +w3·T P +w4·T id +w5·T disc (2)

[0115] W1, W2, W3, W4, and W5 are the corresponding weighting factors, T L1 is the reconstruction loss L1 loss function, T syncis the lip sync discrimination loss function, T P is the Perceptual loss function, T id is the identity recognition loss function, T disc is the adversarial loss function.

[0116] In order to solve the above problems, the embodiment of the present application further proposes a device 800 for generating a speaking face with a standard lip shape, which adopts a method for generating a speaking face video with a standard lip shape, such as Figure 8 , Figure 8 A schematic diagram of the module structure of a device for generating a video of a speaking person's face with a standardized lip shape according to the present application; the device for generating a speaking person's face with a standardized lip shape 800 corresponds one-to-one to the method for generating a speaking person's face with a standardized lip shape in the above-mentioned embodiment, including:

[0117] The first acquisition module 801 is used to acquire the text features, amplitude features and the first lip shape key points of the previous frame of the audio;

[0118] A heat map generation module 802 is used to generate a heat map feature of the audio according to the text feature, the amplitude feature and the key point feature of the first utterance;

[0119] A sampling module 803 is used to sample the heat map features to obtain a first feature vector;

[0120] A second acquisition module 804 is used to extract an audio segment of the audio, and acquire a second feature vector by capturing a time sequence feature of the audio segment;

[0121] The encoding module 805 is used to input the image sequence of the speaking person's face into the video encoder to obtain the third feature vector of the image;

[0122] An enhancement module 806, configured to superimpose the third eigenvector using the second eigenvector to obtain a fourth eigenvector;

[0123] The generating module 807 is used to decode the first feature vector and the fourth feature vector by using a video decoder to generate a face video corresponding to the audio.

[0124] By converting the speaking lip shape viewpoint features of the audio into heat maps and sampling them to obtain the corresponding feature vectors, the feature vectors of the image are enhanced by using the feature vectors of the audio clips and then the video is decoded, a more standardized speaking lip shape face video is generated, which improves the viewing experience of the speaking lip shape face video.

[0125] Furthermore, the heat map generation module 802 includes: an amplitude acquisition unit, which is used to obtain the amplitude characteristics of the audio; a combination unit, which is used to combine the amplitude characteristics with the content characteristics to obtain an audio feature set; an encoding unit, which is used to encode the audio feature set to generate a second speaking lip shape key point feature of the audio; a conversion unit, which is used to use the first speaking lip shape key point feature as a reference point feature, and use an implicit model or a heat map regression technology to convert the second speaking lip shape key point feature into a heat map feature of the audio.

[0126] The heat map regression technology is highly robust to noise, illumination changes and partial occlusion in the image. Since it is based on probability distribution representation, it can provide lip key point information even in the case of poor image quality.

[0127] In order to solve the above problems, an embodiment of the present application also proposes a computer device, which adopts the following technical solution: it includes a memory, a communication module and a processor, the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a method for generating a speaking face video with standardized lip shape.

[0128] The computer device may be a computer, server, workstation or other device, or a mobile phone, tablet, vehicle-mounted mobile terminal or other device with program execution capability. The internal structure diagram of the computer device may be as follows: Fig. 9 As shown, Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. The computer device includes a processor, a memory and a communication module. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, instructions or codes. The internal memory provides an environment for the operation of the operating system and instructions or codes in the non-volatile storage medium. When the instructions or codes are executed by the processor, the functions or steps of a method for fine-tuning the above-mentioned pre-trained model are implemented. The communication module of the computer device may include a network interface and / or a wireless communication module, and the computer device may communicate with other devices or service platforms through the communication module. In addition, the computer device may also include a display screen and an input device, etc.

[0129] The memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the processor implements the steps of the method for generating a speaking face with a standardized lip shape when executing the instructions or codes.

[0130] In order to solve the above problems, the embodiment of the present application further proposes a computer-readable storage medium, which adopts the following technical solution:

[0131] The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes an adaptive parameter efficient fine-tuning method.

[0132] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, Figures 1 to 6 A method for generating a speaking face video with standardized lip shape provided in each step, for details, please refer to the implementation methods provided in the above steps, which will not be repeated here.

[0133] The computer-readable storage medium may be a device for generating a video of a speaking person with a standardized lip shape provided by any of the aforementioned embodiments, or an internal storage unit of the terminal device, such as a hard disk or memory of a computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device.

[0134] Furthermore, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0135] However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0136] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0137] Compared with the prior art, the embodiments of the present application mainly have the following technical effects: by converting the speaking lip shape viewpoint features of the audio into heat map features and sampling to obtain the corresponding feature vectors to input into the video decoder, and using the feature vector of the audio clip to enhance the image feature vector and then perform video decoding, a more standardized speaking lip shape face video is generated, which improves the viewing experience of the speaking lip shape face video, increases the user's acceptance, recognition and satisfaction with the video, and improves the practical application value of the model.

[0138] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0139] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a video of a speaking face with standardized lip shape, comprising: Obtaining the text features, amplitude features and the first lip shape key points of the previous frame of the audio; Generate a heat map feature of the audio according to the text feature, the amplitude feature and the key point feature of the first speaking mouth shape; Sampling the heat map feature to obtain a first feature vector; Extracting an audio segment of the audio, and acquiring a second feature vector by capturing a time sequence feature of the audio segment; Input the image sequence of the speaking person's face into the video encoder to obtain the third eigenvector of the image; Using the second eigenvector to superimpose the third eigenvector to obtain a fourth eigenvector; The first feature vector and the fourth feature vector are decoded by using a video decoder to generate a face video corresponding to the audio.

2. The method for generating a speaking face video with standardized lip shape according to claim 1, characterized in that: Get the text features, amplitude features and the first lip shape key points of the previous frame of the audio, including: Preprocess the input audio to obtain the target audio; Extracting content features of the target audio using an automatic speech recognition model to obtain the text features and amplitude features; The facial features are extracted, and the lip shape key points in the facial features are detected using a deep neural network to obtain the first speaking lip shape key points.

3. The method for generating a speaking face video with standardized lip shape according to claim 2, characterized in that: Generating the heat map feature of the audio according to the text feature, the amplitude feature and the key point feature of the first speaking mouth shape includes: Acquire the amplitude characteristics of the audio; Combining the amplitude feature with the text feature to obtain an audio feature set; Encoding the audio feature set to generate a second speaking lip key point feature of the audio; The first lip shape key point feature is used as a reference point feature, and the second lip shape key point feature is converted into a heat map feature of the audio using a heat map regression technique.

4. The method for generating a speaking face video with standardized lip shape according to claim 3, characterized in that: The step of sampling the heat map feature to obtain a first feature vector includes: Using a multi-branch attention module to enhance the heat map features; The enhanced heat map features are compressed and sampled to obtain the first feature vector.

5. The method for generating a speaking face video with standardized lip shape according to claim 1, characterized in that: The step of extracting the audio segment of the audio and acquiring the second feature vector by capturing the time sequence feature of the audio segment includes: Acquire an audio segment of a preset length from the audio; Obtaining a latent feature vector of the audio clip; The latent feature vector is used to construct a second feature vector of the spectral features of the audio segment.

6. The method for generating a speaking face video with standardized lip shape according to claim 1, characterized in that: The step of inputting the image sequence of the speaking person's face into a video encoder to obtain a third feature vector of the image comprises: Obtain a target image sequence and a reference image sequence of a speaking person's face; The target image sequence and the reference image sequence are input into a video encoder to obtain a third eigenvector of the speaking person's face image.

7. A device for generating a face of a person speaking with a standard lip shape, using the method for generating a face video of a person speaking with a standard lip shape according to any one of claims 1 to 6, characterized in that: include: A first acquisition module is used to acquire text features, amplitude features and the first lip shape key points of the previous frame of the audio; A heat map generation module, used for generating a heat map feature of the audio according to the text feature, the amplitude feature and the key point feature of the first speaking mouth shape; A sampling module, used for sampling the heat map feature to obtain a first feature vector; A second acquisition module is used to extract an audio segment of the audio, and acquire a second feature vector by capturing a time sequence feature of the audio segment; An encoding module, used for inputting the image sequence of the speaking person's face into a video encoder to obtain a third eigenvector of the image; an enhancement module, configured to superimpose the third eigenvector using the second eigenvector to obtain a fourth eigenvector; The third feature vector is superimposed and generated by the module, and is used to decode the first feature vector and the fourth feature vector by using a video decoder to generate a face video corresponding to the audio.

8. The device for generating a speaking face with a standard lip shape according to claim 7, characterized in that: The heat map generation module includes: An amplitude acquisition unit, used to acquire the amplitude characteristics of the audio; A combining unit, configured to combine the amplitude feature with the content feature to obtain the audio feature set; An encoding unit, used for encoding the audio feature set to generate a second lip shape key point feature of the audio; A conversion unit is used to use the first speaking lip shape key point feature as a reference point feature, and use an implicit model or a heat map regression technology to convert the second speaking lip shape key point feature into a heat map feature of the audio.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method for generating a speaking face with a standard lip shape as claimed in any one of claims 1 to 6 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for generating a speaking face with a standard lip shape according to any one of claims 1 to 6 are implemented.

Images