Real-person voice lip animation generation method and system, electronic device, and storage medium

Abstract

The application discloses a real human voice mouth shape animation generation method and system, electronic equipment and a storage medium, and comprises the following steps: performing audio extraction, face tracking and registration on a multi-segment speech video material library and a target speech video, and extracting input audio, mouth shape region texture and face region texture; pre-training a general audio encoder for audio-driven mouth shape according to the input audio and the mouth shape region texture of the multi-segment speech video material library; training a model for audio-driven mouth shape for the target speech video according to the pre-trained general audio encoder and the input audio and the mouth shape region texture data of the target speech video; training a model for mouth shape-driven face for the target speech video according to the mouth shape region texture and the face region texture of the target speech video; and generating a real human voice mouth shape animation sequence from the input audio according to the trained model for audio-driven mouth shape and the model for mouth shape-driven face.

Description

Technical Field

[0001] This invention relates to the field of speech and image technology, specifically to a method and system for generating animation of real-person speech lip movements, an electronic device, and a storage medium. Background Technology

[0002] Most current methods rely on explicit intermediate representations (facial landmarks, 3DMM coefficients) or semi-explicit representations (NeRF, 3DGS) to achieve audio-driven live lip-sync animation. However, intermediate representations are often sparse and lossy, and the extracted intermediate representations are often inaccurate.

[0003] To generate high-quality lip-sync animations for a single person, a substantial amount of data is often required for that individual. The paper *Neural Voice Puppetry: Audio-driven Facial Reenactment*, based on 3DMM coefficients, used 3-5 minutes of data. Similarly, NeRF-based methods like AD-NeRF and ER-NeRF, and 3DGS-based methods like GaussianTalker and TalkingGaussian, also require 3-5 minutes of data. A key characteristic of these methods is that the quality of the generated result is strongly dependent on the length of the input target speech video; the quality of the generated lip-sync animation increases with the amount of data. However, a drawback is that at least several minutes, or even hours, of data need to be collected for each target character, resulting in poor scalability. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for generating lip-sync animation of real-person speech, an electronic device, and a storage medium.

[0005] The first objective of this invention is to provide a method for generating lip-sync animation of real human voice, comprising:

[0006] The system extracts audio, tracks and registers faces from a library of multiple speech video clips and a target speech video, extracting input audio, lip-sync textures, and facial textures.

[0007] Based on the input audio and lip-sync region textures from a multi-segment speech video material library, a general audio encoder that drives lip-sync is pre-trained.

[0008] Based on a pre-trained general audio encoder, an audio-driven lip-sync model is trained on the input audio and lip-sync region texture data of a target speech video.

[0009] Based on the lip-shape region texture and face region texture of the target speech video, train a lip-shape driven face model for the target speech video.

[0010] Audio from any language of any country or region is input into a trained audio-driven lip-sync model, and the output of the audio-driven lip-sync model is input into a trained lip-sync-driven face model to generate a realistic speech lip-sync animation sequence.

[0011] As a further improvement of the present invention, the step of extracting audio, tracking and registering faces from a library of multiple speech video materials and a target speech video, and extracting input audio, lip-sync texture, and facial texture, includes:

[0012] Audio extraction is performed on the speech video. One audio frame is extracted for each frame of the image. Each audio frame is obtained by sampling audio features from a fixed-length window, denoted as:

[0013] ;

[0014] in, This represents a collection of input audio extracted from a speech video, which may be derived from a library of multiple speech video clips or a single target speech video. express The first in frame, The dimension representing the basic audio features (which can be MFCC or pre-trained basic audio features). This represents the sampling window length for each frame of audio (typically, the base audio feature is 50 FPS, therefore the time span of the sampling window for each frame of audio is...). );

[0015] Face tracking and registration were performed on the speech video. For each frame of the image, one frame of lip region texture and one frame of face region texture were extracted, denoted as:

[0016] ;

[0017] ;

[0018] in, , These represent the sets of lip shape textures and face textures extracted from a speech video, respectively. The speech video comes from a library of multiple speech video materials or a single target speech video. , They represent , The first in The frames have image sizes of [number] and [number] respectively. , ;

[0019] Extracted facial region texture By using facial feature point detection, the lower half of the face region in each frame is masked to obtain the texture of the masked facial region, denoted as:

[0020] .

[0021] As a further improvement of the present invention, the step of pre-training a general audio encoder that drives lip movements based on input audio and lip-sync region textures from a multi-segment speech video material library includes:

[0022] set up This indicates the number of videos in the speech video library;

[0023] The input audio and lip-sync region textures of the multi-segment speech video material library are denoted as:

[0024] ;

[0025] ;

[0026] , These represent the first and second videos from the speech video library. A collection of input audio and lip-sync textures extracted from a short speech video. ;

[0027] The audio-driven lip-sync encoding and decoding process is denoted as:

[0028] ;

[0029] in, This represents a general audio encoder. This means encoding each frame of input audio into a single... 3D vector Indicates the first Lip-sync decoder for a speech video. There are a total of A different lip-sync decoder; Indicates that it will be encoded The dimensional vector is restored to the 1st dimension. Lip-sync texture in a short speech video;

[0030] For different videos from a multi-speech video archive, all input audio Sharing the same audio encoder The lip-sync textures from different videos Depend on A different lip-sync decoder The decoding process yields the audio lip-sync encoding and decoding process for multiple speech videos sharing a single audio encoder. This process is then pre-trained to obtain a more general audio encoder suitable for audio-driven lip-sync tasks. .

[0031] As a further improvement of the present invention, the step of training an audio-driven lip-sync model for a target speech video based on the input audio and lip-sync region texture data of a target speech video using a pre-trained general audio encoder includes:

[0032] The input audio and lip-sync texture data for a target speech video are denoted as:

[0033] ;

[0034] ;

[0035] , These represent the sets of input audio and lip-sync textures extracted from a target speech video, respectively.

[0036] The process of audio-driven lip-syncing for a target speech video is denoted as:

[0037] ;

[0038] in, This represents a pre-trained general audio encoder whose weights are fixed. This means inputting each frame of the target speech video into the audio. Encode as a A dimensional vector; This represents a lip-sync decoder for the target speech video. Indicates that it will be encoded The 3D vector is used to reconstruct the lip-sync region texture of the target speech video. ;

[0039] Based on the audio lip-sync encoding and decoding process of this single video, an audio-driven lip-sync model suitable for the target speech video is trained, which includes an audio encoder. A lip-sync decoder .

[0040] As a further improvement of the present invention, the step of training a lip-shape-driven face model for the target speech video based on the lip-shape region texture and face region texture of the target speech video includes:

[0041] The lip-sync texture and face texture of the target speech video are denoted as:

[0042] ;

[0043] ;

[0044] ;

[0045] in, , , These represent the sets of lip-shape region textures, face region textures, and masked face region textures extracted from a target speech video, respectively.

[0046] The process of lip movements driving facial expressions in a target speech video is denoted as:

[0047] ;

[0048] in, It is a mouth-type encoder. This indicates the texture of the lip-sync area in the target speech video. Encode as a A dimensional vector; It is a mask face encoder. This indicates the texture of the facial area behind the mask in the target speech video. Encode as a A dimensional vector; This indicates that lip-shape texture encoding and masked facial texture encoding are performed in... Additive fusion is performed on the feature space of the dimensionality; This refers to a face generator that will merge the results. 3D features were used to restore the facial texture of the target speech video. ;

[0049] Based on the lip-syncing process in this single video, a lip-syncing-driven face model suitable for the target speech video was trained, which includes a lip-syncing encoder. A mask face encoder A face generator .

[0050] The second objective of this invention is to provide a system for generating lip-sync animation of real-person speech, comprising:

[0051] The extraction module is used to extract audio, track and register faces from a library of multiple speech video materials and a target speech video, and to extract input audio, lip shape texture and face texture.

[0052] A general audio encoder pre-training module is used to pre-train a general audio encoder that drives lip movements based on input audio and lip-sync region textures from a library of multiple speech video clips.

[0053] The audio-driven lip-sync module is used to train an audio-driven lip-sync model for a target speech video based on the input audio and lip-sync region texture data of the target speech video, using a pre-trained general audio encoder.

[0054] The lip-sync module is used to train a lip-sync model for a target speech video based on the lip and face region textures of the target speech video.

[0055] The generation module is used to input audio from any country or region into a trained audio-driven lip-sync model, and input the output of the audio-driven lip-sync model into a trained lip-sync-driven face model to generate a sequence of realistic speech lip-sync animations.

[0056] A third objective of the present invention is to provide an electronic device comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the aforementioned method for generating lip-sync animation of real human voice.

[0057] A fourth objective of the present invention is to provide a storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the above-described method for generating lip-sync animation of real human voice.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] This invention trains an audio-driven lip-sync model by extracting input audio, lip-sync texture, and facial texture from a library of multiple speech video clips and a short target speech video; thus enabling the generation of high-quality, realistic lip-sync animation videos from a 10-second target speech video. Attached Figure Description

[0060] Figure 1 This is a flowchart of the method for generating lip-sync animation of real human voice disclosed in this invention;

[0061] Figure 2 This is a framework diagram of the real-person speech lip-sync animation generation system disclosed in this invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] The present invention will now be described in further detail with reference to the accompanying drawings:

[0064] like Figure 1 As shown, the present invention provides a method for generating lip-sync animation of real human voice, including:

[0065] Step 1: Extract audio, track and register faces from a library of multiple speech video clips and a target speech video (5-15 seconds in length), extracting the input audio, lip-sync texture, and facial texture.

[0066] Specifically, it includes:

[0067] Audio extraction is performed on the speech video. One audio frame is extracted for each frame of the image. Each audio frame is obtained by sampling audio features from a fixed-length window, denoted as:

[0068] ;

[0069] in, This represents a collection of input audio extracted from a speech video, which may be derived from a library of multiple speech video clips or a single target speech video. express The first in frame, The dimension representing the basic audio features (MFCC is used in this example). ), This represents the sampling window length for each frame of audio (since the base audio characteristic is 50 FPS, this example uses...). Therefore, the time span of each audio sampling window is ).

[0070] Face tracking and registration were performed on the speech video. For each frame of the image, one frame of lip region texture and one frame of face region texture were extracted, denoted as:

[0071] ;

[0072] ;

[0073] in, , These represent the sets of lip shape textures and face textures extracted from a speech video, respectively. The speech video comes from a library of multiple speech video materials or a single target speech video. , They represent , The first in The frames have image sizes of [number] and [number] respectively. , This example uses , .

[0074] Extracted facial region texture By using facial feature point detection, the lower half of the face region in each frame is masked to obtain the texture of the masked facial region, denoted as:

[0075] ;

[0076] In this embodiment, the target speech video is a very short one, only about 10 seconds long. The multi-segment speech video library is a fixed auxiliary dataset, consisting of... The dataset consists of several supplementary speech videos, totaling 74.7 minutes in length. It's important to note that this multi-segment speech video library is a fixed supplementary dataset and does not include the characters from the target speech videos.

[0077] Step 2: Based on the input audio and lip-sync region textures from a multi-segment speech video material library, pre-train a general audio encoder that drives lip-sync.

[0078] Specifically, it includes:

[0079] set up This indicates the number of videos in the speech video library.

[0080] The input audio and lip-sync textures for the multi-segment speech video material library are denoted as:

[0081] ;

[0082] ;

[0083] , These represent the first and second videos from the speech video library. part( The input audio and lip-sync textures extracted from the speech video.

[0084] The audio-driven lip-sync encoding and decoding process is denoted as:

[0085]

[0086] in, This represents a general audio encoder. This means encoding each frame of input audio into a single... A dimensional vector. Indicates the first part( There are a total of lip-sync decoders for speech videos. A different lip-sync decoder Indicates that it will be encoded The dimensional vector is restored to the 1st dimension. The texture of the lip-sync area in a short speech video.

[0087] For different videos from multiple speech video material libraries ( All input audio Sharing the same audio encoder The lip-sync textures from different videos Depend on A different lip-sync decoder The decoding is obtained. Based on this, the audio lip-sync encoding and decoding process of multiple speech videos sharing a single audio encoder can be pre-trained to obtain a more general audio encoder suitable for audio-driven lip-sync tasks. .

[0088] In this embodiment, the feature dimension of the audio encoding is taken. Dimensions. Among them, the general audio encoder. The network structure consists of a regular Transformer, with an input dimension of... The output dimension is Lip-sync decoder The network structure consists of a standard CNN, with an input dimension of... The output dimension is It is important to note that the audio encoder... lip-sync decoder The choice of network structure is not limited to the network structure used in this example; any neural network structure with matching input and output dimensions can be adapted. Training uses... The loss function was set, and the Adam optimizer was used with a learning rate of 0.0002, a batch size of 64, and 170,000 iterations. This was done to efficiently train different lip-sync decoders from different speech videos. It adopts an alternating batch sampling strategy, that is, each batch in each iteration samples only data from the same video segment for training.

[0089] Step 3: Based on the pre-trained general audio encoder, train an audio-driven lip-sync model for the target speech video on the input audio and lip-sync region texture data of the target speech video.

[0090] Specifically, it includes:

[0091] A pre-trained general audio encoder, using express.

[0092] The input audio and lip-sync texture data for a target speech video are denoted as:

[0093] ;

[0094] ;

[0095] , These represent the sets of input audio and lip-sync textures extracted from a target speech video, respectively.

[0096] The process of audio-driven lip-syncing for a target speech video is denoted as:

[0097] ;

[0098] in, This represents a pre-trained general audio encoder with fixed weights. This means inputting each frame of the target speech video into the audio. Encode as a A dimensional vector. This represents a lip-sync decoder for the target speech video. Indicates that it will be encoded The 3D vector is used to reconstruct the lip-sync region texture of the target speech video. .

[0099] Based on the audio lip-sync encoding and decoding process of this single video, an audio-driven lip-sync model suitable for the target speech video can be trained, which includes an audio encoder. A lip-sync decoder .

[0100] In this embodiment, a general audio encoder The weights are fixed, and only the lip-sync decoder of the target speech video is optimized. The weights. Training uses The loss function was set, and the Adam optimizer was used with a learning rate of 0.0002, a batch size of 64, and 50,000 iterations. Other parameters and network structure were the same as those of a general audio encoder. Training implementation parameters.

[0101] Step 4: Based on the lip-shape region texture and face region texture of the target speech video, train a lip-shape driven face model for the target speech video.

[0102] Specifically, it includes:

[0103] The lip-sync texture and face texture of the target speech video are denoted as:

[0104] ;

[0105] ;

[0106] ;

[0107] in, , , These represent the sets of lip-sync textures, face textures, and masked face textures extracted from a target speech video, respectively.

[0108] The process of lip movements driving facial expressions in a target speech video is denoted as:

[0109] ;

[0110] in, It is a mouth-type encoder. This indicates the texture of the lip-sync area in the target speech video. Encode as a A dimensional vector; It is a mask face encoder. This indicates the texture of the facial area behind the mask in the target speech video. Encode as a A dimensional vector. Then, This indicates that lip-shape texture encoding and masked facial texture encoding are performed in... The feature space of the dimension is fused by addition. Finally, This refers to a face generator that will merge the results. 3D features were used to restore the facial texture of the target speech video. Based on the lip-sync driven face process of this single video, a lip-sync driven face model suitable for the target speech video can be trained, which includes a lip-sync encoder. A mask face encoder A face generator .

[0111] In this embodiment, the training loss function is as follows:

[0112] ;

[0113] in, It is a perceptual loss function based on VGG16. It is an image loss function based on SSIM.

[0114] In this embodiment, the feature dimension of the intermediate encoding is taken. Dimensions. Among them, the mouth encoder. Face encoder with mask The network structure is the same, consisting of a regular CNN, with an input dimension of... The output dimension is Due to the texture of the mouth area The original size is In the input mouth encoder The front dimensions will first be transformed into Face generator The network structure is a CNN-based upsampled image generator with an input dimension of . The output dimension is It is important to note that lip encoders... Face encoder with mask Face generator The choice of network structure is not limited to the network structure used in this example; any neural network structure with matching input and output dimensions can be adapted. Training used the Adam optimizer, a learning rate of 0.0004, a batch size of 8, and 120,000 iterations.

[0115] Step 5: Input audio from any language of any country or region into the trained audio-driven lip-sync model, and input the output of the audio-driven lip-sync model into the trained lip-sync-driven face model to generate a realistic speech lip-sync animation sequence.

[0116] like Figure 2 As shown, the present invention provides a system for generating lip-sync animation of real-person speech, comprising:

[0117] The extraction module is used to extract audio, track and register faces from a library of multiple speech video materials and a target speech video, and to extract input audio, lip shape texture and face texture.

[0118] A general audio encoder pre-training module is used to pre-train a general audio encoder that drives lip movements based on input audio and lip-sync region textures from a library of multiple speech video clips.

[0119] The audio-driven lip-sync module is used to train an audio-driven lip-sync model for a target speech video based on the input audio and lip-sync region texture data of the target speech video, using a pre-trained general audio encoder.

[0120] The lip-sync module is used to train a lip-sync model for a target speech video based on the lip and face region textures of the target speech video.

[0121] The generation module is used to generate a sequence of realistic speech lip-sync animations based on two pre-trained models for a target speech video: an audio-driven lip-sync model and a lip-sync-driven face model. It can take audio from any country or region as input.

[0122] An electronic device provided by an embodiment of the present invention includes at least one processing unit and at least one storage unit, wherein the storage unit stores a program, and when the program is executed by the processing unit, the processing unit performs the above-described method for generating lip-sync animation of real human voice.

[0123] The present invention provides a computer-readable storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, it causes the electronic device to execute the above-described method for generating lip-sync animation of real human voice.

[0124] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a real human voice mouth animation, characterized by, Comprise: Audio extraction, face tracking and registration are performed on the multi-lecture video material library and a target lecture video, and input audio, lip region texture and face region texture are extracted; A general audio encoder driven by audio and lip is pre-trained according to the input audio and the lip region texture of the multi-lecture video material library; Specifically, it comprises: Set represents the number of videos in the presentation video library; The input audio and the lip region texture of the multi-lecture video material library are denoted as: ； ； , represent a set of input audio, lip region texture extracted from a 5th segment of a lecture video in the lecture video material library, ;​ The encoding and decoding process of the audio-driven lip is denoted as: ； wherein, represents a general audio encoder, represents encoding each frame of input audio into a dimensional vector, represents a mouth decoder for the segment of the speech video, , there are different mouth decoders in total; represents restoring the encoded dimensional vector into a mouth region texture for the segment of the speech video; For different videos from a multi-segment speech video material library, all the input audios share the same audio encoder , while the lip region texture from different videos are decoded by different lip decoders ; based on this, the audio-lip codec process of multi-segment speech video sharing a single audio encoder is pre-trained to obtain a general audio encoder more suitable for the audio-driven lip task ; According to the pre-trained general audio encoder, an audio-driven lip model for the target lecture video is trained on the input audio and the lip region texture data of a target lecture video; According to the lip region texture and the face region texture of the target lecture video, a lip-driven face model for the target lecture video is trained; The audio input of any country or region language is input into the trained audio-driven lip model, and the output result of the audio-driven lip model is input into the trained lip-driven face model to generate real human voice and lip animation sequence.

2. The method of claim 1, wherein, The audio extraction, face tracking and registration are performed on the multi-lecture video material library and a target lecture video, and input audio, lip region texture and face region texture are extracted; Comprise: Audio extraction is performed on the lecture video, and one frame of audio is extracted for each frame of image, and each frame of audio is sampled by a fixed length window to obtain audio features, denoted as: ； wherein, represents a set of input audio extracted from a lecture video, the lecture video being from a library of lecture video materials or a target lecture video; represents the frame, represents a dimension of the base audio feature, represents a length of a sampling window of each frame of audio; Face tracking and registration are performed on the lecture video, and one frame of lip region texture and one frame of face region texture are extracted for each frame of image, denoted as: ； ； wherein, , respectively represent a set of lip region texture, face region texture extracted from a speech video, the speech video is from a speech video material library or a target speech video; , respectively represent the first frame in , , and the image size of , , , respectively The face region texture extracted The lower half of the face region of each frame is masked by using face feature point detection to obtain a face region texture after masking, denoted as: 。 3. The method for generating lip-sync animation of real-person speech as described in claim 2, characterized in that, According to the pre-trained general audio encoder, an audio-driven lip model for the target lecture video is trained on the input audio and the lip region texture data of a target lecture video; comprising: The input audio and the lip region texture data of a target lecture video are denoted as: ； ； , respectively denote a set of input audio, lip region texture extracted from a target speech video; The process of the audio-driven lip for the target lecture video is denoted as: ； wherein, represents a pre-trained general audio encoder whose weights are fixed; represents inputting each frame of the target speech video to the audio encoder to obtain a dimensional vector; represents a lip decoder of the target speech video, represents restoring the encoded dimensional vector to the lip region texture of the target speech video ; Based on the audio mouth shape coding process of the single video, an audio-driven mouth shape model suitable for the target speech video is trained, including an audio encoder , a mouth shape decoder .

4. The method for generating lip-sync animation of real-person speech as described in claim 3, characterized in that, According to the lip region texture and the face region texture of the target lecture video, a lip-driven face model for the target lecture video is trained; comprising: The lip region texture and the face region texture of the target lecture video are denoted as: ； ； ； wherein, , , respectively represent a set of lip region texture, face region texture, masked face region texture extracted from a target speech video. The process of the lip-driven face for the target lecture video is denoted as: ； wherein, is a lip encoder, represents encoding the lip region texture of the target speech video into a dimensional vector; is a masked face encoder, represents encoding the masked face region texture of the target speech video into a dimensional vector; represents additively fusing the lip texture encoding, the masked face texture encoding in a dimensional feature space; represents a face generator, restoring the fused dimensional feature into the face region texture of the target speech video ; Based on the single-video lip-driven face process, a model suitable for lip-driven face of the target speech video is trained, including a lip encoder , a mask face encoder , a face generator .

5. A real human voice lip animation generation system for implementing the real human voice lip animation generation method according to any one of claims 1 to 4, characterized by, Comprise: An extraction module is configured to perform audio extraction, face tracking and registration on the multi-lecture video material library and a target lecture video, and extract input audio, lip region texture and face region texture; A general audio encoder pre-training module is configured to pre-train a general audio encoder driven by audio and lip according to the input audio and the lip region texture of the multi-lecture video material library; An audio-driven lip module is configured to train an audio-driven lip model for the target lecture video on the input audio and the lip region texture data of a target lecture video according to the pre-trained general audio encoder; A lip-driven face module is configured to train a lip-driven face model for the target lecture video according to the lip region texture and the face region texture of the target lecture video; The generating module is configured to input audio in any national or regional language into the trained audio-driven lip model, input an output result of the audio-driven lip model into the trained lip-driven face model, and generate a real human voice lip animation sequence.

6. An electronic device, comprising: The computer program product comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the real human voice lip animation generation method in any one of claims 1-4.

7. A storage medium, characterized by The computer program product comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the real human voice lip animation generation method in any one of claims 1-4.

Images