Mouth shape action generation method based on voice synchronization

Abstract

The invention discloses a mouth shape action generation method based on voice synchronization, and belongs to the technical field of voice synchronization, and the generation method comprises the following specific steps: I, collecting and preprocessing a plurality of groups of video corpus data, and building an adversarial network comprising a generative model and a discrimination model; iI, constructing input of a generative model according to the preprocessed video corpus data, and obtaining a prediction image corresponding to each input through the generative model; iII, constructing the input of a discrimination model according to the predicted image, and judging the difference between the real video and the predicted image through the discrimination model; according to the method, the accuracy of voice mouth shape synchronization can be improved, the occurrence probability of strange mouth shape states is reduced, and more accurate mouth shape synchronization in the aspect of Chinese voice is realized.

Description

Technical Field

[0001] This invention relates to the field of speech synchronization technology, and in particular to a method for generating lip movements based on speech synchronization. Background Technology

[0002] With the advent and widespread use of GPT (Google Sense Point), AI digital humans have sprung up like mushrooms after rain. Digital humans can serve as customer service representatives for corporate websites or applications, providing users with immediate assistance and answering frequently asked questions. In the field of digital advertising, digital humans can act as brand ambassadors or advertising spokespeople, attracting user attention and conveying information. Digital humans can also act as tour guides, providing tourists with virtual travel experiences, introducing attractions and history. Combining GPT Q&A and voice generation, and bringing these to life through 3D character models or 2D visuals, this becomes the core of digital human technology. In practical applications, the synchronization of 3D / 2D character mouth movements with speech content, emotion, and speech rate, along with natural and fluid body movements, is extremely important. Mouth movements are particularly crucial because humans are highly sensitive to the matching of mouth movements and speech; therefore, achieving a high degree of synchronization between mouth movements and speech is a very important technological foundation for the practical application of digital human technology. With the application of deep learning models in the field of digital humans, speech generation models that synchronize lip movements have a wide range of applications: by generating realistic lip movements through speech, digital humans can open their mouths and speak, replacing real people in many application scenarios.

[0003] Currently, there are two approaches to voice-synchronized lip-shape generation: one approach is to use voice to drive a 3D face model and generate facial vertex animation data; the other approach is to extract information from voice to generate a new lip-shape image from an existing 2D face image.

[0004] The bottleneck of the first approach lies in the significant limitations of 3D models. When users want to use custom face models, complex face vertex rigging must be performed first. Even after complex vertex rigging, the final generated mouth shape differs from that of the default model. This means that whenever a new speaking face model needs to be customized, this method requires a significant amount of time to process the 3D model and vertex mapping. Furthermore, the quality of the 3D model and the accuracy of vertex mapping severely impact the final generated result. Even with open-source implementations on GitHub and available solutions from NVIDIA, this approach lacks general applicability until accurate and convenient 3D modeling technology matures (which is currently immature).

[0005] The second approach involves synchronizing the lip movements of the speaker in a 2D facial video with the target speech. One method is to use a deep learning model to learn the lip movements of a large number of different pronunciations in human speech, thus generating realistic lip movements to match the desired speech in the 2D video. This method has yielded some research results in the field of English lip-syncing, with one model, wav2lip, capable of basic lip-syncing in English 2D videos. However, this model has several problems: due to the low resolution of the training video source, the generated lip movements are often blurry, out of sync, or exhibit strange lip movements. The problems are even more pronounced in Chinese speech synchronization. Because the wav2lip model is trained on a large number of English videos, the out-of-sync and strange lip movements are particularly noticeable when used for Chinese speech generation. Meanwhile, the overall design of the model is also unsuitable for lip-syncing in Chinese speech. Even when the model is trained with a large amount of Chinese video data, it still encounters problems such as the inability to train the synchronization judgment model, the loss equation not decreasing, and insufficient use of speech data by the generation model. As a result, even with a large amount of Chinese video training, the final generation effect is still poor. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for generating lip movements based on speech synchronization.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for generating lip movements based on speech synchronization, the specific steps of which are as follows:

[0009] Ⅰ: Collect and preprocess multiple sets of video corpus data, and simultaneously establish an adversarial network that includes a generative model and a discriminative model;

[0010] II: Based on the preprocessed video corpus data, construct the input of the generation model, and obtain the predicted image corresponding to each input through the generation model;

[0011] III: Based on the predicted image, construct the input of the discrimination model, and use the discrimination model to determine the difference between the real video and the predicted image;

[0012] IV: Alternately optimize the generative model and the discriminative model until they reach the preset performance requirements to complete the training of the adversarial network;

[0013] V: The trained adversarial network predicts the user's lip movements in real time and monitors the performance changes of the adversarial network in real time.

[0014] As a further aspect of the present invention, the specific steps for collecting and preprocessing multiple sets of video corpus data in step I are as follows:

[0015] S1.1: Collect multiple sets of video corpus data, perform time alignment on the video frames and audio files in the multiple sets of video corpus data, and then perform frame-level splitting on the video frames in the multiple sets of video corpus data;

[0016] S1.2: Use a face detection algorithm to locate the face regions corresponding to each video frame, perform geometric alignment on each face region, and then crop the aligned face regions to a uniform size.

[0017] S1.3: Simultaneously extract the speech features from the speech files corresponding to each video frame after processing, divide the speech in each speech file into frames and encode them into semantic units, and then extract the pitch information corresponding to each frame of speech in each speech file.

[0018] S1.4: Establish generative and discriminative models, and then form a complete adversarial network framework based on the generative and discriminative models.

[0019] As a further aspect of the present invention, the specific steps for obtaining the predicted image corresponding to each input through the generative model in step II are as follows:

[0020] S2.1: Extract multi-frame image sequences from each preprocessed video frame and scale each frame image sequence to a uniform size. At the same time, extract the speech segments corresponding to each image sequence from the preprocessed speech file, obtain the speech features of the corresponding speech segments, and then mask the mouth of the person in the last frame of each image sequence.

[0021] S2.2: Concatenate multiple image sequences and corresponding speech features according to a preset format to generate input samples for the model, and then input the input samples into the generation model;

[0022] S2.3: Extract low-level edge and texture visual features of each frame image through shallow convolutional networks in the generative model, and repeatedly extract visual features through stacked convolutional layers, including mouth corner shape, jaw contour and facial muscle changes. Then, integrate the spatial information between each frame image by channel-by-channel stitching to capture the motion trend and dynamic changes of the context frame of the multi-frame image sequence.

[0023] S2.4: Through a cross-modal attention mechanism, speech features are saliency-aligned with the mouth region of a multi-frame image sequence. Then, the fused speech features are fused with the information from the previous and next frames of the multi-frame image sequence to generate the predicted image corresponding to the input sample.

[0024] As a further aspect of the present invention, the specific steps for determining the difference between the real video and the predicted image using a discriminant model in step III are as follows:

[0025] S3.1: The multi-frame image sequence in the input sample is concatenated with the predicted image of the generator model to form a continuous image sequence of a preset length. The corresponding speech segments of the continuous image sequence are extracted from the speech file, and the speech features in the corresponding speech segments are obtained. Then, the continuous image sequence and the corresponding speech features are aligned in the time domain.

[0026] S3.2: The continuous image sequence is fused with the corresponding speech features to form the prediction sample of the discriminant model. The prediction sample is then input into the discriminant model. The discriminant model extracts the spatial and temporal features of the prediction sample through the convolutional layer, encodes the speech features, and then determines whether the prediction image generated by the model in the prediction sample is a real video frame, and outputs the corresponding probability value or label.

[0027] S3.3: Based on the probability value or label, mark the predicted image of the generation model as real or generated. If the model is identified as generated, it means that the predicted image of the generation model is not realistic. Otherwise, it means that the generation effect is natural.

[0028] As a further aspect of the present invention, the specific steps of alternating optimization of the generation model and the discrimination model in step IV are as follows:

[0029] S4.1: Use the cross-entropy loss function to calculate the adversarial loss when the predicted image of the occluded region in the current frame is judged as true, and use the pixel difference measurement method to calculate the reconstruction loss between the predicted image of the generator model and the image in the discriminant sample. At the same time, extract the intermediate layer features of the discriminant model and the intermediate layer features corresponding to the discriminant sample image in the discriminant model, and calculate the feature matching loss.

[0030] S4.2: Weight the adversarial loss, reconstruction loss and feature matching loss, and update the parameters of the generated model based on the weighting result. At the same time, repeatedly generate the model prediction image, make the judgment of the discriminator model and update the parameters until the preset number of rounds is reached.

[0031] S4.3: After training, use the validation set to evaluate the clarity of the predicted images generated by the generative model and the lip-sync of speech, and the accuracy of the discriminative model in recognizing real and fake video clips. Then, use quantitative indicators and qualitative evaluation to judge the performance of the generative model and the discriminative model.

[0032] S4.4: Monitor the loss curve during the verification process in real time. When the loss curve is lower than the preset threshold, save the parameters of the final generated model and the discriminant model and terminate the training. Otherwise, repeat the alternating training and verification.

[0033] As a further aspect of the present invention, the specific steps of real-time prediction of the user's lip movements by the trained adversarial network in step V are as follows:

[0034] S5.1: Collect real-time audio files from each user, segment each real-time audio file frame by frame, extract the audio features corresponding to each audio segment, and simultaneously acquire the previous consecutive video image corresponding to each user, and unify the format of each video image.

[0035] S5.2: Concatenate each video image with the corresponding speech features and input them into the generation model. Cover the mouth of the person in the last frame of each video image and then use the generation model to predict the mouth shape of the person through the corresponding speech features.

[0036] S5.3: Add the predicted current frame image to the image sequence as one of the previous frame inputs for the next round. Then the generation model continues to read the next segment of speech features and repeats the preset rounds to form a continuous predicted video frame of the person's mouth shape changing with the speech.

[0037] S5.4: The predicted continuous video frames are stitched together in real time into a corresponding dynamic image sequence, and the corresponding dynamic image sequence is superimposed on the user's real avatar to generate a corresponding mouth shape action image. Then, each mouth shape action is output through image rendering.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] This invention collects and preprocesses video corpus data to construct a generative model and a discriminative model, forming an adversarial network. The generative model uses multiple frames of facial images and speech features for joint modeling to generate lip-sync images synchronized with the speech. The discriminative model judges whether the predicted image is realistic and outputs the judgment result. The generative model is jointly optimized using adversarial loss, reconstruction loss, and feature matching loss to make the predicted image more natural. After training, the system can receive user speech in real time, generate lip-sync images synchronized with the speech frame by frame, and dynamically overlay them onto the user's avatar. This achieves high-precision, natural, and smooth lip-sync synthesis, improving the accuracy of speech-lip-sync synchronization and reducing the probability of strange lip-sync states, thus achieving more accurate lip-sync synchronization in Chinese speech. Attached Figure Description

[0040] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0041] Figure 1 This is a flowchart of the lip-syncing generation method proposed in this invention. Detailed Implementation

[0042] Example 1

[0043] Reference Figure 1 A method for generating lip movements based on speech synchronization is described, and the specific steps of this method are as follows:

[0044] Multiple sets of video corpus data were collected and preprocessed, and an adversarial network containing both a generative model and a discriminative model was established.

[0045] Specifically, multiple sets of video corpus data are collected, and the video frames and audio files in the multiple sets of video corpus data are time-aligned. Then, the video frames in the multiple sets of video corpus data are split into frames, and face detection algorithms are used to locate the face regions corresponding to each video frame. The face regions are then geometrically aligned, and the aligned face regions are cropped to a uniform size. At the same time, the speech features in the audio files corresponding to each video frame are extracted, and the speech in each audio file is framed and encoded into semantic units. Then, the pitch information corresponding to each frame of speech in each audio file is extracted, and a generative model and a discriminative model are established. Finally, a complete adversarial network framework is formed based on the generative model and the discriminative model.

[0046] Based on the preprocessed video corpus data, the input to the generative model is constructed, and the predicted image corresponding to each input is obtained through the generative model.

[0047] Specifically, multiple image sequences are extracted from each preprocessed video frame, and each image sequence is scaled to a uniform size. Simultaneously, speech segments corresponding to each image sequence are extracted from the preprocessed speech file, and their speech features are obtained. Then, the mouth of the person in the last frame of each image sequence is masked. The multiple image sequences and their corresponding speech features are then concatenated according to a preset format to generate the model's input sample. This input sample is then fed into the generative model. A shallow convolutional network in the generative model extracts low-level edge and texture visual features from each frame, and these visual features, including mouth shape, jaw contour, and facial muscle changes, are repeatedly extracted through stacked convolutional layers. The spatial information between each frame is then integrated through channel-by-channel concatenation to capture the motion trends and dynamic changes of the context frames of the multi-frame image sequence. A cross-modal attention mechanism is used to saliency-align the speech features with the mouth region of the multi-frame image sequence. Finally, the fused speech features are fused with the information from the preceding and following frames of the multi-frame image sequence to generate the predicted image corresponding to the input sample.

[0048] Based on the predicted image, the input to the discrimination model is constructed, and the difference between the real video and the predicted image is determined by the discrimination model.

[0049] Specifically, the multi-frame image sequence from the input sample is concatenated with the predicted image from the generator model to form a continuous image sequence of a preset length. Speech segments corresponding to the continuous image sequence are extracted from the speech file, and speech features from these segments are obtained. The continuous image sequence and corresponding speech features are then temporally aligned and fused to form the prediction sample for the discriminator model. This prediction sample is then input into the discriminator model, which extracts the spatial and temporal features of the prediction sample through convolutional layers and encodes the speech features. The model then determines whether the predicted image from the generator model is a real video frame and outputs a corresponding probability value or label. Based on the probability value or label, the predicted image is labeled as real or generated. If the discriminator model identifies it as generated, it indicates that the predicted image from the generator model is not yet realistic; otherwise, it indicates that the generation effect is natural.

[0050] Example 2

[0051] Reference Figure 1 A method for generating lip movements based on speech synchronization is described, and the specific steps of this method are as follows:

[0052] The generative and discriminative models are alternately optimized until they reach the preset performance requirements to complete the training of the adversarial network.

[0053] Specifically, the cross-entropy loss function is used to calculate the adversarial loss when the predicted image of the occluded region in the current frame is judged as real. The pixel difference metric is used to calculate the reconstruction loss between the predicted image of the generator model and the image in the discriminant sample. At the same time, the intermediate layer features of the discriminant model and the intermediate layer features corresponding to the discriminant sample image in the discriminant model are extracted, and the feature matching loss is calculated. The adversarial loss, reconstruction loss and feature matching loss are weighted and the parameters of the generator model are updated according to the weighting result. The generator model predicts images, the discriminant model judges and updates parameters repeatedly until a preset number of rounds are reached. After training, the clarity of the predicted images generated by the generator model and the lip-sync of speech are evaluated using a validation set, and the accuracy of the discriminant model in recognizing real and fake video segments is evaluated. The performance of the generator model and the discriminant model is judged by quantitative indicators and qualitative evaluation. The loss curve during the validation process is monitored in real time. When the loss curve is lower than a preset threshold, the parameters of the final generator model and the discriminant model are saved and the training is terminated. Otherwise, the alternating training and validation are repeated.

[0054] The trained adversarial network predicts users' lip movements in real time and monitors changes in the network's performance in real time.

[0055] Specifically, real-time audio files from each user are collected and segmented frame by frame. The audio features corresponding to each frame are extracted, and the previous consecutive video frame corresponding to each user is obtained. The format of each video image is standardized, and the video images are concatenated with the corresponding audio features and input into the generation model. The mouth of the person in the last frame of each video image is covered. The generation model then predicts the mouth shape based on the corresponding audio features. The predicted current frame image is added to the image sequence as one of the previous frame inputs for the next round. The generation model then continues to read the next audio feature segment, repeating the preset rounds to form continuous predicted video frames showing the mouth shape changing with the audio. The predicted continuous video frames are then concatenated in real-time into a corresponding dynamic image sequence, which is then overlaid on the user's real avatar to generate corresponding mouth shape action images. Finally, each mouth shape action is rendered and output.

Claims

1. A method for generating lip movements based on speech synchronization, characterized in that, The specific steps of this generation method are as follows: Ⅰ: Collect and preprocess multiple sets of video corpus data, and simultaneously establish an adversarial network that includes a generative model and a discriminative model; II: Based on the preprocessed video corpus data, construct the input of the generation model, and obtain the predicted image corresponding to each input through the generation model; III: Based on the predicted image, construct the input of the discrimination model, and use the discrimination model to determine the difference between the real video and the predicted image; IV: Alternately optimize the generative model and the discriminative model until they reach the preset performance requirements to complete the training of the adversarial network; V: The trained adversarial network predicts the user's lip movements in real time and monitors the performance changes of the adversarial network in real time.

2. The method for generating lip movements based on speech synchronization according to claim 1, characterized in that, The specific steps for collecting and preprocessing multiple sets of video corpus data in Step I are as follows: S1.1: Collect multiple sets of video corpus data, perform time alignment on the video frames and audio files in the multiple sets of video corpus data, and then perform frame-level splitting on the video frames in the multiple sets of video corpus data; S1.2: Use a face detection algorithm to locate the face regions corresponding to each video frame, perform geometric alignment on each face region, and then crop the aligned face regions to a uniform size. S1.3: Simultaneously extract the speech features from the speech files corresponding to each video frame after processing, divide the speech in each speech file into frames and encode them into semantic units, and then extract the pitch information corresponding to each frame of speech in each speech file. S1.4: Establish generative and discriminative models, and then form a complete adversarial network framework based on the generative and discriminative models.

3. The method for generating lip movements based on speech synchronization according to claim 2, characterized in that, The specific steps for obtaining the predicted image corresponding to each input through the generative model in step II are as follows: S2.1: Extract multi-frame image sequences from each preprocessed video frame and scale each frame image sequence to a uniform size. At the same time, extract the speech segments corresponding to each image sequence from the preprocessed speech file, obtain the speech features of the corresponding speech segments, and then mask the mouth of the person in the last frame of each image sequence. S2.2: Concatenate multiple image sequences and corresponding speech features according to a preset format to generate input samples for the model, and then input the input samples into the generation model; S2.3: Extract low-level edge and texture visual features of each frame image through shallow convolutional networks in the generative model, and repeatedly extract visual features through stacked convolutional layers, including mouth corner shape, jaw contour and facial muscle changes. Then, integrate the spatial information between each frame image by channel-by-channel stitching to capture the motion trend and dynamic changes of the context frame of the multi-frame image sequence. S2.4: Through a cross-modal attention mechanism, speech features are saliency-aligned with the mouth region of a multi-frame image sequence. Then, the fused speech features are fused with the information from the previous and next frames of the multi-frame image sequence to generate the predicted image corresponding to the input sample.

4. The lip-syncing generation method based on speech synchronization according to claim 3, characterized in that, The specific steps for determining the difference between the real video and the predicted image using the discriminant model in step III are as follows: S3.1: The multi-frame image sequence in the input sample is concatenated with the predicted image of the generator model to form a continuous image sequence of a preset length. The corresponding speech segments of the continuous image sequence are extracted from the speech file, and the speech features in the corresponding speech segments are obtained. Then, the continuous image sequence and the corresponding speech features are aligned in the time domain. S3.2: The continuous image sequence is fused with the corresponding speech features to form the prediction sample of the discriminant model. The prediction sample is then input into the discriminant model. The discriminant model extracts the spatial and temporal features of the prediction sample through the convolutional layer, encodes the speech features, and then determines whether the prediction image generated by the model in the prediction sample is a real video frame, and outputs the corresponding probability value or label. S3.3: Based on the probability value or label, mark the predicted image of the generation model as real or generated. If the model is identified as generated, it means that the predicted image of the generation model is not realistic. Otherwise, it means that the generation effect is natural.

5. The lip-syncing generation method based on speech synchronization according to claim 4, characterized in that, The specific steps for alternating optimization of the generative and discriminative models described in step IV are as follows: S4.1: Use the cross-entropy loss function to calculate the adversarial loss when the predicted image of the occluded region in the current frame is judged as true, and use the pixel difference measurement method to calculate the reconstruction loss between the predicted image of the generator model and the image in the discriminant sample. At the same time, extract the intermediate layer features of the discriminant model and the intermediate layer features corresponding to the discriminant sample image in the discriminant model, and calculate the feature matching loss. S4.2: Weight the adversarial loss, reconstruction loss and feature matching loss, and update the parameters of the generated model based on the weighting result. At the same time, repeatedly generate the model prediction image, make the judgment of the discriminator model and update the parameters until the preset number of rounds is reached. After S4.3 training, the validation set is used to evaluate the clarity of the predicted images generated by the generative model and the lip-sync of speech, as well as the accuracy of the discriminative model in recognizing real and fake video clips. Then, the performance of the generative model and the discriminative model is judged by quantitative indicators and qualitative evaluation. S4.4: Monitor the loss curve during the verification process in real time. When the loss curve is lower than the preset threshold, save the parameters of the final generated model and the discriminant model and terminate the training. Otherwise, repeat the alternating training and verification.

6. The method for generating lip movements based on speech synchronization according to claim 5, characterized in that, The specific steps for the adversarial network trained in step V to predict the user's lip movements in real time are as follows: S5.1: Collect real-time audio files from each user, segment each real-time audio file frame by frame, extract the audio features corresponding to each audio segment, and simultaneously acquire the previous consecutive video image corresponding to each user, and unify the format of each video image. S5.2: Concatenate each video image with the corresponding speech features and input them into the generation model. Cover the mouth of the person in the last frame of each video image and then use the generation model to predict the mouth shape of the person through the corresponding speech features. S5.3: Add the predicted current frame image to the image sequence as one of the previous frame inputs for the next round. Then the generation model continues to read the next segment of speech features and repeats the preset rounds to form a continuous predicted video frame of the person's mouth shape changing with the speech. S5.4: The predicted continuous video frames are stitched together in real time into a corresponding dynamic image sequence, and the corresponding dynamic image sequence is superimposed on the user's real avatar to generate a corresponding mouth shape action image. Then, each mouth shape action is output through image rendering.

Images