Digital human voice lip synchronization method

Abstract

The invention provides a digital human voice lip shape synchronization method, and relates to the technical field of digital human generation, and the method comprises the steps: obtaining a source video, carrying out the preprocessing of the source video, and obtaining audio data and image data; performing synchronous processing on the audio coding features and the image coding features by adopting a lip shape synchronous model to obtain synchronous features; the audio coding features are output by the audio data through the lip shape synchronization model; the image coding features are output by the image data through the lip shape synchronization model; the synchronous processing comprises linear feature projection and feature similarity calculation; and generating a video according to the synchronization feature to obtain a target video. Through the method, the problems that the matching degree of the voice and the lip shape of the digital human generated in the prior art is not high and the digital human cannot adapt to a multi-language environment are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human generation, and in particular to a digital human voice lip synchronization method. Background Art

[0002] With the rapid development of artificial intelligence technology, 2D digital human technology has become a hot topic in many fields such as virtual reality, augmented reality, games, entertainment and education. Digital humans, that is, virtual digitized characters, can imitate the behavior and expressions of real people and provide interactive user experience. In these applications, lip synchronization technology is one of the key factors to achieve natural and smooth digital humans.

[0003] Traditional 2D digital human generation algorithms usually rely on pre-recorded animations or simple deformation techniques to simulate lip shape, but these methods are difficult to adapt to real-time changing voice signals, resulting in the phenomenon that the generated digital human often has a lack of synchronization between sound and picture when speaking. To solve this problem, researchers began to explore lip synchronization technology based on deep learning. These technologies achieve high-precision lip synchronization effects by analyzing audio signals and video frames, allowing digital humans to present lip shape changes that match the speech content when speaking.

[0004] Existing lip synchronization technology is generally implemented based on Wav2Lip technology, but it has problems such as difficulty in model convergence and inability to adapt to multi-language environments. Summary of the invention

[0005] The present application provides a digital human voice lip synchronization method to solve the problem that the digital human voice and lip shape generated by the prior art are not well matched and cannot adapt to a multi-language environment.

[0006] The method comprises: Acquire a source video, and preprocess the source video to obtain audio data and image data; The lip synchronization model is used to synchronously process the audio coding features and the image coding features to obtain synchronization features; the audio coding features are output by the audio data through the lip synchronization model; the image coding features are obtained by image segmentation, feature fusion and feature coding by the lip synchronization model; the synchronization processing includes linear feature projection and feature similarity calculation; Video generation is performed according to the synchronization feature to obtain a target video.

[0007] Preferably, the lip synchronization model includes: An image processing module, wherein the image processing module is configured to perform image coding according to the image data to obtain a plurality of image coding features; An audio processing module, wherein the audio processing module is configured to perform audio encoding according to the audio data to obtain a plurality of audio encoding features; A synchronization processing module is configured to perform audio and image synchronization processing according to the audio coding feature and the image coding feature to obtain the synchronization feature.

[0008] Preferably, the image processing module includes: a first grouping unit, wherein the first grouping unit is configured to divide the input image data into a plurality of continuous image sequences; each of the image sequences includes a plurality of continuous image frames; A feature fusion unit, wherein the feature fusion unit includes a plurality of fusion layers that are distributed in parallel, and the plurality of fusion layers are configured to perform feature fusion on the image frames in the plurality of image sequences respectively to obtain a plurality of corresponding first sequences; An image coding unit, wherein the image coding unit comprises a plurality of image coding layers using parallel distribution, wherein the plurality of image coding layers correspond one-to-one to the plurality of fusion layers, and the plurality of image coding layers are configured to respectively perform convolution coding on a plurality of the first sequences to obtain a plurality of corresponding image coding features.

[0009] Preferably, the audio processing module includes: a second grouping unit, wherein the second grouping unit is configured to divide the input audio data into a plurality of continuous audio sequences; each of the audio sequences includes a plurality of continuous audio sub-data, and each of the audio sub-data corresponds to one of the image frames; An audio encoding unit, wherein the audio encoding unit comprises a plurality of audio encoding layers that are distributed in parallel, and the plurality of audio encoding layers are configured to respectively perform convolution encoding on a plurality of the audio sequences to obtain a plurality of corresponding audio encoding features.

[0010] Preferably, the image encoding unit further comprises a plurality of spatial attention layers using parallel distribution, and the plurality of spatial attention layers correspond one to one with the plurality of image encoding layers; The spatial attention layer is further configured to calculate a spatial attention weight according to the first sequence; The image encoding layer is also configured to perform convolution encoding according to the first sequence and the spatial attention weight to output the image encoding feature.

[0011] Preferably, the image encoding unit further comprises a channel attention layer; The image encoding unit is further configured to divide the first sequence into a plurality of subsequences, perform feature encoding processing on each subsequence, calculate attention weights corresponding to different subsequences through the spatial attention layer, and fuse a plurality of feature encoding processing results based on the attention weight calculation results to obtain local feature encoding; The local feature code is divided into multiple subsequences, feature coding processing is performed on each subsequence, attention weights corresponding to different subsequences are calculated by a preset spatial attention module, and multiple feature coding processing results are fused based on the attention weight calculation results to obtain the local feature code; Iteratively updating, and independently outputting the local feature codes obtained in each round to obtain multiple local feature codes; Attention weights are calculated for the multiple local feature codes according to the channel attention layer, and feature processing is performed based on the attention weight calculation results to obtain a global feature code; the multiple local feature codes are fused with the global feature code to obtain the image coding feature.

[0012] Preferably, the synchronization processing module includes: A feature projection module, wherein the feature projection module includes a plurality of linear layers, wherein the plurality of linear layers correspond one-to-one to the plurality of image coding layers and the plurality of audio coding layers, respectively; the plurality of linear layers are configured to map the image coding features and the audio coding features into a multimodal embedding space; A similarity calculation module, wherein the similarity calculation module includes a plurality of calculation layers, and the plurality of calculation layers correspond one-to-one to the plurality of linear layers; the plurality of calculation layers are configured to calculate the similarity of the image coding features and the audio coding features in the multimodal embedding space, and perform feature fusion on the image coding features and the audio coding features with the highest similarity to obtain the synchronization features.

[0013] Preferably, the training process of the lip synchronization model includes: Obtaining a training video, and extracting audio training data and image training data based on the training video; Performing model training on the audio training data, the image training data and the lip synchronization model; The lip synchronization model is converged using a loss function until it meets the preset model requirements.

[0014] Preferably, the step of using the loss function to converge the lip synchronization model comprises: Performing contrast loss training on the synchronization features using a contrast loss function; When the synchronization feature meets the requirements of the first preset model, the lip synchronization model is trained with positive samples and negative samples using a ternary loss function until it meets the requirements of the second preset model.

[0015] Preferably, the step of performing positive sample training and negative sample training on the lip synchronization model comprises: Acquire an anchor feature, a positive feature, and a negative feature; the anchor feature is a randomly selected image feature or audio feature; the positive feature is an image feature or audio feature that matches the anchor feature; the negative feature is an image feature or audio feature that does not match the anchor feature; Positive sample training is performed according to the anchor feature and the positive feature, and negative sample training is performed according to the anchor feature and the negative feature, until both the positive sample training and the negative sample training meet the second preset model requirement.

[0016] Preferably, the step of preprocessing the source video includes: Extracting a source image according to the source video, and sequentially performing image framing processing, face detection processing, face region cropping processing, occlusion detection processing, and image covering processing on the source image to obtain the image data; The source audio is extracted according to the source video, and the source audio is subjected to audio frame processing and audio denoising processing in sequence to obtain the audio data.

[0017] As can be seen from the above content, the present application provides a method for lip synchronization of digital human speech, the method comprising obtaining a source video, preprocessing the source video to obtain audio data and image data; using a lip synchronization model to synchronize audio coding features and image coding features to obtain synchronization features; the audio coding features are output by the audio data through the lip synchronization model; the image coding features are output by the image data through the lip synchronization model; the synchronization processing includes linear feature projection and feature similarity calculation; video generation is performed according to the synchronization features to obtain a target video. The present application solves the problem that the existing technology for generating digital human speech and lip matching is not high and cannot adapt to multilingual environments through the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A flowchart of a digital human voice lip synchronization method for this application; Figure 2 A schematic diagram of a lip synchronization model in a digital human speech lip synchronization method of the present application; Figure 3 A schematic diagram of an image processing module in a digital human voice lip synchronization method of the present application; Figure 4 A schematic diagram of an audio processing module in a digital human voice lip synchronization method of the present application; Figure 5 A schematic diagram of a synchronization processing module in a digital human voice lip synchronization method of the present application; Figure 6 A training flow chart of a lip synchronization model in a digital human speech lip synchronization method in this application; Figure 7 A flow chart of the convergence mode of the lip synchronization model in the digital human speech lip synchronization method of the present application; Figure 8 A flowchart of positive sample training and negative sample training in a digital human speech lip synchronization method for this application; Fig. 9 This is a flowchart of preprocessing in a digital human speech lip synchronization method of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In recent years, a technology called Wav2Lip has attracted much attention. Wav2Lip is a model of a universal speaker that can generate videos with lip sync accuracy that matches real sync videos. Its core architecture consists of a generator and two discriminators: an expert lip sync discriminator and a visual quality discriminator. The expert lip sync discriminator is responsible for accurately discriminating the sound and lip sync in the video, while the visual quality discriminator is used to improve the image quality.

[0022] In practical applications, such as lip syncing of videos recorded handheld outdoors, there are still challenges. These challenges include the diversity of expressions of people in the video, lighting changes, occlusion issues, and different speaking styles. In order to improve the robustness and accuracy of the generative network, the researchers proposed a method to correct the generative network using an expert discriminator. This method uses a pre-trained expert-level lip sync discriminator to penalize the inaccurate generation of the generative network during training, thereby improving the lip sync quality of the generated frames.

[0023] However, the expert lip sync discriminator is prone to mode collapse during training, especially when dealing with large datasets composed of multiple languages, and the model is difficult to converge, which brings great difficulties to multilingual applications. In addition, since Wav2Lip is mainly trained on English datasets, its accuracy and robustness when dealing with non-English languages ​​may be reduced, which poses an additional challenge for applications in multilingual environments.

[0024] Based on the above problems, the present application provides the following embodiments to solve the above problems.

[0025] Figure 1 The present invention provides a flow chart of a digital human voice lip synchronization method.

[0026] See also Figure 1 It can be seen that this embodiment provides a digital human voice lip synchronization method, the method comprising: S10, obtaining a source video, and preprocessing the source video. Specifically, in this embodiment, since conventional video data sources are generally obtained from open source databases, and the quality of such videos is generally uneven, it is necessary to perform relevant preprocessing on the video before generating a digital human, so as to meet the standards for generating a digital human, thereby improving the quality of the generated digital human.

[0027] A video with sound is generally composed of sound and image. Through the preprocessing of this embodiment, the sound and image are separated to obtain audio data and image data.

[0028] Fig. 9 This is a flowchart of preprocessing in a digital human speech lip synchronization method of the present application.

[0029] See also Fig. 9 It can be seen that, further, in some embodiments, the step of preprocessing the source video includes: S11, extracting a source image according to the source video, and sequentially performing image frame segmentation processing, face detection processing, face region cropping processing, occlusion detection processing, and image masking processing on the source image to obtain the image data; S12, extracting source audio according to the source video, and performing audio frame processing and audio denoising processing on the source audio in sequence to obtain the audio data.

[0030] Specifically, in this embodiment, before generating a digital human, video acquisition is first performed to obtain continuous video data containing clear speech and lip shape changes. Subsequently, the video data is decomposed into separate frames, and the audio in the accompanying video is also divided into corresponding audio frames for synchronous processing.

[0031] In the video frame processing stage, face detection technology is used to locate the face area in each frame, and then the face part is accurately cropped and the upper half of the face is covered, because the lip shape changes mainly occur in this area. The cropped face is also subjected to occlusion detection to ensure that the face area is not blocked by other objects, thereby ensuring the accuracy of lip shape recognition.

[0032] Next, the segmented audio frames are denoised to extract clearer and more accurate audio features, which are essential for subsequent lip sync analysis. After the audio is processed, the extracted audio features are matched with the face area in the video frame for analysis. In order to make the training model focus more on lip movement rather than other facial features, the upper half of the face is also masked.

[0033] It should be noted that step S11 and step S12 are performed simultaneously, and there is no order of precedence.

[0034] The method further comprises: S20, using a lip synchronization model to synchronize the audio coding features and the image coding features. Specifically, in this embodiment, the lip synchronization model is used to synchronize the audio coding features and the image coding features to obtain relevant features for generating digital human videos, namely, synchronization features.

[0035] Correspondingly, the audio coding features are obtained by inputting the audio data into the lip synchronization model for calculation, and the image coding features are obtained by inputting the image data into the lip synchronization model for calculation; the audio coding features and the image coding features are respectively extracted through the lip synchronization model, and the audio coding features and the image coding features are used for synchronization processing to combine the sound with the lip shape changes of the character, thereby achieving speech lip synchronization.

[0036] The synchronization processing includes linear feature projection and feature similarity calculation, and the image features and audio features are mapped together through the linear feature projection and the feature similarity calculation, so as to achieve the fusion of audio and features.

[0037] The method further comprises: S30, generating a video according to the synchronization feature. Specifically, after completing step S20, a synchronization feature for characterizing the digital human video is obtained, and a video is generated according to the synchronization feature to obtain a final target video.

[0038] Figure 2 This is a schematic diagram of a lip synchronization model in a digital human speech lip synchronization method of the present application.

[0039] See also Figure 2 It can be seen that, further, in some embodiments, the lip synchronization model includes: An image processing module, wherein the image processing module is configured to perform image encoding according to the image data. Specifically, in the present embodiment, the image processing module performs image encoding on the image data in the source video to obtain features corresponding to the image in the source video, wherein the features obtained include a number of image encoding features.

[0040] The lip sync model also includes: An audio processing module, wherein the audio processing module is configured to perform audio encoding based on the audio data. Specifically, in this embodiment, the audio data in the source video is audio encoded by the audio processing module to obtain features corresponding to the audio in the source video, wherein the obtained features include a number of audio encoding features.

[0041] It should be noted that the image coding features correspond one-to-one to the audio coding features.

[0042] The lip sync model also includes: A synchronization processing module, wherein the synchronization processing module is configured to perform audio and image synchronization processing according to the audio coding features and the image coding features. Specifically, in this embodiment, the image coding features obtained by the image processing module and the audio coding features obtained by the audio processing module are subjected to audio and image synchronization processing by the synchronization processing module to unify the image coding features and the audio coding features together to obtain the synchronization features, thereby achieving synchronization of speech lip shape.

[0043] Figure 3 This is a schematic diagram of an image processing module in a digital human speech lip synchronization method of the present application.

[0044] See also Figure 3 It can be seen that, further, in some embodiments, the image processing module includes: The first grouping unit is configured to divide the input image data into a number of continuous image sequences. Specifically, in the present embodiment, in order to better synchronize the voice and lip shape, a single image cannot well reflect the features in the image, and therefore only one image cannot be used for feature extraction. In the present embodiment, the image data is divided by the first grouping unit to divide the image data into a number of continuous image sequences, and each of the image sequences includes a number of continuous image frames, so as to provide an image data basis for subsequent synchronization processing.

[0045] The image processing module also includes: A feature fusion unit, wherein the feature fusion unit includes a plurality of fusion layers that are distributed in parallel, and the plurality of fusion layers are configured to perform feature fusion on the image frames in the plurality of image sequences respectively. Specifically, in the present embodiment, the first grouping unit divides the plurality of image sequences, and the image sequences include a plurality of continuous image frames. However, the single use of each image frame cannot effectively reflect the features of the lip shape of the person in the image. Therefore, it is necessary to fuse the plurality of image frames in each of the image sequences to obtain a feature map having a plurality of image frames at the same time. For different image sequences, a plurality of continuous feature maps, i.e., a plurality of first sequences, can be obtained.

[0046] The image processing module also includes: An image coding unit, wherein the image coding unit includes a plurality of image coding layers that are distributed in parallel, wherein the plurality of image coding layers correspond one-to-one to the plurality of fusion layers, and the plurality of image coding layers are configured to perform convolution coding on the plurality of first sequences respectively. Specifically, in this embodiment, after the feature fusion unit completes the fusion of image frames in all or part of the image sequences, feature coding is performed on the fused plurality of first sequences to obtain a plurality of corresponding image coding features.

[0047] Among them, convolution encoding is performed on each of the image sequences through image coding layers distributed in parallel in the image coding unit, one or more image sequences correspond to one image coding layer, and convolution encoding is performed on one or more image sequences through one image coding layer, so as to obtain the image coding features corresponding to each of the image sequences.

[0048] Furthermore, in some embodiments, the image encoding unit further includes a plurality of spatial attention layers using parallel distribution, and the plurality of spatial attention layers correspond one-to-one to the plurality of image encoding layers; The spatial attention layer is further configured to calculate a spatial attention weight according to the first sequence; The image encoding layer is also configured to perform convolution encoding according to the first sequence and the spatial attention weight to output the image encoding feature.

[0049] Specifically, in this embodiment, the image processing module is improved based on the residual architecture, in which the global average pooling layer is replaced by a spatial attention mechanism, which uses the spatial attention mechanism to find the most important parts of the face for processing and improve the feature expression of the key areas. This improvement enables the network to focus on the key areas of lip activity in the image and extract more representative visual features.

[0050] Further, in some embodiments, the image encoding unit further includes a channel attention layer; The image encoding unit is further configured to divide the first sequence into multiple subsequences, perform feature encoding processing on each subsequence, calculate the attention weights corresponding to different subsequences through the spatial attention layer, and fuse multiple feature encoding processing results based on the attention weight calculation results to obtain local feature encoding. Specifically, in this embodiment, by performing sequence division on the first sequence, calculating the attention weights, and encoding the local features according to the attention weights, a spatial attention mechanism is introduced in the calculation process of local features. The spatial attention mechanism focuses on the important points of each spatial position in the feature map, so that better presentation results are achieved when calculating the attention of the local area.

[0051] The image encoding unit is further configured to divide the local feature code into multiple subsequences, perform feature coding processing on each subsequence, calculate the attention weights corresponding to different subsequences through a preset spatial attention module, and fuse multiple feature coding processing results based on the attention weight calculation results to obtain local feature code. Specifically, in this embodiment, the processing of the first sequence by the above-mentioned image encoding unit is the same as that of performing local feature analysis on the feature code, but the difference from the above is that after completing the feature analysis of the local feature code, it is necessary to iteratively update the obtained feature code, perform cyclic local feature analysis on the updated feature code, and independently output the obtained local feature code after each round of local feature analysis.

[0052] The image encoding unit is also configured to calculate attention weights for the multiple local feature codes according to the channel attention layer, and perform feature processing based on the attention weight calculation results to obtain a global feature code; the multiple local feature codes are merged with the global feature code to obtain the image coding feature. Specifically, in this embodiment, after completing several rounds of local feature analysis, several local feature codes can be obtained, and then the image encoding unit needs to be used to perform further feature analysis on these local feature codes. The core purpose of this embodiment is to introduce a channel attention mechanism in the feature analysis process. The channel attention mechanism focuses on calculating the importance of different feature channels relative to the global, so it has a better presentation in calculating the effect of global features.

[0053] Figure 4 This is a schematic diagram of an audio processing module in a digital human voice lip synchronization method of the present application.

[0054] See also Figure 4 It can be seen that, further, in some embodiments, the audio processing module includes: The second grouping unit is configured to divide the input audio data into several continuous audio sequences. Specifically, in this embodiment, in order to better synchronize the voice and lip shape, a single audio frame cannot better reflect the features in the audio, and the corresponding image processing module extracts features from multiple image frames. Similarly, features cannot be extracted using only one frame of audio. In this embodiment, the audio data is divided by the second grouping unit to divide the audio data into several continuous image sequences, and each audio sequence includes several continuous audio sub-data, thereby providing an audio data basis for subsequent synchronization processing.

[0055] It should be noted that each audio sub-data corresponds to one image frame, that is, the number of image frames included in the image sequence is the same as the number of audio sub-data included in the audio sequence.

[0056] The audio processing module also includes: An audio coding unit, wherein the audio coding unit comprises a plurality of audio coding layers distributed in parallel, and the plurality of audio coding layers are configured to respectively perform convolution encoding on a plurality of audio sequences. Specifically, in this embodiment, convolution encoding is performed on each of the audio sequences through the audio coding layers distributed in parallel in the audio coding unit, one or more audio sequences correspond to one audio coding layer, and one or more audio sequences are convolutionally encoded through one audio coding layer, so as to obtain the audio coding features corresponding to each of the audio sequences.

[0057] The audio processing module uses a convolutional neural network (CNN) to process audio features. Through a series of convolutional layers, the encoder extracts key features from the audio signal and converts it into a high-dimensional feature vector.

[0058] Figure 5 This is a schematic diagram of a synchronization processing module in a digital human speech lip synchronization method of the present application.

[0059] See also Figure 5 It can be seen that, further, in some embodiments, the synchronization processing module includes: A feature projection module, wherein the feature projection module includes a plurality of linear layers, wherein the plurality of linear layers correspond one-to-one to the plurality of image coding layers and the plurality of audio coding layers, respectively; the plurality of linear layers are configured to map the image coding features and the audio coding features into a multimodal embedding space; A similarity calculation module, wherein the similarity calculation module includes a plurality of calculation layers, and the plurality of calculation layers correspond one-to-one to the plurality of linear layers; the plurality of calculation layers are configured to calculate the similarity of the image coding features and the audio coding features in the multimodal embedding space, and perform feature fusion on the image coding features and the audio coding features with the highest similarity to obtain the synchronization features.

[0060] Specifically, in this embodiment, after encoding, the audio coding features and image coding features are sent to a linear layer for projection through the feature projection module and mapped to a shared multimodal embedding space, that is, two sets of high-dimensional vectors are found to represent the audio coding features and image coding features respectively, and multimodal technology is used to project them to a shared potential representation, so that the reconstruction error of each modality in the common potential space is minimized, and these projection matrices are as sparse as possible. In this space, the network is trained by contrastive learning method, so that the embedded vectors of synchronized audio and video pairs are closer in space, while the asynchronous pairs are farther away.

[0061] Using optimization algorithms such as back propagation and gradient descent, the network parameters are adjusted to minimize the contrast loss function and improve the accuracy of the model in identifying lip sync. Ultimately, the network outputs embedding vectors that can represent lip shape and speech synchronization, which can be used for applications such as lip sync discrimination, speech enhancement, or digital human animation generation.

[0062] Exemplary Embodiment 1 In this exemplary embodiment, the digital human voice lip synchronization method can determine the audio-video synchronization between the mouth movement and voice in a digital human broadcast or singing video, and judge whether there is a problem of audio and video being out of sync in the video.

[0063] Exemplary Embodiment 2 In this exemplary embodiment, the digital human voice lip synchronization method can determine the character subject that speaks the current voice in a video of multiple digital humans reporting or singing.

[0064] Figure 6 This is a training flow chart of a lip synchronization model in a digital human speech lip synchronization method in the present application.

[0065] See also Figure 6 It can be seen that, further, in some embodiments, the training process of the lip synchronization model includes: S100, obtaining a training video, and extracting audio training data and image training data according to the training video; S200, performing model training on the audio training data, the image training data and the lip synchronization model; S300, using a loss function to converge the lip synchronization model until it meets preset model requirements.

[0066] Specifically, in this embodiment, before performing model training, it is necessary to first obtain the training video, and use the training video to perform model training on the lip synchronization model, and during the model training process, use the loss function to converge the lip synchronization model until the lip synchronization model meets the preset model requirements.

[0067] Figure 7 This is a flow chart of the convergence mode of the lip synchronization model in a digital human speech lip synchronization method of the present application.

[0068] See also Figure 7 It can be seen that, further, in some embodiments, the step of using the loss function to converge the lip synchronization model includes: S310, using a contrast loss function to perform contrast loss training on the synchronization feature. Specifically, in this embodiment, the loss function includes the contrast loss function, and the contrast loss function is used to perform contrast loss training on the synchronization feature to achieve preliminary loss convergence.

[0069] The step of using the loss function to converge the lip synchronization model also includes: S320, when the synchronization feature meets the requirements of the first preset model, the lip synchronization model is trained with positive samples and negative samples using a ternary loss function until it meets the requirements of the second preset model. Specifically, in the present embodiment, the loss function also includes the ternary loss function, and the ternary loss function is used to perform loss convergence again on the lip synchronization model that has completed loss convergence of the contrast loss function, so as to enhance the confidence of the model.

[0070] Figure 8 This application discloses a flowchart of positive sample training and negative sample training in a digital human speech lip synchronization method.

[0071] See also Figure 8 It can be seen that, further, in some embodiments, the step of performing positive sample training and negative sample training on the lip synchronization model includes: Acquire an anchor feature, a positive feature, and a negative feature; the anchor feature is a randomly selected image feature or audio feature; the positive feature is an image feature or audio feature that matches the anchor feature; the negative feature is an image feature or audio feature that does not match the anchor feature; Positive sample training is performed according to the anchor feature and the positive feature, and negative sample training is performed according to the anchor feature and the negative feature, until both the positive sample training and the negative sample training meet the second preset model requirement.

[0072] Specifically, in this embodiment, the loss convergence using the ternary loss function mainly includes the positive sample training and the negative sample training, wherein the positive sample training and the negative sample training are embodied in that the anchor feature and the positive feature are used for positive sample training, and the anchor feature and the negative feature are used for negative sample training, so as to achieve the loss convergence of the ternary loss function.

[0073] Exemplarily, the loss convergence of the lip synchronization model can be understood as the following two stages.

[0074] Stage 1: Contrastive Loss Training: ; In the initial stage of training, the model is trained with contrastive loss using only matched audio and video data pairs. Contrastive loss is widely used in unsupervised learning. This loss function is mainly used in dimensionality reduction, that is, the originally similar samples are still similar in the feature space after dimensionality reduction (feature extraction); while the originally dissimilar samples are still dissimilar in the feature space after dimensionality reduction. Similarly, this loss function can also well express the matching degree of paired samples. d represents the Euclidean distance of the two sample features, y is the label of whether the two groups of samples match, and m (margin) is the set threshold. The distance exceeding m is regarded as 0, that is, if the two dissimilar features are far apart, then the contrastive loss should be very low. The goal of this stage is to learn a feature space in which synchronized audio and video feature vectors are close to each other, while unsynchronized feature vectors are far apart. The most important thing is to use the similarity matrix to calculate the loss. The similarity matrix is ​​calculated at the batch level, which means that all sample pairs in a batch are considered at the same time. By constructing a similarity matrix, the model can compare the similarity of a sample with all other samples in the batch at the same time. This is an innovative idea in multimodal alignment methods. This contrastive learning strategy helps the model learn to distinguish between positive and negative samples because mismatched sample pairs provide rich negative sample information.

[0075] Phase 2: Fine-tuning Based on the first stage, the second stage of training fine-tunes the model by introducing mismatched audio and video data pairs to improve the generalization performance of the model. Mismatched audio and video data pairs include two situations: one is audio and video frames from the same video but not corresponding in time; the other is a random combination of audio and video frames from different videos. These mismatched data pairs are randomly selected with a certain probability to simulate the asynchrony that may be encountered in the real world.

[0076] ; In the second stage, a ternary loss function is used to simultaneously process matched and unmatched audio and video data pairs. The input is a triplet, including an anchor example, a positive example, and a negative example. The similarity calculation between samples is achieved by optimizing the distance between the anchor feature and the positive feature to be smaller than the distance between the anchor feature and the negative feature. The ternary loss function can simultaneously optimize the model's learning of positive samples for matched pairs and negative samples for unmatched pairs, thereby further distinguishing synchronized and unsynchronized data in the feature space. The design of this loss function helps the model better understand the complex relationship between audio and video data and improve its recognition ability when facing unseen data.

[0077] Through these two stages of training, the lip synchronization network can not only effectively learn the data in the training set, but also make accurate synchronization judgments on new, unseen audio and video data pairs, thereby showing better robustness and adaptability in practical applications.

[0078] In actual use, the speech lip synchronization network no longer participates in the training process of the digital human. The digital human generation network uses five consecutive frames generated by speech to form the input of the speech lip synchronization network. After passing through the speech lip synchronization network, the corresponding image multimodal latent space vector and speech multimodal latent space vector are obtained. These two vectors are used to calculate the cosine similarity to punish the images with inaccurate lip shapes generated by the generation network, so as to generate digital human videos with more accurate lip shapes.

[0079] This embodiment has the following advantages: The deep learning network structure with multimodal feature alignment is used to implement a video-driven digital human speech lip synchronization method. In real-time conditions, it is determined whether the continuous video frames are synchronized with the audio.

[0080] The dual-tower design and dual-stage training method achieves more accurate audio and video synchronization judgment, and also has good performance on videos recorded in the wild and better generalization.

[0081] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed.

Claims

1. A digital human voice lip synchronization method, characterized in that: The method comprises: Acquire a source video, and preprocess the source video to obtain audio data and image data; The lip synchronization model is used to synchronously process the audio coding features and the image coding features to obtain synchronization features; the audio coding features are output by the audio data through the lip synchronization model; the image coding features are obtained by image segmentation, feature fusion and feature coding by the lip synchronization model; the synchronization processing includes linear feature projection and feature similarity calculation; Video generation is performed according to the synchronization feature to obtain a target video.

2. A digital human voice lip synchronization method according to claim 1, characterized in that: The lip sync model includes: An image processing module, wherein the image processing module is configured to perform image coding according to the image data to obtain a plurality of image coding features; An audio processing module, wherein the audio processing module is configured to perform audio encoding according to the audio data to obtain a plurality of audio encoding features; A synchronization processing module is configured to perform audio and image synchronization processing according to the audio coding feature and the image coding feature to obtain the synchronization feature.

3. A digital human voice lip synchronization method according to claim 2, characterized in that: The image processing module comprises: a first grouping unit, wherein the first grouping unit is configured to divide the input image data into a plurality of continuous image sequences; each of the image sequences includes a plurality of continuous image frames; A feature fusion unit, wherein the feature fusion unit includes a plurality of fusion layers that are distributed in parallel, and the plurality of fusion layers are configured to perform feature fusion on the image frames in the plurality of image sequences respectively to obtain a plurality of corresponding first sequences; An image coding unit, wherein the image coding unit comprises a plurality of image coding layers using parallel distribution, wherein the plurality of image coding layers correspond one-to-one to the plurality of fusion layers, and the plurality of image coding layers are configured to respectively perform convolution coding on a plurality of the first sequences to obtain a plurality of corresponding image coding features.

4. A digital human voice lip synchronization method according to claim 3, characterized in that: The audio processing module comprises: a second grouping unit, wherein the second grouping unit is configured to divide the input audio data into a plurality of continuous audio sequences; each of the audio sequences includes a plurality of continuous audio sub-data, and each of the audio sub-data corresponds to one of the image frames; An audio encoding unit, wherein the audio encoding unit comprises a plurality of audio encoding layers that are distributed in parallel, and the plurality of audio encoding layers are configured to respectively perform convolution encoding on a plurality of the audio sequences to obtain a plurality of corresponding audio encoding features.

5. A digital human voice lip synchronization method according to claim 3, characterized in that: The image encoding unit further comprises a plurality of spatial attention layers using parallel distribution, and the plurality of spatial attention layers correspond one to one with the plurality of image encoding layers; The spatial attention layer is further configured to calculate a spatial attention weight according to the first sequence; The image encoding layer is also configured to perform convolution encoding according to the first sequence and the spatial attention weight to output the image encoding feature.

6. A digital human voice lip synchronization method according to claim 5, characterized in that: The image encoding unit also includes a channel attention layer; The image encoding unit is further configured to divide the first sequence into a plurality of subsequences, perform feature encoding processing on each subsequence, calculate attention weights corresponding to different subsequences through the spatial attention layer, and fuse a plurality of feature encoding processing results based on the attention weight calculation results to obtain local feature encoding; The local feature code is divided into multiple subsequences, feature coding processing is performed on each subsequence, attention weights corresponding to different subsequences are calculated by a preset spatial attention module, and multiple feature coding processing results are fused based on the attention weight calculation results to obtain the local feature code; Iteratively updating, and independently outputting the local feature codes obtained in each round to obtain multiple local feature codes; Attention weights are calculated for the multiple local feature codes according to the channel attention layer, and feature processing is performed based on the attention weight calculation results to obtain a global feature code; the multiple local feature codes are fused with the global feature code to obtain the image coding feature.

7. A digital human voice lip synchronization method according to claim 4, characterized in that: The synchronization processing module comprises: A feature projection module, wherein the feature projection module includes a plurality of linear layers, wherein the plurality of linear layers correspond one-to-one to the plurality of image coding layers and the plurality of audio coding layers, respectively; the plurality of linear layers are configured to map the image coding features and the audio coding features into a multimodal embedding space; A similarity calculation module, wherein the similarity calculation module includes a plurality of calculation layers, and the plurality of calculation layers correspond one to one with the plurality of linear layers; the plurality of calculation layers are configured to calculate the similarity of the image coding features and the audio coding features in the multimodal embedding space, and perform feature fusion on the image coding features and the audio coding features with the highest similarity to obtain the synchronization features.

8. A digital human voice lip synchronization method according to claim 1, characterized in that: The training process of the lip synchronization model includes: Obtaining a training video, and extracting audio training data and image training data based on the training video; Performing model training on the audio training data, the image training data and the lip synchronization model; The lip synchronization model is converged using a loss function until it meets the preset model requirements.

9. A digital human voice lip synchronization method according to claim 8, characterized in that: The step of using the loss function to converge the lip synchronization model comprises: Performing contrast loss training on the synchronization features using a contrast loss function; When the synchronization feature meets the requirements of the first preset model, the lip synchronization model is trained with positive samples and negative samples using a ternary loss function until it meets the requirements of the second preset model.

10. A digital human voice lip synchronization method according to claim 9, characterized in that: The step of performing positive sample training and negative sample training on the lip synchronization model comprises: Acquire an anchor feature, a positive feature, and a negative feature; the anchor feature is a randomly selected image feature or audio feature; the positive feature is an image feature or audio feature that matches the anchor feature; the negative feature is an image feature or audio feature that does not match the anchor feature; Positive sample training is performed according to the anchor feature and the positive feature, and negative sample training is performed according to the anchor feature and the negative feature, until both the positive sample training and the negative sample training meet the second preset model requirement.

11. A digital human voice lip synchronization method according to claim 1, characterized in that: The step of preprocessing the source video comprises: Extracting a source image according to the source video, and sequentially performing image framing processing, face detection processing, face region cropping processing, occlusion detection processing, and image covering processing on the source image to obtain the image data; The source audio is extracted according to the source video, and the source audio is subjected to audio frame processing and audio denoising processing in sequence to obtain the audio data.

Images