Lip shape synchronization method and system based on multiple reference frames and controllable style

Abstract

The invention discloses a lip shape synchronization method and system based on multiple reference frames and controllable styles, and belongs to the field of video generation. Acquiring speaker video data and driving audio data; randomly selecting two groups of video frames from the video as a first multi-reference picture and a second multi-reference picture respectively, and regarding driving key points of the reference pictures as reference key points; extracting lip motion style features of a speaker from the video, combining the driving audio features and reference key points of the first multi-reference image, and utilizing a sparse-to-fine audio-to-key point module to predict fine key points; in combination with the second multi-reference image, the reference key points of the second multi-reference image and the fine key points predicted by the audio-to-key-point module, a multi-scale aggregation proportion is introduced into the key point-to-video module to generate a lip-shaped synchronization frame; in the generation task, the lip motion style characteristics of the speaker can be extracted from the original video or the lip motion style characteristics of the speaker can be directly given, so that style-controllable lip shape synchronization is realized.

Description

Technical Field

[0001] The present invention relates to the field of video generation, and in particular to a lip synchronization method and system based on multiple reference frames and controllable styles. Background Art

[0002] At present, the task of generating talking heads in video has attracted wide attention and has become an important field of artificial intelligence. Lip synchronization is a common task in this field, which aims to modify the lip shape of the speaker in another given audio and video through the specified driving audio so that the lip shape is synchronized with the driving audio. Lip synchronization methods and systems have been widely used in many application scenarios such as video dubbing, cross-language education, and e-commerce live broadcasting.

[0003] The existing mainstream lip synchronization methods and technologies are mainly two-stage key point methods, that is, using the key points extracted by the pre-training tool as the intermediate representation, divided into two-stage generation sub-tasks: audio to key points and key to video, and training two sub-models respectively. Compared with tasks without speaking style input such as single-image drive, there is naturally richer information in the lip synchronization task, such as the speaker's speaking style characteristics and texture information brought by more reference frames. However, the existing methods do not fully utilize this information, and there are two main problems: first, the speaking style characteristics of the speaker in the original video are ignored; second, only a single reference frame information is used for lip synchronization generation, or simply the information of multiple reference frames is averaged. Summary of the invention

[0004] In order to overcome the above problems, the present invention proposes a lip synchronization method and system based on multiple reference frames and controllable style, which makes full use of the speaker style characteristic information and multiple reference frame information in the original video to achieve style-controllable and high-fidelity lip synchronization video generation.

[0005] The specific technical solution adopted by the present invention is:

[0006] In a first aspect, the present invention proposes a lip synchronization method based on multiple reference frames and controllable style, comprising the following steps:

[0007] S1, obtaining speaker video data and driving audio data, extracting driving audio features of each audio frame and driving key points of each video frame;

[0008] S2, randomly selecting two groups of video frames from the video as the first multi-reference image and the second multi-reference image, and taking the driving key points of the reference image as the reference key points;

[0009] S3, extracts the speaker's lip movement style features from the video, combines the driving audio features and the reference key points of the first multi-reference graph, and uses the audio to key point module to predict the fine key points;

[0010] S4, combining the second multiple reference images and their reference key points, and the refined key points predicted by the audio-to-key point module, and generating a lip sync frame using the key point-to-video module;

[0011] S5, calculating the loss and updating the module according to the prediction results of the audio to key point module and the generation results of the key point to video module;

[0012] S6, given the driving audio and the original video, extract the speaker's lip movement style features from the original video or directly give the speaker's lip movement style features, use the trained audio to key point module and key point to video module to generate a synthetic video that is lip-synced with the given driving audio, and complete the lip synchronization task.

[0013] Furthermore, the first multiple reference graph and the second multiple reference graph are independent of each other.

[0014] Furthermore, the driving key points include 3D driving key points and 2D driving key points, and the reference key points include 3D reference key points and 2D reference key points; the refined key points predicted by the audio to key point module are refined 2D driving key points.

[0015] Furthermore, the lip movement style feature is a scalar, which is composed of lip width and lip speed. The method of extracting the speaker's lip movement style feature from the video is as follows:

[0016] Select lip key points from the 3D driving key points of the speaker in each frame of the video, including upper lip key points and lower lip key points;

[0017] The average value of the y-coordinate difference between the upper lip key points and the lower lip key points is used to represent the lip width;

[0018] The lip velocity is represented by the time series first-order mean difference of the absolute value of the y-coordinate difference between the upper lip key points and the lower lip key points.

[0019] Furthermore, the audio to key point module includes:

[0020] a sparse 3D driving keypoint predictor, which takes the driving audio features of each audio frame, the speaker's lip movement style features, and the 3D reference keypoints of the first multi-reference graph as input, and outputs predicted sparse 3D driving keypoints;

[0021] A refined 2D driving key point predictor takes the 2D reference key points of the first multiple reference images and the sparse 3D driving key points as inputs, and outputs predicted refined 2D driving key points.

[0022] Furthermore, the sparse 3D driving key point predictor includes a reference key point encoder, a lip width embedding layer, a lip velocity embedding layer, an audio encoder and a driving key point decoder, and the calculation process includes:

[0023] The reference key point encoder encodes the 3D reference key points of the first multi-reference image to obtain encoded 3D reference key point features;

[0024] The lip width embedding layer and the lip velocity embedding layer encode and embed the lip width and lip velocity of the speaker's lip movement style features respectively, and obtain the lip movement style embedding features composed of lip width embedding and lip velocity embedding;

[0025] The audio encoder encodes the driving audio feature of each audio frame to obtain an audio coding feature;

[0026] The encoded 3D reference key point features and lip motion style embedding features are expanded to the same dimension as the audio encoding features, and then connected with the audio encoding features to obtain sparse 3D driving key point features; the number of frames of the sparse 3D driving key point features is consistent with the number of audio frames;

[0027] The sparse 3D driving keypoint features are decoded by the driving keypoint decoder to obtain the predicted sparse 3D driving keypoints.

[0028] Furthermore, the refined 2D driving key point predictor includes a reference key point encoder, a linear layer and a driving key point decoder, and the calculation process includes:

[0029] The reference key point encoder encodes the 2D reference key points of the first multi-reference image to obtain encoded 2D reference key point features;

[0030] The predicted sparse 3D driving keypoints are embedded into sparse 3D driving keypoint embeddings by a linear layer;

[0031] The encoded 2D reference key point features are expanded to the same dimension as the sparse 3D driving key point embedding, and then concatenated with the sparse 3D driving key point embedding to obtain refined 2D driving key point features; the number of frames of the refined 2D driving key point features is consistent with the number of audio frames;

[0032] The refined 2D driving keypoint features are decoded by the driving keypoint decoder to obtain the predicted refined 2D driving keypoints.

[0033] Further, step S4 includes:

[0034] S4-1, warping the feature map of the second reference image of each frame according to the 2D reference key points of the second reference image of each frame and the fine 2D driving key points of the current frame to generate a multi-warped feature map;

[0035] S4-2, using the fine 2D driving key points of the current frame, the 2D reference key points of all the second reference images, and the multi-distortion feature map, calculating a multi-scale aggregation ratio of the multi-distortion feature map, wherein the multi-scale aggregation ratio includes a frame scale ratio and a pixel scale ratio; and using the multi-scale aggregation ratio to aggregate texture information of the multi-distortion feature map to obtain an aggregated feature map;

[0036] S4-3, decoding the aggregated feature map to obtain a generated image, using the driving key points of the original frame image to generate a smooth lower half face mask, and using the smooth lower half face mask to fuse the lower half face of the speaker in the generated image with the original frame image to obtain a lip sync frame image of the current frame;

[0037] S4-4, repeat S4-1 to S4-3, and use the next frame of fine 2D driving key points to generate the lip synchronization frame image of the next frame until all the frames of fine 2D driving key points are traversed.

[0038] In a second aspect, the present invention provides a lip synchronization system based on multiple reference frames and controllable styles, which is used to implement the above-mentioned lip synchronization method based on multiple reference frames and controllable styles.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention realizes style-controllable lip synchronization based on multiple reference frames. By giving driving audio and original video, the speaker's lip movement style features can be extracted from the original video, or the speaker's lip movement style features can be directly given, supporting any explicit or implicit speaker style specification; by going from sparse to fine audio to key point modules, the difficulty of generating results in one step is reduced, making the training effect more stable; by introducing multi-scale aggregation ratios in the key point to video module to generate lip synchronization, it can effectively utilize multi-reference frame information to generate higher fidelity video images. In summary, style-controllable and high-fidelity lip synchronization video generation is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a schematic diagram of a lip synchronization method based on multiple reference frames and controllable style according to an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of the structure of a sparse 3D driven key point predictor shown in an embodiment of the present invention;

[0043] Figure 3 4 is a schematic diagram of the structure of a fine 2D driving key point predictor shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0045] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0046] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0047] See also Figure 1 The present invention proposes a lip synchronization method based on multiple reference frames and controllable style, which mainly includes the following steps:

[0048] Step 1, obtaining the speaker's audio and video data, extracting the 3D driving key points and the camera estimation matrix of the speaker's face in the video frame, and projecting the 3D driving key points using the camera estimation matrix to obtain the 2D driving key points; randomly selecting some video frames to form a first multi-reference image, and using the 3D driving key points and 2D driving key points of the selected video frames as the 3D reference key points and 2D reference key points of the first multi-reference image;

[0049] And, extracting driving audio features of each audio frame.

[0050] Step 2: Calculate the lip movement style features of the speaker in the video based on the 3D driving key points of all video frames.

[0051] Step 3, combining the speaker's lip movement style characteristics, the driving audio characteristics of each audio frame, the 3D reference key points and the 2D reference key points of the first multi-reference image, and using the audio to key point module to predict the fine 2D driving key points; the number of frames corresponding to the fine 2D driving key points is consistent with the number of audio frames and corresponds one to one.

[0052] Step 4: randomly select some video frames from the original video frames to form a second multi-reference image, obtain the second multi-reference image feature map after encoding, and use the same method to obtain the 2D reference key points of the second multi-reference image.

[0053] Step 5: Combine the second reference image and its 2D reference key points, and the refined 2D driving key points predicted by the audio to key point module, and generate a lip sync frame using the key point to video module.

[0054] Step 6: Train the audio to keypoint module and the keypoint to video module.

[0055] Step 7: Given the driving audio and the original video, extract the lip movement style features of the speaker in the video from the original video or directly specify the lip movement style features of the speaker, and use the trained audio to key point module and key point to video module to generate a synthetic video that is lip-synced with the driving audio to complete the lip synchronization task.

[0056] In the above step 1, the 2D and 3D key point information of the face in each frame of the video image in the continuous video frames is extracted by using a pre-trained face key point detection tool. An optional method is: using a pre-trained 3D face key point detection tool including but not limited to Mediapipe, perform 3D key point detection on the face part of each frame of the video image to obtain 3D driving key point coordinates and camera estimation matrix; perform 2D projection on the 3D driving key point coordinates through the camera estimation matrix to obtain 2D driving key point coordinates. Here, each video frame image corresponds to a set of 3D driving key points and a set of 2D driving key points.

[0057] Similarly, the driving audio features of each audio frame in the continuous audio frames are extracted by a pre-trained audio feature extraction tool. An optional pre-trained audio feature extraction tool is HuBERT.

[0058] The present invention also introduces multiple reference frames, where the multiple reference frames are extracted from video frames in the input audio and video data, and the extraction method is random extraction. The 3D driving key points and 2D driving key points corresponding to the selected video frames are used as the 3D reference key points and 2D reference key points of the first multiple reference graphs, where each reference graph corresponds to a group of 3D reference key points and a group of 2D reference key points.

[0059] In the above step 2, an optional method for calculating the lip motion style features based on the 3D driving key points is as follows:

[0060] The lip key points in the 3D driving key points of the speaker in the video are used to extract the estimated and quantized lip movement style features, including two speaking style attributes: lip width and lip speed. Specifically, the lip width is measured by the average value of the y-coordinate difference between the upper lip and lower lip key points, and the lip speed is measured by the time series first-order average difference of the absolute value of the y-coordinate difference between the upper lip and lower lip key points, that is:

[0061]

[0062]

[0063] in, and Indicates lip width and lip speed, is the current The y coordinate difference between the upper and lower lip key points at the time step, is the total time steps in a given video. Lip width and lip velocity together constitute the speaker's speaking style attributes. The lip movement style features calculated here correspond to the entire audio and video.

[0064] During model training, the real lip movement style features are extracted as input according to the above method; during model inference, the user can use the lip movement style features extracted from the original video, or freely specify the values ​​of the lip movement style features. When the lip movement style features extracted from the original video are used in the model inference stage, the lip movement style in the generated lip-sync video is consistent with the input video; since the lip movement style features obtained by the above calculation are a scalar, the style of the generated lip-sync video can be modified by the values ​​of the lip movement style features freely specified by the user.

[0065] In the above step 3, the audio to key point module includes a sparse 3D driving key point predictor and a fine 2D driving key point predictor, wherein the sparse 3D driving key point predictor takes the driving audio features of each audio frame, the speaker's lip movement style features, and the 3D reference key points of the first multiple reference images as input, and outputs predicted sparse 3D driving key points; the fine 2D driving key point predictor takes the 2D reference key points of the first multiple reference images and the sparse 3D driving key points as input, and outputs predicted fine 2D driving key points.

[0066] Here, in the sparse 3D driving key point predictor, the 3D reference key points of the first multiple reference images are encoded and expanded to the same dimension as the driving audio features of each audio frame, so the number of frames of the output predicted sparse 3D driving key points is consistent with the number of audio frames, that is, each frame of audio corresponds to a frame of sparse 3D driving key points, and the one frame of sparse 3D driving key points refers to a group of key points corresponding to a frame of video image. Similarly, in the fine 2D driving key point predictor, the 2D reference key points of the first multiple reference images are encoded and expanded to the same dimension after the sparse 3D driving key point encoding is embedded, so the number of frames of the output predicted fine 2D driving key points is also consistent with the number of audio frames, that is, each frame of audio corresponds to a frame of fine 2D driving key points, and the one frame of fine 2D driving key points refers to a group of key points corresponding to a frame of video image.

[0067] Through the multi-level model structure from sparse to fine, the difficulty of the model generating results in one step is reduced, making the training effect more stable.

[0068] like Figure 2 As shown, in a specific implementation of the present invention, the sparse 3D driving key point predictor includes a reference key point encoder, a lip width embedding layer, a lip velocity embedding layer, an audio encoder and a driving key point decoder; the calculation process of the sparse 3D driving key point predictor is as follows:

[0069] 3-1a) The reference key point encoder encodes the 3D reference key points of the first multi-reference image to obtain encoded 3D reference key point features;

[0070] 3-2a) The lip width embedding layer and the lip velocity embedding layer encode and embed the lip width and lip velocity of the speaker's lip movement style features respectively, and obtain the lip movement style embedding features composed of lip width embedding and lip velocity embedding;

[0071] In this embodiment, the lip width embedding is obtained by multiplying a scalar and a learnable vector, that is, f_amp=s_amp×e_amp, where s_amp is the scalar value of the lip width, e_amp is the learnable vector of the lip width embedding layer, and f_amp is the lip width embedding; the calculation method of the lip velocity embedding is the same and will not be explained again.

[0072] 3-3a) The audio encoder encodes the driving audio features of each audio frame to obtain audio coding features;

[0073] 3-4a) The encoded 3D reference key point features and the lip motion style embedded features are extended to the same length as the audio encoding features, and then connected with the audio encoding features to obtain sparse 3D driving key point features; here, the number of sparse 3D driving key point feature frames is consistent with the number of audio frames; here, the expansion method adopts a repeated filling method, which is a well-known technology in the art;

[0074] 3-5a) The sparse 3D driving key point features are decoded by the driving key point decoder to obtain the predicted sparse 3D driving key points. The sparse 3D driving key points mainly include the lip key points and a sparse part of the facial contour points.

[0075] In this embodiment, the key point decoder can adopt a conventional decoder structure, for example, it can be a stack of N residual modules, each residual module includes a layer normalization layer, a 1D convolution layer, a PReLu activation function, a final 1D convolution layer and a residual connection from input to output in sequence.

[0076] like Figure 3As shown, in a specific implementation of the present invention, the fine 2D driving key point predictor includes a reference key point encoder, a linear layer and a driving key point decoder; the calculation process of the fine 2D driving key point predictor is as follows:

[0077] 3-1b) The reference key point encoder encodes the 2D reference key points of the first multi-reference image to obtain encoded 2D reference key point features;

[0078] 3-2b) The predicted sparse 3D driving key points are embedded into sparse 3D driving key point embeddings by the linear layer;

[0079] 3-3b) The encoded 2D reference key point features are expanded to the same dimension as the sparse 3D driving key point embedding, and concatenated with the sparse 3D driving key point embedding to obtain the refined 2D driving key point features; the number of frames of the refined 2D driving key point features is consistent with the number of audio frames;

[0080] 3-4b) The refined 2D driving key point features are decoded by the driving key point decoder to obtain the predicted refined 2D driving key points.

[0081] The second multi-reference image in step 4 is not related to the first multi-reference image in step 1, and the two can be different. The method for obtaining the 2D reference key points of the second multi-reference image is the same as that of the first multi-reference image. Figure 1 Similarly, the 2D driving key points corresponding to the selected video frame are directly used as the 2D reference key points of the second multiple reference images, where each reference image corresponds to a group of 2D reference key points.

[0082] In the above step 5, an optional implementation process is as follows:

[0083] 5-1) According to the 2D reference key points of the second reference image of each frame and the refined 2D driving key points of the current frame, the feature map of the second reference image of each frame is distorted into a feature map consistent with the driving key points, generating a distorted feature map of multiple reference frames, i.e., a multi-distorted feature map.

[0084] In a specific implementation of the present invention, a method for obtaining a multi-distortion feature map is: including but not limited to using the dense action hourglass network disclosed in the paper "Thin-Plate Spline Motion Model for Image Animation", taking the 2D reference key points and feature maps of each frame of the reference image, and the fine 2D driving key points of any frame as input, and taking the distortion feature map of each frame of the reference image as output. For convenience, the distortion feature map of multiple frames of reference images is referred to as a multi-distortion feature map.

[0085] 5-2) Utilizing the refined 2D driving key points of the current frame, the 2D reference key points of all second reference images, and the multi-distortion feature map, a multi-scale aggregation ratio of the multi-distortion feature map is calculated, wherein the multi-scale aggregation ratio includes a frame scale ratio and a pixel scale ratio; and the texture information of the multi-distortion feature map is aggregated using the multi-scale aggregation ratio to obtain an aggregated feature map.

[0086] In a specific implementation of the present invention, the frame scale ratio calculation process includes:

[0087] Using the selected fine 2D driving key points and the 2D reference key points of all second reference images, calculate the query matrix corresponding to each frame of the second reference image , key-value matrix , value matrix :

[0088]

[0089]

[0090]

[0091] in, Represents three linear transformation matrices, and Represent the selected fine 2D driving key points and the 2D reference key points of the second reference image of the i-th frame respectively;

[0092] The cross attention value of each frame reference image is calculated according to the query matrix, key matrix, and value matrix, and the frame scale ratio is obtained through softmax processing. , here It is a set of numbers between 0 and 1, which is consistent with the frame number of the second reference image.

[0093] The pixel scale is calculated by the cross-class attention of Adaptive Instance Normalization (AdaIN), which includes:

[0094] Calculate the query matrix corresponding to each frame of the reference image , key-value matrix , value matrix :

[0095]

[0096]

[0097]

[0098] in, is the modulated convolutional layer, is the distorted feature map of the i-th frame reference image, is the convolution kernel, which is obtained from the reference key point.

[0099] Will , Expand to The same shape, according to the expanded query matrix, key matrix, value matrix to calculate the cross attention value of each frame reference image, after softmax processing to get the pixel scale ratio , here It is a set of numbers between 0 and 1, which is consistent with the second reference frame number. This ratio has different values ​​between different pixels, reflecting the local similarity between the driving key point at the fine granularity of the pixel level and the pixel area.

[0100] The process of using multi-scale aggregation ratios to aggregate the texture information of multi-distorted feature maps to obtain aggregated feature maps is as follows:

[0101]

[0102] in, Represents the multi-scale aggregation ratio after mixing, which is consistent with the number of frames of the second reference image; represents the frame scale ratio, represents the pixel scale ratio, Represents a preset frame weight. This embodiment uses .

[0103] The information of multiple feature maps is aggregated into a single feature map by the multi-scale attention mixer to reduce the amount of subsequent calculations. The aggregation of multiple twisted feature maps is a weighted average of the pixels of each feature map according to a specific ratio. The aggregation ratio is obtained by mixing the frame scale ratio and the pixel scale ratio, so it is called multi-scale.

[0104] 5-3) Decode the aggregated feature map to obtain a generated image, and fuse the speaker's lower face in the generated image with the original frame image to obtain the lip sync frame image of the current frame;

[0105] In a specific implementation of the present invention, the lower half face of the speaker in the generated image is fused with the original frame image as follows:

[0106] A convex hull is formed using the key points related to the lower half of the face in the original video image, and the convex hull area is regarded as a hard lower half face mask. In the boundary area of ​​the convex hull, a Gaussian model of a certain strength is used for feathering to obtain a smooth lower half face mask. Finally, use the smoothed lower face mask to blend the original frame image with the generated image:

[0107]

[0108] in, is the mixed result image, i.e., the final lip sync frame image; and are the generated image and the original image respectively.

[0109] 5-4) Repeat steps 5-1) to 5-3) to generate the lip sync frame image of the next frame using the next frame of fine 2D driving key points until all frames of fine 2D driving key points are traversed.

[0110] In step 6 above, an optional way to train the audio to keypoint module and the keypoint to video module is as follows:

[0111] For the training of the audio to keypoint module, the loss function is calculated as:

[0112]

[0113] in, is the loss function of the audio to keypoint module, is the L1 loss between the true value of the face 2D driving key points and the predicted fine 2D driving key points, is the L1 loss between the true value of the face 3D driving key point and the predicted sparse 3D driving key point, It is the L1 loss between the true value and the predicted value of the lip movement style feature.

[0114] Here, the true value of the 3D driving key point and the true value of the 2D driving key point are the 3D driving key points and 2D driving key points of each video frame image obtained through step 1; as mentioned earlier, the frame number of the predicted sparse 3D driving key points and the predicted fine 2D driving key points is consistent with the audio frame number, and the audio and video in the training stage are corresponding; the true value of the lip movement style feature is calculated through step 2, and the predicted value of the lip movement style feature is calculated using the predicted sparse 3D driving key points using the same method as step 2.

[0115] For the training of the keypoint-to-video module, the loss function includes two parts: the whole image and the lip region.

[0116] For the entire image, the loss function is calculated as:

[0117]

[0118] in, is the full-image loss function from keypoint to video module, is the mean square error loss between the generated image and the original frame image, It is the mean square error loss between the generated image and the original frame image in the perceptual space of the pre-trained model VGG-19.

[0119] For the lip area, an additional loss enhancement is introduced. The lip area is extracted from the whole image, and the mean square error loss of the lip area of ​​the generated image and the lip area of ​​the original frame image is calculated by the same sampling method. is the mean square error loss between the lip area of ​​the generated image and the lip area of ​​the original frame image in the pre-trained model VGG-19 perception space; the lip area loss function of the key point to video module is recorded as .

[0120] The complete training loss function for the keypoint-to-video module is:

[0121]

[0122] in, is the loss function of the keypoint to video module.

[0123] In the above step 7, by giving a driving audio and an original video, the lip movement style features of the speaker in the video are extracted from the original video or the lip movement style features of the speaker are directly specified, and the trained audio to key point module and key point to video module are used to generate a synthetic video that is lip-synced with the driving audio, thereby completing the lip synchronization task.

[0124] Since the lip movement style feature is a scalar, the style of the generated lip synchronization video can be modified by the value of the lip movement style feature freely specified by the user.

[0125] The above method is applied to the following embodiments to demonstrate the technical effects of the present invention, and the specific steps in the embodiments are not repeated here.

[0126] The present invention conducts experiments on the mainstream lip sync task test set HDTF (High-Definition Talking Face), and 20 random samples in the test set are selected as the test sample set for the comparative experiment. In order to objectively evaluate the performance of the present invention, the present invention is not trained on the HDTF dataset distribution.

[0127] This comparative test uses mainstream quantitative indicators for lip sync tasks, including indicators in multiple dimensions such as image quality, key point reconstruction accuracy, and character identity maintenance, including PSNR, SSIM, LPIPS, FID, CSIM, and LipLDM. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to evaluate the similarity between the generated video and the ground truth video. Perceptual image patch similarity (LPIPS) and Frechet Inception distance (FID) are used to evaluate the distance in the latent feature space, which represents the visual quality of the generation. In order to better evaluate the ability to maintain identity, the cosine similarity (CSIM) of the features extracted by ArcFace is also evaluated. The lip key point distance (LipLDM) is calculated to measure audio-visual lip synchronization. It should be noted that the lip key points extracted by Mediapipe on the generated results and the real video are used to calculate LipLDM, and the distance metric uses L1 distance.

[0128] The benchmark models used for comparison include MakeItTalk, Wav2Lip, Wav2Lip-GAN, PC-AVS, VideoRetalking, IP-LAP, and DINet. Similar to the proposed method, IP-LAP and DINet use multiple reference frames to enhance the prior knowledge of the face. The official open source implementations and pre-trained models of these baselines are used for inference. Some benchmark models, such as PC-AVS, only generate cropped or resized speakers. For these methods, the pre-processed source video is regarded as the ground truth.

[0129] According to the steps described in the specific implementation manner, the experimental results obtained are shown in Table 1.

[0130] Table 1: Lip sync comparison test results obtained by the present invention for HDTF dataset

[0131]

[0132] It can be found from Table 1 that, in general, the present invention trained on publicly accessible datasets has achieved better performance in evaluation results such as PSNR, SSIM, LPIPS, FID and CSIM, as shown in Table 1. IP-LAP uses an audio to key point network trained on two large datasets, so it performs better on LipLDM. Due to the design of the key point to video module and the inference reference selection strategy, the FID and LPIPS scores of the present invention are significantly better than previous methods. For example, the FID of the present invention is about 1.45, while the FID of the baseline model is above 4.9, which shows the high visual quality of the key point to video module. In addition, the CSIM of the present invention reaches a level above 0.95, while the best extreme model is only around 0.92, showing the high identity preservation ability of the audio to key point module of the present invention.

[0133] Based on the same inventive concept, a lip synchronization system based on multiple reference frames and controllable style is also provided in this embodiment, and the system is used to implement the above embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0134] In this embodiment, a lip synchronization system based on multiple reference frames and controllable style includes:

[0135] An audio and video data preprocessing module is used to obtain speaker video data and driving audio data, extract driving audio features of each audio frame and driving key points of each video frame; randomly select two groups of video frames from the video as the first multi-reference image and the second multi-reference image, and regard the driving key points of the reference image as reference key points;

[0136] An audio-to-keypoint module is used to extract the speaker's lip movement style features from the video, and predict fine keypoints by combining the driving audio features and the reference keypoints of the first multi-reference graph;

[0137] a keypoint-to-video module for combining the second multiple reference images and their reference keypoints, and the refined keypoints predicted by the audio-to-keypoint module, to generate a lip sync frame;

[0138] A training module, which is used to calculate the loss and update the module based on the prediction results of the audio to key point module and the generation results of the key point to video module;

[0139] The lip synchronization generation module is used to extract the speaker's lip movement style features from the original video or directly give the speaker's lip movement style features for the given driving audio and original video, and use the trained audio to key point module and key point to video module to generate a synthetic video that is lip-synchronized with the given driving audio to complete the lip synchronization task.

[0140] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0141] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0142] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A lip synchronization method based on multiple reference frames and controllable style, characterized in that: The following steps are involved: S1, obtaining speaker video data and driving audio data, extracting driving audio features of each audio frame and driving key points of each video frame; S2, randomly selecting two groups of video frames from the video as the first multi-reference image and the second multi-reference image, and taking the driving key points of the reference image as the reference key points; S3, extracts the speaker's lip movement style features from the video, combines the driving audio features and the reference key points of the first multi-reference graph, and uses the sparse to fine audio to key point module to predict the fine key points; S4, combining the second multi-reference image and its reference key points, and the refined key points predicted by the audio-to-key point module, and introducing a multi-scale aggregation ratio in the key point-to-video module to generate a lip sync frame; S5, calculating the loss and updating the module according to the prediction results of the audio to key point module and the generation results of the key point to video module; S6, given the driving audio and the original video, extract the speaker's lip movement style features from the original video or directly give the speaker's lip movement style features, use the trained audio to key point module and key point to video module to generate a synthetic video that is lip-synced with the given driving audio, and complete the lip synchronization task.

2. The lip synchronization method based on multiple reference frames and controllable style according to claim 1, characterized in that: The driving key points include 3D driving key points and 2D driving key points, and the reference key points include 3D reference key points and 2D reference key points; the fine key points predicted by the audio to key point module are fine 2D driving key points.

3. The lip synchronization method based on multiple reference frames and controllable style according to claim 2, characterized in that: The lip movement style feature is a scalar, which is composed of lip width and lip speed. The method for extracting the speaker's lip movement style feature from the video is as follows: Select lip key points from the 3D driving key points of the speaker in each frame of the video, including upper lip key points and lower lip key points; The average value of the y-coordinate difference between the upper lip key points and the lower lip key points is used to represent the lip width; The lip velocity is represented by the time series first-order mean difference of the absolute value of the y-coordinate difference between the upper lip key points and the lower lip key points.

4. The lip synchronization method based on multiple reference frames and controllable style according to claim 2, characterized in that: The audio to key point module includes: a sparse 3D driving keypoint predictor, which takes the driving audio features of each audio frame, the speaker's lip movement style features, and the 3D reference keypoints of the first multi-reference graph as input, and outputs predicted sparse 3D driving keypoints; A refined 2D driving key point predictor takes the 2D reference key points of the first multiple reference images and the sparse 3D driving key points as inputs, and outputs predicted refined 2D driving key points.

5. The lip synchronization method based on multiple reference frames and controllable style according to claim 4, characterized in that: The sparse 3D driving key point predictor includes a reference key point encoder, a lip width embedding layer, a lip velocity embedding layer, an audio encoder and a driving key point decoder. The calculation process includes: The reference key point encoder encodes the 3D reference key points of the first multi-reference image to obtain encoded 3D reference key point features; The lip width embedding layer and the lip velocity embedding layer encode and embed the lip width and lip velocity of the speaker's lip movement style features respectively, and obtain the lip movement style embedding features composed of lip width embedding and lip velocity embedding; The audio encoder encodes the driving audio feature of each audio frame to obtain an audio coding feature; The encoded 3D reference key point features and lip motion style embedding features are expanded to the same dimension as the audio encoding features, and then connected with the audio encoding features to obtain sparse 3D driving key point features; the number of frames of the sparse 3D driving key point features is consistent with the number of audio frames; The sparse 3D driving keypoint features are decoded by the driving keypoint decoder to obtain the predicted sparse 3D driving keypoints.

6. The lip synchronization method based on multiple reference frames and controllable style according to claim 4, characterized in that: The refined 2D driving key point predictor includes a reference key point encoder, a linear layer and a driving key point decoder, and the calculation process includes: The reference key point encoder encodes the 2D reference key points of the first multi-reference image to obtain encoded 2D reference key point features; The predicted sparse 3D driving keypoints are embedded into sparse 3D driving keypoint embeddings by a linear layer; The encoded 2D reference key point features are expanded to the same dimension as the sparse 3D driving key point embedding, and then concatenated with the sparse 3D driving key point embedding to obtain refined 2D driving key point features; the number of frames of the refined 2D driving key point features is consistent with the number of audio frames; The refined 2D driving keypoint features are decoded by the driving keypoint decoder to obtain the predicted refined 2D driving keypoints.

7. The lip synchronization method based on multiple reference frames and controllable style according to claim 2, characterized in that: Step S4 includes: S4-1, warping the feature map of the second reference image of each frame according to the 2D reference key points of the second reference image of each frame and the fine 2D driving key points of the current frame to generate a multi-warped feature map; S4-2, using the fine 2D driving key points of the current frame, the 2D reference key points of all the second reference images, and the multi-distortion feature map, calculating a multi-scale aggregation ratio of the multi-distortion feature map, wherein the multi-scale aggregation ratio includes a frame scale ratio and a pixel scale ratio; and using the multi-scale aggregation ratio to aggregate texture information of the multi-distortion feature map to obtain an aggregated feature map; S4-3, decoding the aggregated feature map to obtain a generated image, using the driving key points of the original frame image to generate a smooth lower half face mask, and using the smooth lower half face mask to fuse the lower half face of the speaker in the generated image with the original frame image to obtain a lip sync frame image of the current frame; S4-4, repeat S4-1 to S4-3, and use the next frame of fine 2D driving key points to generate the lip synchronization frame image of the next frame until all the frames of fine 2D driving key points are traversed.

8. The lip synchronization method based on multiple reference frames and controllable style according to claim 7, characterized in that: The frame scale ratio is obtained by calculating the cross-attention value between the selected fine 2D driving key points and the 2D reference key points of the second reference image; the pixel scale ratio is obtained by calculating the adaptive instance-normalized class cross-attention between the selected fine 2D driving key points and the 2D reference key points of the second reference image.

9. The lip synchronization method based on multiple reference frames and controllable style according to claim 2, characterized in that: The training loss of the audio to keypoint module includes the loss between the true value of the 2D driving keypoint and the predicted fine 2D driving keypoint, the loss between the true value of the 3D driving keypoint and the predicted sparse 3D driving keypoint, and the loss between the lip movement style features calculated using the true value of the 3D driving keypoint and the predicted sparse 3D driving keypoint respectively; The training loss of the key point to video module includes two parts: full image loss and lip area loss, including the mean square error loss between the generated image and the original frame image, the mean square error loss between the image features of the generated image and the original frame image, the mean square error loss between the lip area of ​​the generated image and the lip area of ​​the original frame image, and the mean square error loss between the image features of the lip area of ​​the generated image and the lip area of ​​the original frame image.

10. A lip synchronization system based on multiple reference frames and controllable style, used to implement the method of claim 1; characterized in that: The system comprises: An audio and video data preprocessing module is used to obtain speaker video data and driving audio data, extract driving audio features of each audio frame and driving key points of each video frame; randomly select two groups of video frames from the video as the first multi-reference image and the second multi-reference image, and regard the driving key points of the reference image as reference key points; An audio-to-keypoint module is used to extract the speaker's lip movement style features from the video, and predict fine keypoints by combining the driving audio features and the reference keypoints of the first multi-reference graph; a keypoint-to-video module for combining the second multiple reference images and their reference keypoints, and the refined keypoints predicted by the audio-to-keypoint module, to generate a lip sync frame; A training module, which is used to calculate the loss and update the module based on the prediction results of the audio to key point module and the generation results of the key point to video module; The lip synchronization generation module is used to extract the speaker's lip movement style features from the original video or directly give the speaker's lip movement style features for the given driving audio and original video, and use the trained audio to key point module and key point to video module to generate a synthetic video that is lip-synchronized with the given driving audio to complete the lip synchronization task.

Images