High-definition video customized face fusion method and system

Abstract

This application discloses a customized face fusion method and system for high-definition video, relating to the field of face fusion. The specific implementation scheme is as follows: a small amount of customized face data is input into a customized face online fine-tuning module to obtain a customized face fusion model FaceMerge_xx; the source video is preprocessed and facial key points with different IDs are extracted to identify the target face; the FaceMerge_xx model is loaded into the face fusion module, and the target face is aligned and input into the customized face model to obtain the fused face; the fused face undergoes portrait restoration and super-resolution processing, and video encoding is performed to output the face-fused video. This application's method achieves customized face fusion for high-definition video. The system obtains a customized face model through fine-tuning of customized face data, reconstructs the target face using this model, and improves the naturalness and clarity of the reconstructed face through post-processing. A lightweight network structure is selected for final deployment to improve the performance of the inference process.

Description

Technical Field

[0001] This application relates to the field of face fusion technology, and in particular to a high-definition video customized face fusion method and system. Background Technology

[0002] Face fusion technology refers to the process of combining a source face with a target face, given both, to ensure that the fused face retains the identity features of the source face and the attributes (expression, posture, skin color, etc.) of the target face. It is widely used in live streaming, film and television, and entertainment. Currently, face fusion technology mainly employs the following methods:

[0003] Traditional shape-fitting methods calculate the deformation between two facial shapes based on key facial landmarks, and then use image fusion technology to achieve face swapping. However, the results of this method often suffer from severe distortion, and the face fusion effect is unnatural.

[0004] This 3D face modeling method first estimates the 3D model and pose of the face, then obtains the region to be swapped through face parsing, samples the source face to obtain a texture map, transfers it to the target face, and finally fuses the images to achieve the face swap. This method works well for profile views, but 3D modeling is time-consuming and cannot guarantee natural-looking teeth when the mouth is open. Summary of the Invention

[0005] Based on this, this application provides a high-definition video-customized face fusion method and system to address the problems of poor naturalness of face fusion results, large model size, and high deployment costs.

[0006] The first aspect of this application provides a high-definition video customized face fusion method, comprising:

[0007] Input a small amount of customized facial data and a short video clip;

[0008] The small amount of customized face data is input into the customized face online fine-tuning module to obtain the customized face fusion model FaceMerge_xx;

[0009] The source video is preprocessed to extract facial key points with different IDs and to identify the target face.

[0010] The customized face fusion model FaceMerge_xx is loaded into the face fusion module, and the target face is aligned. The aligned target face is then input into the customized face model to obtain the fused face.

[0011] The merged faces are then subjected to portrait restoration and super-resolution processing, followed by video encoding to output the merged face video.

[0012] The acquisition of the customized face fusion model FaceMerge_xx includes,

[0013] The two base models, unsupervised_baseline and supervised_baseline, are pre-trained on a large dataset.

[0014] Based on the aforementioned small amount of customized face data, unsupervised_baseline is finely adjusted to obtain unsupervised_xx;

[0015] A paired dataset is generated based on unsupervised_xx, and the supervised_baseline is fine-tuned using the paired dataset to obtain a customized face fusion model FaceMerge_xx.

[0016] The operations for fine-tuning a self-supervised network to obtain unsupervised_xx include:

[0017] Customized face X s and the preset target face X t Input the identity encoder and attribute encoder into the self-supervised training network respectively, and extract the customized face X. s Identity characteristics z id and the preset target face X t Attribute characteristics z att ;

[0018] The identity feature z id and the attribute feature z att Inject a face generator to generate a reconstructed face Y. unsup ;

[0019] By using the loss function Loss constraint, the self-supervised network is guided to fully learn the customized facial identity features.

[0020] By constraining the loss function, the self-supervised network is guided to fully learn the calculation of customized facial identity features, including:

[0021] The calculation of the loss function Loss includes,

[0022] Loss = L gan +λ att L att +λ id L id +λ rec L rec

[0023] Among them, L ganThe GAN loss is represented by the adversarial loss generated by the discriminator, which constrains the reconstruction of the face Y. unsup The distribution tends to resemble that of a real human face, L att This represents the attribute preservation loss, used to preserve the preset target face X. t The attribute characteristics of L id This represents the identity preservation loss, used to preserve customized face X. s Identity characteristics, L rec Represents the reconstruction loss, used to constrain the reconstruction of the face Y at the pixel level. unsup X-ray of the target face t Consistent, λ att , λ id and λ rec L respectively att L id and L rec Loss weights;

[0024] Attribute retention loss L att The calculations include,

[0025]

[0026] Loss of identity retention L id The calculations include,

[0027] L id =1-cos(z) id (Y unsup ),z id (X s ))

[0028] Where cos(·,·) represents the cosine similarity between two vectors;

[0029] Reconstruction loss L rec The calculations include,

[0030]

[0031] The steps to fine-tune the supervised network to obtain the customized face fusion model FaceMerge_xx include:

[0032] X the preset target face t The input is a supervised training network that extracts features, and the customized face X extracted by the identity encoder is used to generate the input. s Identity characteristics z id Injecting a supervised training network transformation layer to generate a reconstructed face Y sup ;

[0033] The reconstructed face Y is generated by the self-supervised fine-tuned model unsupervised_xx under the Loss function constraint.unsup As a ground truth (GT), it guides supervised network transfer learning to customize face fusion effects.

[0034] The reconstructed face Y is generated by the self-supervised fine-tuned model unsupervised_xx under the Loss function constraint. unsup As the ground truth (gt), the computation for guiding supervised network transfer learning to customize face fusion results includes,

[0035] The calculation of the loss function Loss includes,

[0036] Loss = L gan +λ perc L perc +λ SSIM L SSIM +λ rec L rec

[0037] Among them, L gan L represents the loss of gan. perc L represents perceptual loss, used for constrained feature reconstruction. SSIM L represents the structural similarity loss, constraining the consistency of local structural changes. rec Represents the reconstruction loss, constrained at the pixel level to reconstruct the face Y. sup The generation of λ perc , λ SSIM and λ rec L respectively perc L SSIM and L rec Loss weights;

[0038] Perceived loss L perc The calculations include,

[0039]

[0040] Where φ represents the pre-trained VGG network, φ i (x) represents the feature of input x in the i-th layer of φ, C i ×H i ×W i Represents φ i The dimension of (x);

[0041] Reconstruction loss L rec The calculations include,

[0042] L rec =||Y sup -Y unsup ||1.

[0043] The steps of preprocessing the source video, extracting facial key points with different IDs from the source video, and labeling the target face include:

[0044] Decode and extract frames from the input video source material;

[0045] Key points are detected and extracted from the faces in each frame, and face tracking technology is used to classify the key points of faces with different IDs.

[0046] After extracting facial key points from all frames, ID rematching is performed to further integrate the key point data;

[0047] The face ID of the person in the video whose face is to be swapped is used as the target face.

[0048] The acquisition of the fused face includes,

[0049] Load the customized online face fine-tuning module and fine-tune it to the customized face fusion model;

[0050] Based on the extracted facial key points, face alignment is performed on the identified target faces in the source video.

[0051] The aligned target face is input into a customized face fusion model, the synthesized face is output, and the synthesized face is remapped back into the original video frame.

[0052] A second aspect of this application provides a high-definition video customized face fusion system, comprising:

[0053] The model building unit is used to input a small amount of customized face data and a video clip; the small amount of customized face data is input into the customized face online fine-tuning module to build a customized face fusion model FaceMerge_xx;

[0054] The face processing unit is used to preprocess the source video, extract facial key points with different IDs in the source video, and mark the target face; load the customized face fusion model FaceMerge_xx into the face fusion module, and simultaneously perform alignment processing on the target face, input the aligned target face into the customized face model, and obtain the fused face;

[0055] The video output unit is used to perform face restoration and super-resolution processing on the fused faces and to encode the video, outputting the fused face video.

[0056] A third aspect of this application provides an apparatus comprising:

[0057] processor;

[0058] Memory used to store processor-executable instructions;

[0059] The processor is configured to invoke instructions stored in the memory to implement the method described in any embodiment of this application.

[0060] A fourth aspect of this application provides a computer-readable storage medium having computer program instructions stored thereon, characterized in that the computer program instructions are executed by a processor to implement the methods described in any embodiment of this application.

[0061] Beneficial Effects: The method described in this application achieves customized face fusion effects in high-definition videos. The entire system obtains a customized face model through fine-tuning of customized face data, reconstructs the target face in the source video using this model, and improves the naturalness and clarity of the reconstructed face through post-processing. Finally, the fused face video is returned. The supervised transfer learning network is stable during training, therefore a lightweight network structure was selected for final deployment, which greatly improves the performance of the inference process.

[0062] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0063] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:

[0064] Figure 1 This is based on the overall process diagram provided in this application;

[0065] Figure 2 It is based on the customized face fusion flowchart provided in this application;

[0066] Figure 3 This is a detailed implementation diagram of pre-training and online fine-tuning provided in this application;

[0067] Figure 4 It is a display image extracted from facial key points provided in this application. Detailed Implementation

[0068] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0069] like Figure 1-4As shown, the first aspect of this application provides a high-definition video customized face fusion method, comprising:

[0070] S1: Input a small amount of customized facial data and a short video clip;

[0071] S2: Input a small amount of customized face data into the customized face online fine-tuning module to obtain the customized face fusion model FaceMerge_xx. The overall framework diagram is shown below. Figure 3 As shown. It should be noted that:

[0072] Obtaining the customized face fusion model FaceMerge_xx includes:

[0073] The two base models, unsupervised_baseline and supervised_baseline, are pre-trained on a large dataset.

[0074] Unsupervised_xx is obtained by fine-tuning unsupervised_baseline based on a small amount of customized face data;

[0075] A paired dataset is generated based on unsupervised_xx, and the supervised_baseline is fine-tuned using the paired dataset to obtain a customized face fusion model FaceMerge_xx.

[0076] like Figure 2 As shown, the operations for fine-tuning the self-supervised network to obtain unsupervised_xx include:

[0077] Customized face X s and the preset target face X t Input the identity encoder and attribute encoder into the self-supervised training network respectively, and extract the customized face X. s Identity characteristics z id and the preset target face X t Attribute characteristics z att ;

[0078] Identity feature z id and attribute features z att Inject a face generator to generate a reconstructed face Y. unsup ;

[0079] By using the loss function Loss constraint, the self-supervised network is guided to fully learn the customized facial identity features.

[0080] By constraining the loss function, the self-supervised network is guided to fully learn the calculation of customized facial identity features, including:

[0081] The calculation of the loss function Loss includes,

[0082] Loss = L gan +λ att L att +λ id L id +λ rec L rec

[0083] Among them, L gan The GAN loss is represented by the adversarial loss generated by the discriminator, which constrains the reconstruction of the face Y. unsup The distribution tends to resemble that of a real human face, L att This represents the attribute preservation loss, used to preserve the preset target face X. t The attribute characteristics of L id This represents the identity preservation loss, used to preserve customized face X. s Identity characteristics, L rec Represents the reconstruction loss, used to constrain the reconstruction of the face Y at the pixel level. unsup X-ray of the target face t Consistent, λ att , λ id and λ rec L respectively att L id and L rec Loss weights;

[0084] Attribute retention loss L att The calculations include,

[0085]

[0086] Loss of identity retention L id The calculations include,

[0087] L id =1-cos(z) id (Y unsup ),z id (X s ))

[0088] Where cos(·,·) represents the cosine similarity between two vectors;

[0089] Reconstruction loss L rec The calculations include,

[0090]

[0091] The steps to fine-tune the supervised network to obtain the customized face fusion model FaceMerge_xx include:

[0092] X the preset target face t The input is a supervised training network that extracts features, and the customized face X extracted by the identity encoder is used to generate the input. s Identity characteristics z id Injecting a supervised training network transformation layer to generate a reconstructed face Y sup ;

[0093] The reconstructed face Y is generated by the self-supervised fine-tuned model unsupervised_xx under the Loss function constraint. unsup As a ground truth (GT), it guides supervised network transfer learning to customize face fusion effects.

[0094] The reconstructed face Y is generated by the self-supervised fine-tuned model unsupervised_xx under the Loss function constraint. unsup As the ground truth (gt), the computation for guiding supervised network transfer learning to customize face fusion results includes,

[0095] The calculation of the loss function Loss includes,

[0096] Loss = L gan +λ perc L perc +λ SSIM L SSIM +λ rec L rec

[0097] Among them, L gan L represents the loss of gan. perc L represents perceptual loss, used for constrained feature reconstruction. SSIM L represents the structural similarity loss, constraining the consistency of local structural changes. rec Represents the reconstruction loss, constrained at the pixel level to reconstruct the face Y. sup The generation of λ perc , λ SSIM and λ rec L respectively perc L SSIM and L rec Loss weights;

[0098] Perceived loss L perc The calculations include,

[0099]

[0100] Where φ represents the pre-trained VGG network, φ i (x) represents the feature of input x in the i-th layer of φ, C i ×H i ×W iRepresents φ i The dimension of (x);

[0101] Reconstruction loss L rec The calculations include,

[0102] L rec =||Y sup -Y unsup ||1.

[0103] S3: Preprocess the source video, extract facial landmarks with different IDs, and label the target faces. It should be noted that:

[0104] The steps of preprocessing the source video, extracting facial landmarks with different IDs, and labeling the target face include:

[0105] Decode and extract frames from the input video source material;

[0106] Keypoint detection and extraction are performed on faces in each frame, and face tracking technology is used to classify the keypoints of faces with different IDs. The keypoint extraction is as follows: Figure 4 As shown;

[0107] After extracting facial key points from all frames, ID rematching is performed to further integrate the key point data;

[0108] The face ID of the person in the video whose face is to be swapped is used as the target face.

[0109] S4: Load the customized face fusion model FaceMerge_xx into the face fusion module, and simultaneously align the target face. Input the aligned target face into the customized face model to obtain the fused face. It should be noted that:

[0110] The acquisition of the merged face includes,

[0111] Load the customized online face fine-tuning module and fine-tune it to the customized face fusion model;

[0112] Based on the extracted facial key points, face alignment is performed on the identified target faces in the source video.

[0113] The aligned target face is input into a customized face fusion model, the synthesized face is output, and the synthesized face is remapped back into the original video frame.

[0114] S5: Performs portrait restoration and super-resolution processing on the fused faces and encodes the video to output the fused face video. The naturalness and clarity of the synthesized face are further improved through portrait restoration and super-resolution processing.

[0115] A second aspect of this application provides a high-definition video customized face fusion system, comprising:

[0116] The model building unit is used to input a small amount of customized face data and a video clip; the small amount of customized face data is input into the customized face online fine-tuning module to build a customized face fusion model FaceMerge_xx;

[0117] The face processing unit is used to preprocess the source video, extract facial key points with different IDs in the source video, and label the target face; load the customized face fusion model FaceMerge_xx into the face fusion module, and perform alignment processing on the target face. Then, input the aligned target face into the customized face model to obtain the fused face.

[0118] The video output unit is used to perform face restoration and super-resolution processing on the fused faces and to encode the video, outputting the fused face video.

[0119] A third aspect of this application provides an apparatus comprising:

[0120] processor;

[0121] Memory used to store processor-executable instructions;

[0122] The processor is configured to invoke instructions stored in the memory to implement the method described in any embodiment of this application.

[0123] A fourth aspect of this application provides a computer-readable storage medium having computer program instructions stored thereon, characterized in that the computer program instructions are executed by a processor to implement the methods described in any embodiment of this application.

[0124] This application's method achieves customized face fusion effects in high-definition videos. The entire system obtains a customized face model through fine-tuning of customized face data, reconstructs the target face in the source video using this model, and improves the naturalness and clarity of the reconstructed face through post-processing. Finally, the fused face video is returned. The supervised transfer learning network is stable during training, therefore a lightweight network structure was selected for final deployment, significantly improving the performance of the inference process.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed.

[0126] The units may or may not be physically separate. The components shown as units can be one or more physical units, meaning they can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, essentially, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A high-definition video customized face fusion method, characterized in that, include: Input a small amount of customized facial data and a short video clip; The small amount of customized face data is input into the customized face online fine-tuning module to obtain the customized face fusion model FaceMerge_xx; The source video is preprocessed to extract facial key points with different IDs and to identify the target face. The customized face fusion model FaceMerge_xx is loaded into the face fusion module, and the target face is aligned. The aligned target face is then input into the customized face model to obtain the fused face. The merged faces are then subjected to portrait restoration and super-resolution processing, followed by video encoding to output the merged face video. The acquisition of the customized face fusion model FaceMerge_xx includes, The two base models, unsupervised_baseline and supervised_baseline, are pre-trained on a large dataset. Based on the aforementioned small amount of customized face data, unsupervised_baseline is finely adjusted to obtain unsupervised_xx; A paired dataset is generated based on unsupervised_xx, and the supervised_baseline is fine-tuned using the paired dataset to obtain a customized face fusion model FaceMerge_xx; The operations for fine-tuning a self-supervised network to obtain unsupervised_xx include: Customized face and the preset target face Input the identity encoder and attribute encoder from the self-supervised training network respectively to extract customized faces. Identity characteristics and the preset target face Attributes and characteristics ; The identity features and the attribute features Inject a face generator to generate a reconstructed face. ; By constraining the loss function, the self-supervised network is guided to fully learn the customized facial identity features; By constraining the loss function, the self-supervised network is guided to fully learn the calculation of customized facial identity features, including: The calculation of the loss function Loss includes, ； in, Indicates gan loss, This represents the attribute retention loss. This indicates a loss of identity retention. Indicates the reconstruction loss. , and They represent , and Loss weights; Attribute Preservation Loss The calculations include, ； Loss of identity retention The calculations include, ； in, Cosine similarity between two vectors; Reconstruction loss The calculations include, 。 2. The high-definition video customized face fusion method according to claim 1, characterized in that: The steps to fine-tune the supervised network to obtain the customized face fusion model FaceMerge_xx include: The preset target face The input is a supervised training network that extracts features, and the customized face extracted by the identity encoder is used as the input. Identity characteristics Injecting a supervised training network transformation layer to generate a reconstructed face ; Reconstructed faces generated by the self-supervised fine-tuned model unsupervised_xx, constrained by the loss function Loss. As a ground truth (GT), it guides supervised network transfer learning to customize face fusion effects.

3. The high-definition video customized face fusion method according to claim 2, characterized in that: Reconstructed faces generated by the self-supervised fine-tuned model unsupervised_xx, constrained by the loss function Loss. As the ground truth (gt), the computation for guiding supervised network transfer learning to customize face fusion results includes, The calculation of the loss function Loss includes, ； in, Indicates gan loss, This represents perceptual loss, used for the reconstruction of constrained features. This represents the structural similarity loss, constraining the consistency of local structural changes. This represents the reconstruction loss, used to reconstruct the face under pixel-level constraints. The generation, , and They represent , and Loss weights; Perceived loss The calculations include, ； in, This represents a pre-trained VGG network. Indicates that input x is in Features of the i-th layer express The dimension; Reconstruction loss The calculations include, 。 4. The high-definition video customized face fusion method according to claim 1, characterized in that: The steps of preprocessing the source video, extracting facial key points with different IDs from the source video, and labeling the target face include: Decode and extract frames from the input video source material; Key points are detected and extracted from the faces in each frame, and face tracking technology is used to classify the key points of faces with different IDs. After extracting facial key points from all frames, ID rematching is performed to further integrate the key point data; The face ID of the person in the video whose face is to be swapped is used as the target face. The acquisition of the fused face includes, Load the customized online face fine-tuning module and fine-tune it to the customized face fusion model; Based on the extracted facial key points, face alignment is performed on the identified target faces in the source video. The aligned target face is input into a customized face fusion model, the synthesized face is output, and the synthesized face is remapped back into the original video frame.

5. A high-definition video customized face fusion system, characterized in that, The high-definition video customized face fusion method according to any one of claims 1-4 includes: The model building unit is used to input a small amount of customized face data and a video clip; the small amount of customized face data is input into the customized face online fine-tuning module to build a customized face fusion model FaceMerge_xx; The face processing unit is used to preprocess the source video, extract facial key points with different IDs in the source video, and mark the target face; load the customized face fusion model FaceMerge_xx into the face fusion module, and simultaneously perform alignment processing on the target face, input the aligned target face into the customized face model, and obtain the fused face; The video output unit is used to perform face restoration and super-resolution processing on the fused faces and to encode the video, outputting the fused face video.

6. A computer device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 4.

Images