An end-to-end face changing data preprocessing method and terminal based on deep learning
By employing a deep learning-based end-to-end face-swapping data preprocessing method, which utilizes facial feature extraction and clustering, combined with image quality assessment and segmentation models, high-quality images with minimal occlusion are automatically selected. This solves the problem of difficult manual screening in existing technologies and improves the efficiency and quality of face-swapping model training data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUJIAN IMPERIAL VISION INFORMATION TECH CO LTD
- Filing Date
- 2022-11-08
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies require extensive manual collaboration and screening in scenarios with diverse image data sources, complex scenes with multiple faces and occlusions, and large-scale face-swapping data to accurately filter irrelevant data and obtain effective data for training face-swapping models.
An end-to-end face-swapping data preprocessing method based on deep learning is adopted. Through face feature extraction, clustering, image quality evaluation and face segmentation models, high-quality and less occluded images are automatically selected as training sets, including the use of face detection, image quality evaluation and face segmentation models.
It automates the processing of face-swapped image datasets, extracting and selecting clear images with minimal occlusion as the training set, reducing manual intervention and improving data screening efficiency and quality.
Smart Images
Figure CN115620376B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to an end-to-end face-swapping data preprocessing method and terminal based on deep learning. Background Technology
[0002] The widely used deep learning face-swapping framework DFL (DeepFaceLab) provides a face-swapping data processing workflow. Through basic steps such as face detection, facial landmark detection and alignment, and facial region segmentation, it achieves the initial processing of face-swapping training data. At the same time, DFL provides basic filtering functions for training data, including sorting face images by Euler angles, sorting (aligned) face images by the proportion of black border areas, sorting images by histogram similarity, and sorting images by blur level. These filtering schemes can effectively filter face images in general scenarios.
[0003] However, in scenarios with diverse image data sources, complex scenes with multiple faces and occlusions, and large-scale face-swapping data, a lot of manual collaboration and screening are often required to accurately filter out irrelevant data and obtain effective data for training face-swapping models. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to provide an end-to-end face-swapping data preprocessing method and terminal based on deep learning, which can perform preliminary processing on the raw data to obtain effective data for training the face-swapping model.
[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0006] An end-to-end face-swapping data preprocessing method based on deep learning, comprising the following steps:
[0007] S1. Obtain the source image dataset and target image dataset for face swapping, and extract the source face set and target face set from them;
[0008] S2. Extract facial features from the source face set and the target face set for face swapping, and then perform face clustering to obtain the source face subset and the target face subset for face swapping.
[0009] S3. For the face-swapping source face subset, use the image quality evaluation module to divide the high-quality image set of the subset;
[0010] S4. For the high-quality image set of the face-swapping source face subset, use the face segmentation model to divide it into a high-quality image set with less occlusion.
[0011] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:
[0012] An end-to-end face-swapping data preprocessing terminal based on deep learning includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:
[0013] S1. Obtain the source image dataset and target image dataset for face swapping, and extract the source face set and target face set from them;
[0014] S2. Extract facial features from the source face set and the target face set for face swapping, and then perform face clustering to obtain the source face subset and the target face subset for face swapping.
[0015] S3. For the face-swapping source face subset, use the image quality evaluation module to divide the high-quality image set of the subset;
[0016] S4. For the high-quality image set of the face-swapping source face subset, use the face segmentation model to divide it into a high-quality image set with less occlusion.
[0017] The beneficial effects of this invention are as follows: an end-to-end face-swapping data preprocessing method and terminal based on deep learning, for a face-swapping image dataset, extracts all faces through face detection, then obtains a subset of faces excluding irrelevant faces through face feature extraction and clustering, and then selects clear and less occluded images one by one as the training set, which can perform preliminary processing on the original data to obtain the final effective data for training the face-swapping model. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating an end-to-end face-swapping data preprocessing method based on deep learning, according to an embodiment of the present invention.
[0019] Figure 2 This is a schematic diagram of the structure of an end-to-end face-swapping data preprocessing terminal based on deep learning, according to an embodiment of the present invention.
[0020] Label Explanation:
[0021] 1. An end-to-end face-swapping data preprocessing terminal based on deep learning; 2. A processor; 3. A memory. Detailed Implementation
[0022] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.
[0023] Please refer to Figure 1 An end-to-end face-swapping data preprocessing method based on deep learning includes the following steps:
[0024] S1. Obtain the source image dataset and target image dataset for face swapping, and extract the source face set and target face set from them;
[0025] S2. Extract facial features from the source face set and the target face set for face swapping, and then perform face clustering to obtain the source face subset and the target face subset for face swapping.
[0026] S3. For the face-swapping source face subset, use the image quality evaluation module to divide the high-quality image set of the subset;
[0027] S4. For the high-quality image set of the face-swapping source face subset, use the face segmentation model to divide it into a high-quality image set with less occlusion.
[0028] As can be seen from the above description, the beneficial effects of the present invention are as follows: an end-to-end face-swapping data preprocessing method and terminal based on deep learning, for a face-swapping image dataset, extracts all faces through face detection, then obtains a subset of faces excluding irrelevant faces through face feature extraction and clustering, and then selects clear and less occluded images one by one as the training set, which can perform preliminary processing on the original data to obtain the final effective data for training the face-swapping model.
[0029] Furthermore, the step of dividing the image into a high-quality image set through the image quality evaluation module specifically involves:
[0030] The first image quality evaluation model was trained on a manually labeled face image quality dataset, and the second image quality evaluation model was trained on a computer-simulated face image dataset with different image qualities.
[0031] The image to be segmented is input into the first image quality evaluation model and the second image quality evaluation model. The image to be segmented is determined to be a high-quality image and stored in the high-quality image set if and only if the first image quality evaluation model determines that it is high quality and the second image quality evaluation model determines that each type of degradation is weak degradation.
[0032] As can be seen from the above description, the evaluation and judgment of image quality has been achieved.
[0033] The computer-simulated dataset of facial images of different qualities is specifically:
[0034] The high-resolution original image is subjected to degradation operations of varying intensities, including Gaussian blur, downsampling, compression, and the addition of random noise. The degradation levels of the images after the operations are then classified.
[0035] As described above, computer-generated facial image data of different qualities has been achieved.
[0036] Furthermore, the process of segmenting a high-quality image set with minimal occlusion using a face segmentation model specifically involves:
[0037] The visible area masking of the eyes and the visible area masking of the mouth are extracted from each image in the high-quality image set by the face segmentation model. The image is stored in the high-quality image set with less occlusion if and only if the area of the visible area of the mouth m1>α and the area of the visible area of the eyes m2>β. Here, α and β are the boundary thresholds of the masking area of the visible area of the mouth and the eye, respectively.
[0038] As described above, the determination of the degree of occlusion in an image has been achieved.
[0039] Furthermore, the step of extracting facial features from the source face set and the target face set for face swapping, followed by face clustering to obtain the source face subset and the target face subset for face swapping, specifically involves:
[0040] The face recognition module extracts facial features from both the source and target face sets for face swapping. The extracted facial features are then clustered, resulting in multiple subsets for both sets. The subset with the most elements in the source face set is designated as the source face subset, and the subset with the most elements in the target face set is designated as the target face subset.
[0041] As described above, an automated process has been implemented to filter irrelevant facial data and select the required face-swapping data.
[0042] An end-to-end face-swapping data preprocessing terminal based on deep learning includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:
[0043] S1. Obtain the source image dataset and target image dataset for face swapping, and extract the source face set and target face set from them;
[0044] S2. Extract facial features from the source face set and the target face set for face swapping, and then perform face clustering to obtain the source face subset and the target face subset for face swapping.
[0045] S3. For the face-swapping source face subset, use the image quality evaluation module to divide the high-quality image set of the subset;
[0046] S4. For the high-quality image set of the face-swapping source face subset, use the face segmentation model to divide it into a high-quality image set with less occlusion.
[0047] As can be seen from the above description, the beneficial effects of the present invention are as follows: an end-to-end face-swapping data preprocessing method and terminal based on deep learning, for a face-swapping image dataset, extracts all faces through face detection, then obtains a subset of faces excluding irrelevant faces through face feature extraction and feature clustering, and then selects clear and less occluded images one by one as the training set, which can perform preliminary processing on the original data to obtain the final effective data for training the face-swapping model.
[0048] Furthermore, the step of dividing the image into a high-quality image set through the image quality evaluation module specifically involves:
[0049] The first image quality evaluation model was trained on a manually labeled face image quality dataset, and the second image quality evaluation model was trained on a computer-simulated face image dataset with different image qualities.
[0050] The image to be segmented is input into the first image quality evaluation model and the second image quality evaluation model. The image to be segmented is determined to be a high-quality image and stored in the high-quality image set if and only if the first image quality evaluation model determines that it is high quality and the second image quality evaluation model determines that each type of degradation is weak degradation.
[0051] As can be seen from the above description, the evaluation and judgment of image quality has been achieved.
[0052] The computer-simulated dataset of facial images of different qualities is specifically:
[0053] The high-resolution original image is subjected to degradation operations of varying intensities, including Gaussian blur, downsampling, compression, and the addition of random noise. The degradation levels of the images after the operations are then classified.
[0054] As described above, computer-generated facial image data of different qualities has been achieved.
[0055] Furthermore, the process of segmenting a high-quality image set with minimal occlusion using a face segmentation model specifically involves:
[0056] The visible area masking of the eyes and the visible area masking of the mouth are extracted from each image in the high-quality image set by the face segmentation model. The image is stored in the high-quality image set with less occlusion if and only if the area of the visible area of the mouth m1>α and the area of the visible area of the eyes m2>β. Here, α and β are the boundary thresholds of the masking area of the visible area of the mouth and the eye, respectively.
[0057] As described above, the determination of the degree of occlusion in an image has been achieved.
[0058] Furthermore, the step of extracting facial features from the source face set and the target face set for face swapping, followed by face clustering to obtain the source face subset and the target face subset for face swapping, specifically involves:
[0059] The face recognition module extracts facial features from both the source and target face sets for face swapping. The extracted facial features are then clustered, resulting in multiple subsets for both sets. The subset with the most elements in the source face set is designated as the source face subset, and the subset with the most elements in the target face set is designated as the target face subset.
[0060] As described above, an automated process has been implemented to filter irrelevant facial data and select the required face-swapping data.
[0061] This invention is used for preliminary processing of DFL training data.
[0062] Please refer to Figure 1 Embodiment 1 of the present invention is as follows:
[0063] An end-to-end face-swapping data preprocessing method based on deep learning includes the following steps:
[0064] S1. Obtain the source image dataset and target image dataset for face swapping, and extract the source face set and target face set from them.
[0065] Let S be the dataset of face-swapped source images. src The dataset of target images for face swapping is denoted as S. dst Here, the face-swapping source represents the face that is replaced in the final output, and the face-swapping target represents the face that is replaced in the final output. Generally, S src A portrait image of the same person scraped from the internet, or a sequence of images extracted from a video clip of a person; S dst This is an image sequence extracted frame by frame from a given video clip. Through the DFL (Digital Functional Linear Algorithm) process of face detection, facial landmark detection and alignment, and face region segmentation, the set F of all detectable faces in the dataset can be obtained. src With F dst , where F src With F dst These are the source face set and the target face set for face swapping, respectively, both of which may include face data not related to face swapping.
[0066] S2. Extract facial features from the source face set and the target face set for face swapping, and then perform face clustering to obtain the source face subset and the target face subset for face swapping.
[0067] Using a face recognition module, the face set F from the face-swapping source was analyzed. src With the target face set F dst Facial feature extraction and face clustering with the same ID were performed separately to obtain...
[0068] in For F src n mutually exclusive subsets of distinct face IDs obtained by clustering face features. For F dst The m mutually exclusive subsets of faces with distinct IDs obtained from face feature clustering are sorted in descending order of the number of elements in each subset. Generally, the source face subset and the target face subset, i.e., the subset of faces with relevant IDs for face swapping, are respectively... and Adjustments can be made based on the specific details of the facial data.
[0069] S3. For the face-swapping source subset, use the image quality evaluation module to divide the subset into a high-quality image set.
[0070] The image quality assessment module will evaluate f src Divided into in, This represents a high-definition image set. This represents a set of low-quality images; the two are mutually exclusive. Here, only f is considered. src The source face set is divided into segments. Provide clear and valid data whenever possible; dst All faces in the image need to be replaced; no image quality filtering is required.
[0071] The image quality assessment model integrates the prediction results of two Mobilenet_v3 classification models. The first image quality assessment model is trained on a manually labeled face image quality dataset, with high quality labeled as 0 and low quality labeled as 1. The second image quality assessment model is trained on a computer-simulated face image dataset of different quality. Specifically, the high-definition original image is subjected to degradation operations such as Gaussian blurring, downsampling, compression, and addition of random noise of different intensities, and is divided into three degradation levels: weak (0), medium (1), and strong (2). The correspondence between the intensity (coefficient) of each degradation class and the degradation level is manually set, and the model is trained to estimate the degradation level.
[0072] Let g1 be the first image quality evaluation model and g2 be the second image quality evaluation model.
[0073]
[0074] Given a face image to be evaluated, an image is classified as a high-quality image if and only if the first image quality evaluation model classifies it as high-quality and the second image quality evaluation model classifies each type of degradation as weak degradation. All high-quality images constitute the above-mentioned... A collection of high-definition images.
[0075] S4. For the high-quality image set of the face-swapping source face subset, use the face segmentation model to divide it into a high-quality image set with less occlusion.
[0076] Using a face segmentation model, Divided into in, This indicates a high-quality image set with minimal occlusion. This refers to a set of high-quality images with significant occlusion; the two are mutually exclusive. Here, we only consider... Divide into sections.
[0077] Face segmentation uses a BiSeNet model trained on the CelebAMask-HQ dataset. BiSeNet extracts the visible areas of the mouth and eyes. When there is occlusion covering a large area of the mouth or eyes, the face image needs to be filtered. Let m1(*) and m2(*) represent the areas of the visible mouth and eyes, and Sign(*) represent the sign function.
[0078]
[0079] α and β are the area boundary thresholds for the visible regions of the mouth and eyes, respectively, and these parameters can be adjusted according to the actual scene. The above formula indicates that a face image is considered unobstructed if and only if m1(c)-α>0 and m2(c)-β>0. All unobstructed images constitute the above... A high-quality image set with minimal or no occlusion, which is the dataset used for the final training.
[0080] Please refer to Figure 2 Embodiment two of the present invention is as follows:
[0081] An end-to-end face-swapping data preprocessing terminal 1 based on deep learning includes a memory 3, a processor 2, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, it implements the steps of the above embodiment 1.
[0082] In summary, the present invention provides an end-to-end face-swapping data preprocessing method and terminal based on deep learning. For face-swapping image datasets, all faces are extracted through face detection, and then a subset of faces excluding irrelevant faces is obtained through face feature extraction and feature clustering. Then, clear and minimally occluded images are selected one by one as training sets. This allows for preliminary processing of the original data to obtain effective data for training the face-swapping model.
[0083] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A deep learning-based end-to-end face-swapping data preprocessing method, characterized in that, Including the following steps: S1. Obtain the source image dataset and target image dataset for face swapping, and extract the source face set and target face set from them; S2. Extract facial features from the source face set and the target face set for face swapping, and then perform face clustering to obtain the source face subset and the target face subset for face swapping. S3. For the face-swapped source subset, a high-quality image set of the subset is divided using the image quality evaluation module. Specifically, the high-quality image set is divided using the image quality evaluation module: a first image quality evaluation model is trained on a manually labeled face image quality dataset, and a second image quality evaluation model is trained on a computer-simulated face image dataset of different image qualities. The image to be divided is input into the first image quality evaluation model and the second image quality evaluation model. If and only if the first image quality evaluation model determines that the image is of high quality and the second image quality evaluation model determines that each type of degradation is a weak degradation, the image to be divided is determined to be a high-quality image and stored in the high-quality image set. S4. For the high-quality image set of the face-swapping source face subset, use the face segmentation model to divide it into a high-quality image set with less occlusion.
2. The end-to-end face-swapping data preprocessing method based on deep learning according to claim 1, characterized in that, The computer-simulated dataset of facial images of different qualities is specifically: The high-resolution original image is subjected to degradation operations of varying intensities, including Gaussian blur, downsampling, compression, and the addition of random noise. The degradation levels of the images after the operations are then classified.
3. The end-to-end face-swapping data preprocessing method based on deep learning according to claim 1, characterized in that, The process of segmenting a face using a face segmentation model to create a high-quality image set with minimal occlusion specifically involves: The visible eye region mask and the visible mouth region mask of each image in the high-quality image set were extracted using a face segmentation model. If and only if the area of the visible mouth region in the image is [area missing]. And the area visible to the eye The image is stored in a high-quality image set with minimal occlusion, where... These are the boundary thresholds for the occlusion area of the visible regions of the mouth and eyes, respectively.
4. The end-to-end face-swapping data preprocessing method based on deep learning according to claim 1, characterized in that, The process of extracting facial features from the source and target face sets for face swapping, followed by face clustering to obtain subsets of source and target faces, specifically involves: The face recognition module extracts facial features from both the source and target face sets for face swapping. The extracted facial features are then clustered, resulting in multiple subsets for both sets. The subset with the most elements in the source face set is designated as the source face subset, and the subset with the most elements in the target face set is designated as the target face subset.
5. A deep learning-based end-to-end face-swapping data preprocessing terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it performs the following steps: S1. Obtain the source image dataset and target image dataset for face swapping, and extract the source face set and target face set from them; S2. Extract facial features from the source face set and the target face set for face swapping, and then perform face clustering to obtain the source face subset and the target face subset for face swapping. S3. For the face-swapped source subset, a high-quality image set of the subset is divided using the image quality evaluation module. Specifically, the high-quality image set is divided using the image quality evaluation module: a first image quality evaluation model is trained on a manually labeled face image quality dataset, and a second image quality evaluation model is trained on a computer-simulated face image dataset of different image qualities. The image to be divided is input into the first image quality evaluation model and the second image quality evaluation model. If and only if the first image quality evaluation model determines that the image is of high quality and the second image quality evaluation model determines that each type of degradation is a weak degradation, the image to be divided is determined to be a high-quality image and stored in the high-quality image set. S4. For the high-quality image set of the face-swapping source face subset, use the face segmentation model to divide it into a high-quality image set with less occlusion.
6. The end-to-end face-swapping data preprocessing terminal based on deep learning according to claim 5, characterized in that, The computer-simulated dataset of facial images of different qualities is specifically: The high-resolution original image is subjected to degradation operations of varying intensities, including Gaussian blur, downsampling, compression, and the addition of random noise. The degradation levels of the images after the operations are then classified.
7. The deep learning-based end-to-end face-swapping data preprocessing terminal according to claim 5, characterized in that, The process of segmenting a face using a face segmentation model to create a high-quality image set with minimal occlusion specifically involves: The visible eye region mask and the visible mouth region mask are extracted from each image in the high-quality image set using a face segmentation model. The mask is determined if and only if the area of the visible mouth region of the image is... And the area visible to the eye The image is stored in a high-quality image set with minimal occlusion, where... These are the boundary thresholds for the occlusion area of the visible regions of the mouth and eyes, respectively.
8. The end-to-end face-swapping data preprocessing terminal based on deep learning according to claim 5, characterized in that, The process of extracting facial features from the source and target face sets for face swapping, followed by face clustering to obtain subsets of source and target faces, specifically involves: The face recognition module extracts facial features from both the source and target face sets for face swapping. The extracted facial features are then clustered, resulting in multiple subsets for both sets. The subset with the most elements in the source face set is designated as the source face subset, and the subset with the most elements in the target face set is designated as the target face subset.