A face deocclusion method based on feature reconstruction
Through the face deocclusion framework based on feature reconstruction, multiple occlusions are automatically detected and removed, which solves the problem of difficulty in handling multiple occlusions in existing technologies and improves the accuracy and efficiency of face recognition.
Patent Information
- Application Number
- CN202310758421.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-26
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2043-06-26
AI Technical Summary
Existing face deocclusion methods are unable to effectively handle situations with multiple occluders, resulting in a decrease in the accuracy of face-related tasks. In addition, the need to manually mark occluded areas is time-consuming and restrictive.
A feature-based reconstruction-based face deocclusion framework is adopted to automatically detect and remove single and multiple occluders through a prediction-extraction-inpainting method, including building an occlusion-robust face segmentation module, an occluder extractor and a face generator, and training using synthetic datasets with various occlusion situations.
It effectively enhances the model's ability to extract occluders and restore facial attribute features in multi-occlusion scenarios, improves the accuracy of face-related tasks, and reduces the need for manual labeling.
Smart Images
Figure FDA0005490828440000012 
Figure FDA0005490828440000013 
Figure FDA0005490828440000014
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a face de-occlusion method based on feature reconstruction. Background Art
[0002] In the study of face-related tasks, facial occlusion (such as hands, masks, glasses, etc.) has always been a common interference factor. When there are occluders in a face image, some facial features will disappear, resulting in a significant decrease in the accuracy of face-related tasks. At the same time, due to the various shapes and types of facial occlusions, their non-fixed positions, and other unpredictable occlusion characteristics, they are more difficult to handle than other interference factors such as lighting and posture. In order to automatically remove facial occlusions and generate unobstructed face images, people have proposed face deocclusion technology, which can be used as a preprocessing step to assist other face-related tasks. For example, it can also be integrated into access control systems to improve the efficiency of face recognition. Therefore, research on face recognition technology has important practical significance.
[0003] The main idea of traditional technologies is to repair images based on existing information. Since each part of a face image has its own characteristics, their results are far from satisfactory.
[0004] Recently, deep learning-based face deocclusion techniques have been proposed and have achieved promising results. For example, Pathak D et al. first proposed a deep learning-based image restoration method. Iizuka S et al. proposed using a global discriminator and a local discriminator for training. Yu J et al. proposed a gated convolution-based technique for repairing missing regions of arbitrary shapes. Dong J et al. introduced an occlusion-aware stage, freeing the model from the constraints of manually labeled occlusions and enabling the model to freely handle various facial occlusions. These methods make it possible to use deep learning technology to address the problem of facial occlusion. However, the images generated by these methods have low resolution and are difficult to meet real-world needs.
[0005] In order to meet the application needs of real-world scenarios, people have proposed deocclusion methods for high-resolution face images. Nazeri K et al. proposed a technique that uses edge maps to guide image restoration. Guo X et al. proposed a two-stream network that models structure-constrained texture synthesis and texture-guided structure reconstruction in a coupled manner. Wang N et al. added a dynamic selection mechanism to the U-Net structure. Zhu M et al. proposed a mechanism of "using different convolution kernels for different convolution windows" to guide image restoration. These methods have shown good results in dealing with face deocclusion. However, most of them are general image restoration methods and are not specifically designed for face deocclusion tasks. In addition, they usually require manual marking of occluded areas, which can be time-consuming and have certain limitations in practical applications.
[0006] In addition, there are some face deocclusion techniques designed for specific occluders. For example, Hu B et al., Lee YH et al., and Lyu J et al. proposed frameworks for removing glasses. Although these methods have shown success in removing glasses, they are difficult to directly generalize to other types of occluders commonly found in real-world scenes. Summary of the Invention
[0007] Purpose of the invention:
[0008] Face deocclusion is crucial for improving the accuracy of face-related tasks. However, most existing methods target single occlusions, which makes them suboptimal when multiple occlusions are present. To address this issue, we propose a novel feature-based reconstruction-based face deocclusion framework. It automatically detects and removes both single and multiple occlusions through a prediction-extraction-inpainting approach, making it a universal solution for handling diverse occlusions.
[0009] To achieve the above object, the technical solution adopted by the present invention is:
[0010] A face deocclusion method based on feature reconstruction includes the following steps:
[0011] Step 1: obtain an original face image dataset and an occlusion dataset, filter out unobstructed face images from the original face image dataset, add one or more occlusions to the unobstructed face images using the occlusion dataset to obtain an occluded face image dataset, and obtain a triplet dataset consisting of the original face image dataset, the occlusion dataset, and the occluded face image dataset;
[0012] Step 2: Build an occlusion-robust face segmentation module. The input of the occlusion-robust face segmentation module is the occluded face image dataset obtained in step 1. The output is an occlusion segmentation map and a face segmentation map. The occlusion segmentation map contains the location information of the occluders.
[0013] Step 3: Build an occluder extractor. The input of the occluder extractor is the occluded face image dataset obtained in step 1 and the occluder position information output in step 2. The output is a binary occluder mask image.
[0014] Step 4: Build a face generator. The input of the face generator is the occluded face image dataset obtained in step 1, the face segmentation map output in step 2, and the binary occlusion mask map output in step 3. The output is the unoccluded face image.
[0015] Step 5: Construct a network model including an occlusion-robust face segmentation module, an occluder extractor, and a face generator, train the network model, and use the trained network model for face deocclusion.
[0016] Preferably, in step 1, the CelebA-HQ dataset is selected as the original face image dataset, and a number of glasses images, mask images, and hand images are collected as occlusion datasets; obtaining the occluded face image dataset includes the following steps:
[0017] First, we screen out unobstructed face images from the CelebA-HQ dataset. We use the dlib and OpenCV face detection modules to obtain the facial pose angles of the unobstructed face images. Based on the facial pose angles, we classify the unobstructed face images into three categories: frontal upward view, frontal downward view, left angle of 10° to 40°, left angle of 40° to 60°, left angle of more than 60°, right angle of 10° to 40°, right angle of 40° to 60°, and right angle of more than 60°. At the same time, we classify the images in the occluded dataset according to the same angles. Next, we use the face_alignment library to extract facial features and obtain 68 facial key points. Finally, we add the images in the occluded dataset to the unobstructed face images to obtain the occluded face image dataset.
[0018] Preferably, in order to further fit the real-life situation, random transparency is set for some of the glasses images; in order to further simulate the diverse types of obstructions in real life and the possibility of multiple obstructions occurring at the same time, when adding images in the obstruction dataset to the unobstructed face images, hand images at random positions are added to the face area of some of the unobstructed face images, and glasses images and mask images are added to the other unobstructed face images at the same time.
[0019] Preferably, the occlusion-robust face segmentation module uses the PP-LiteSeg semantic segmentation model. The input of the PP-LiteSeg semantic segmentation model is an occluded face image dataset, and the output is an occlusion segmentation map and a face segmentation map, wherein the occlusion segmentation map contains the occlusion position information.
[0020] Preferably, the occlusion mask extractor includes a coarse occlusion mask extractor and a refined occlusion mask extractor:
[0021] The rough occluder mask extractor includes five encoders and five decoders, each of which includes a 3*3Conv-BN-ReLU module, each of which is followed by three residual blocks in series, and the last residual block is followed by a maximum pooling layer; each decoder uses bilinear interpolation to upsample the output of the maximum pooling layer, then passes it through a 3*3Conv-BN-ReLU module and is connected to the output of the corresponding encoder through a jump connection to obtain a spliced feature map, and then passes the spliced feature map through a 3*3Conv-BN-ReLU module and three concatenated residual blocks to obtain a new feature map, and the output result of the last decoder is passed through a 1*1Conv and sigmoid activation function to obtain a rough occluder mask map;
[0022] The coarse occluder mask extractor is trained using binary cross entropy loss:
[0023] Binary Cross Entropy Loss is defined as:
[0024]
[0025] Among them, M gt Represents the true binary occluder mask, M c Represents the output rough occluder mask map, i, j represent coordinate parameters, Represents M gt The (i,j)th item of Represents M c The (i,j)th item of ;
[0026] The refined occluder mask extractor takes the coarse occluder mask as input, filters out the portion with pixel values greater than 0.5 as the main area of the occluder, then performs a dot product between the main area of the occluder and the feature map output by the last decoder of the coarse occluder mask extractor to calculate the main features of the occluder in the latent space. Simultaneously, the feature map is projected into the latent space to obtain the features of the coarse occluder mask in the latent space. Finally, similarity matching is performed between the main area of the occluder and the features of other positions of the coarse occluder mask, retaining the positively correlated areas to obtain the refined occluder mask.
[0027] The refined occluder mask extractor is trained using binary cross entropy loss and intersection over union loss (IoU):
[0028] Binary Cross Entropy Loss is defined as:
[0029]
[0030] Among them, M r represents the output refined occluder mask map, Represents M r The (i,j)th item of ;
[0031] The intersection-over-union loss IoU is defined as:
[0032]
[0033] Then we have: The objective function of the occluder mask extractor for:
[0034]
[0035] Among them, λ1, λ2 and λ3 are and The weight parameter of .
[0036] Preferably, the face generator includes a coarse face generator and a refined face generator. The coarse face generator includes eight encoders and eight decoders. Each encoder downsamples the refined occlusion mask map through a convolution layer. The decoder upsamples the output of the encoder through a transposed convolution layer and restores the feature map to the size of the input refined occlusion mask map, thereby obtaining a coarse face image. According to the refined binary occlusion mask map, a coarse synthetic face image is obtained.
[0037]
[0038] Among them, I f-occ is the input occluded face image, is the output rough face image, is the rough synthetic face image obtained;
[0039] The rough face generator is trained using pixel reconstruction loss, adversarial loss, and gender consistency loss:
[0040] Pixel reconstruction loss is defined as:
[0041]
[0042] in, Represents the pixel reconstruction loss of the valid area, represent The number of non-zero elements in I gt represents the original face image, Represents the pixel reconstruction loss of the occluded area, sum(M r ) represents M r The number of non-zero elements in , λ o is the weight parameter;
[0043] The generative adversarial loss is defined as:
[0044]
[0045]
[0046] represents the generative adversarial loss of the generator, represents the expectation of the distribution function, represents the distribution of generated data, I mer represents the samples sampled from the generating data distribution, Represents the discriminator to generate data The judgment result of represents the generative adversarial loss of the discriminator, P data represents the distribution of real data, I represents the sample obtained from the real data distribution, D(I gt ) represents the discriminator's response to the real data I gt The judgment result of
[0047] Gender consistency loss Defined as:
[0048]
[0049] in, Represents the real image I gt The probability of being male, Represents the generated image Probability of being male;
[0050] Then, the goal of the rough face generator is for:
[0051]
[0052] Among them, λ G and λ gen They are and The weight parameter of
[0053] The refined face generator includes a local area refinement module and an important area refinement module:
[0054] The local area refinement module extracts information from the neighboring area through the residual network; the important area refinement module uses the pixel similarity within the feature to identify the pixels related to the missing pixels from the valid pixels of the corresponding feature, fills the occluded area, and adopts the adaptive merging method to fuse the outputs of the local area refinement module and the important area refinement module to generate a refined binary occluder mask image. According to the refined binary occluder mask image, the input image and the generated refined face image are merged to obtain a refined synthetic face image.
[0055]
[0056] Among them, I f-occ is the input occluded face image, is the output refined face image, is the obtained refined synthetic face image;
[0057] The refined face generator is trained using pixel reconstruction loss, perceptual loss, style loss, smoothness loss, and identity loss:
[0058] Pixel reconstruction loss is defined as:
[0059]
[0060] in, Represents the pixel reconstruction loss of the valid area, represent The number of non-zero elements in I gt represents the original face image, Represents the pixel reconstruction loss of the occluded area, sum(M r ) represents M r The number of non-zero elements in , λ o is the weight parameter;
[0061] Perceptual loss is defined as:
[0062]
[0063] Among them, φ i represents the feature map output by the i-th layer in the pre-trained VGG-16 network, i∈{5,10,17};
[0064] Style Loss is defined as:
[0065]
[0066] in, is the Gram matrix;
[0067] Smoothing loss is defined as:
[0068]
[0069] identity loss is defined as:
[0070]
[0071] in, and I gt and The output vector of the input face recognition network, ∈, is a minimum value of 1e-8 to avoid division by zero;
[0072] Ultimately, the goal of the refined face generator is:
[0073]
[0074] Among them, λ per ,λ sty ,λ tv and λ id They are and The weight parameter of .
[0075] Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects:
[0076] Previous face deocclusion methods have been unable to effectively cope with complex occlusion scenarios in reality. This invention effectively enhances the model's ability to extract occlusions through a two-stage occlusion extractor, and effectively enhances the model's ability to restore facial attribute features through a two-stage face generator. At the same time, by creating a synthetic dataset containing multiple occlusion situations, the model can effectively learn the ability to remove multiple occlusions during training. DETAILED DESCRIPTION
[0077] The present invention will be further explained below.
[0078] A face deocclusion method based on feature reconstruction of the present invention comprises the following steps:
[0079] Step 1: Create a dataset. We selected CelebA-HQ as the face image dataset, and collected 362 glasses images, 324 masks images, and 1,000 hand images as the occlusion dataset.
[0080] First, facial images were classified using attribute annotations from CelebA-HQ, filtering out those without occlusions. Second, facial pose angles were determined using dlib and OpenCV. Based on the pose angles, facial images were categorized into frontal upward view, frontal downward view, leftward angle of 10° to 40°, leftward angle of 40° to 60°, leftward angle of 60° or greater, rightward angle of 10° to 40°, rightward angle of 40° to 60°, and rightward angle of 60° or greater. Images with glasses and masks were also categorized based on these angles. Next, facial features were extracted using face_alignment, resulting in 68 facial landmarks. Finally, based on the head pose and facial landmarks, glasses and masks were added to the unobstructed facial images. To further align with real-life conditions and make the synthesized images appear more realistic, a random transparency was set for the glasses occlusion. In order to further simulate the diverse types of occlusions in real life and the possibility of multiple occlusions occurring at the same time, hand-shaped occlusions are added to the face area of some face images at random positions, and glasses and masks are added to other face images at the same time.
[0081] Finally, based on the original division of the CelebA dataset, we obtained 88,932 sets of data as training datasets, 11,120 sets of data as validation datasets, and 10,244 sets of data as test datasets. Each set of data includes original face images, occluded face images, and binary occlusion mask maps.
[0082] Step 2: Build an occlusion-robust face segmentation module.
[0083] Its overall structure is as follows:
[0084] Input → Occlusion-robust face segmentation module → seg_mask+seg_face
[0085] Among them, input is the occluded face image, seg_mask is the predicted occlusion segmentation map, and seg_face is the predicted face segmentation map.
[0086] In this paper, PP-LiteSeg is used to implement this module and trained with a face segmentation dataset. During the face deocclusion process, this module provides prior information about occlusion locations and facial structure, thereby guiding occlusion extraction and face restoration.
[0087] Step 3: Build an occluder extractor.
[0088] The occluder mask extractor consists of a coarse stage and a fine stage. The overall structure of the coarse stage is as follows:
[0089] input+seg_mask→Occluder mask extractor coarse stage→mask_coarse
[0090] Among them, input is the occluded face image, seg_mask is the occlusion position information obtained in step 2, and mask_coarse is the output rough occlusion mask.
[0091] In the coarse phase, an occlusion extractor is constructed based on the U-Net network structure. It mainly consists of an encoder and a decoder, and uses skip connections to connect corresponding features. Specifically, the encoder first doubles the number of channels in the feature map using a 3*3 Conv-BN-ReLU module, then uses three residual blocks to increase the model's receptive field to further extract features. Finally, it downsamples the feature map through a max pooling layer to reduce the dimensionality of the feature map. The decoder performs the reverse operation, first upsampling the feature map using bilinear interpolation to expand the dimensionality of the feature map, then reducing the number of channels by half using a 3*3 Conv-BN-ReLU module. The result is connected to the output feature map of the corresponding encoder through a skip connection to obtain a concatenated feature map. The concatenated feature map is then passed through a 3*3 Conv-BN-ReLU module to fuse the two feature maps and reduce the number of channels by half. Finally, the fused feature map is passed through three residual blocks to obtain a new feature map. The output of the last decoder is passed through 1*1Conv and sigmoid activation function to obtain a rough occluder mask.
[0092] The coarse occluder mask extractor is trained using binary cross entropy loss:
[0093] Binary Cross Entropy Loss is defined as:
[0094]
[0095] Among them, M gt Represents the true binary occluder mask, M c Represents the output rough occluder mask, i, j represent coordinate parameters, Represents M gt The (i,j)th item of Represents M c The (i,j)th item of .
[0096] The overall structure of the refinement phase is as follows:
[0097] mask_coarse → Occluder Mask Extractor Refinement Stage → mask_refine
[0098] Among them, mask_coarse is the coarse occluder mask, and mask_refine is the output refined occluder mask.
[0099] During the refinement phase, the coarse occlusion mask is input and the portion with pixel values greater than 0.5 is filtered as the primary region of the occlusion. A dot product is then performed between the primary region and the feature map output by the last decoder of the coarse occlusion mask extractor to calculate the primary features of the occlusion in latent space. Simultaneously, the feature map is projected into latent space to obtain the latent-space features of the coarse occlusion mask. Finally, similarity matching is performed between the primary features and features at other positions in the coarse occlusion mask, retaining regions with positive correlations to obtain the refined occlusion mask. This operation allows for calibration based on the characteristics of the occlusion, improving the performance of the occlusion extractor.
[0100] The refined occluder mask extractor is trained using binary cross entropy loss and intersection over union (IoU) loss:
[0101] Binary Cross Entropy Loss is defined as:
[0102]
[0103] Among them, M gt Represents the true binary occluder mask, M r represents the output refined occluder mask, Represents M gt The (i,j)th item of Represents M r The (i,j)th item of .
[0104] Intersection over Union (IoU) loss is defined as:
[0105]
[0106] Among them, M gt Represents the true binary occluder mask, M r represents the output refined occluder mask, Represents M gt The (i,j)th item of Represents M r The (i,j)th item of .
[0107] The final occlusion generator's goals are:
[0108]
[0109] Among them, λ1, λ2 and λ3 are and The weight parameter of .
[0110] Step 4: Build a face generator.
[0111] The face generator consists of a coarse stage and a fine stage. The overall structure of the coarse stage is as follows:
[0112] input+seg_face+mask_refine→face generator rough stage→face_coarse
[0113] Among them, input is the occluded face image, seg_face is the face segmentation map obtained in step 2, mask_refine is the refined occlusion mask obtained in step 3, and face_coarse is the unoccluded face image generated in the coarse stage of the face generator.
[0114] In the coarse phase, a U-Net architecture with a large receptive field is used to reconstruct the global structure of the face, guided by the face segmentation map. Specifically, the coarse face generator consists of eight encoder-decoder blocks. Each encoder downsamples the feature map through a convolutional layer, doubling its number of channels while reducing its size. The decoder upsamples the feature map to the same size as the input image through a transposed convolutional layer, resulting in a coarse face image. Based on the refined binary occlusion mask image, the input image and the generated coarse face image are merged to produce a coarse composite face image.
[0115]
[0116] Among them, I f-occ is the input occluded face image, is the output rough face image, This is the rough synthetic face image obtained.
[0117] In this stage, the rough face generator is trained using pixel reconstruction loss, adversarial loss, and gender consistency loss.
[0118] Pixel reconstruction loss is defined as:
[0119]
[0120] in, Represents the pixel reconstruction loss of the valid area, represent The number of non-zero elements in I gt represents the original face image, Represents the pixel reconstruction loss of the occluded area, sum(m r ) represents Mr The number of non-zero elements in , λ o is the weight parameter.
[0121] The adversarial loss is defined as:
[0122]
[0123] represents the generative adversarial loss of the generator, represents the expectation of the distribution function, represents the distribution of generated data, I mer represents the samples sampled from the generating data distribution, Represents the discriminator to generate data The judgment result of represents the generative adversarial loss of the discriminator, P data represents the distribution of real data, I represents the sample obtained from the real data distribution, D(I gt ) represents the discriminator's response to the real data I gt The judgment result of .
[0124] Using the vgg-16 network, the gender classification network is obtained through transfer learning, thereby reducing the gender consistency loss Defined as:
[0125]
[0126] in, Representative image I gt The probability of being male, Representative image The probability of being male.
[0127] Ultimately, the goals of the rough face generator are:
[0128]
[0129] Among them, λ G and λ gen They are and The weight parameter of .
[0130] The overall structure of the refinement phase is as follows:
[0131] face_coarse+mask_refine→Face generator refinement stage→face_refine
[0132] Among them, face_coarse is the rough unoccluded face image generated by the coarse stage of the face generator, mask_refine is the refined occluder mask obtained in step 3, and face_coarse is the refined unoccluded face image generated by the refinement stage of the face generator.
[0133] In the refinement stage, it is divided into two different modules: the local region refinement module and the important region refinement module. The local region refinement module extracts information from neighboring regions through a residual network with a small receptive field, thereby enhancing local texture. The important region refinement module, on the other hand, utilizes intra-feature pixel similarity to identify pixels related to missing pixels from the valid pixels of the corresponding feature, and then uses them to fill the occluded area, thereby ensuring feature harmony. An adaptive merging method is used to fuse the outputs of the two branches to produce the final refined de-occluded face image. According to the refined binary occluder mask image, the input image and the generated rough face image are merged to obtain a refined synthetic face image.
[0134]
[0135] Among them, I f-occ is the input occluded face image, is the output refined face image, is the obtained refined synthetic face image
[0136] In this stage, the refined face generator is trained using pixel reconstruction loss, style loss, perceptual loss, smoothness loss, and identity loss.
[0137] Pixel reconstruction loss is defined as:
[0138]
[0139] in, Represents the pixel reconstruction loss of the valid area, represent The number of non-zero elements in I gt represents the original face image, Represents the pixel reconstruction loss of the occluded area, sum(M r ) represents M r The number of non-zero elements in , λ o is the weight parameter.
[0140] At the same time, the perceptual loss and style loss are calculated using the VGG-16 model pre-trained on ImageNet.
[0141] Perceptual loss is defined as:
[0142]
[0143] Among them, φ i Represents the feature map output by the i-th layer in the pre-trained VGG-16 network, i∈{5,10,17}.
[0144] Style Loss is defined as:
[0145]
[0146] in, is the Gram matrix.
[0147] The smoothing loss is defined as:
[0148]
[0149] The identity loss is defined using ResNet-50 trained with ArcFace on MS1M as the face recognition network. The identity loss is defined as:
[0150]
[0151] in, and I gt and The output vector of the input face recognition network is minimized to 1e-8 to avoid division by zero.
[0152] Ultimately, the goal of the refined face generator is:
[0153]
[0154] Among them, λ per ,λ sty ,λ tv and λ id They are and The weight parameter of .
[0155] In the present invention, a new face deocclusion framework based on feature reconstruction is proposed, which consists of three stages: structure prediction, occlusion extraction and face restoration. In the process of occlusion extraction, a rough occlusion mask is first extracted using the prior information of the occlusion position, and then the mask is refined according to the occlusion features. In the face restoration process, the global structure is first reconstructed under the guidance of the face segmentation map, and then the important areas are refined using local information and the internal information of the corresponding features. In addition, a high-quality synthetic face dataset for the face deocclusion task is established, which contains various types and numbers of occlusions, better fits the occlusion scenes in reality, and provides effective supervision for the training of the model. A large number of experimental results show that the model can effectively remove face occlusions and retain attribute information to better support face-related tasks.
[0156] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.
Claims
1. A face deocclusion method based on feature reconstruction, characterized in that: The steps include: Step 1: obtain an original face image dataset and an occlusion dataset, filter out unobstructed face images from the original face image dataset, add one or more occlusions to the unobstructed face images using the occlusion dataset to obtain an occluded face image dataset, and obtain a triplet dataset consisting of the original face image dataset, the occlusion dataset, and the occluded face image dataset; Step 2: Build an occlusion-robust face segmentation module. The input of the occlusion-robust face segmentation module is the occluded face image dataset obtained in step 1. The output is an occlusion segmentation map and a face segmentation map. The occlusion segmentation map contains the location information of the occluders. Step 3: Build an occluder extractor. The input of the occluder extractor is the occluded face image dataset obtained in step 1 and the occluder position information output in step 2. The output is a binary occluder mask map. The occluder mask extractor includes a rough occluder mask extractor and a refined occluder mask extractor. The rough occluder mask extractor includes five encoders and five decoders. Each encoder includes a 3*3 Conv-BN-ReLU module. Each Conv-BN-ReLU module is connected in series with three residual blocks. The last residual block is followed by a maximum pooling layer; each decoder uses bilinear interpolation to upsample the output of the maximum pooling layer, then passes it through a 3*3 Conv-BN-ReLU module and is connected to the output of the corresponding encoder through a jump connection to obtain a spliced feature map, and then the spliced feature map is passed through a 3*3 Conv-BN-ReLU module and three residual blocks in series to obtain a new feature map. The output result of the last decoder is passed through a 1*1 Conv and a sigmoid activation function to obtain a rough occluder mask map; The coarse occluder mask extractor is trained using binary cross entropy loss: Binary Cross Entropy Loss is defined as: Among them, M gt Represents the true binary occluder mask, M c Represents the output rough occluder mask map, i, j represent coordinate parameters, Represents M gt The (i,j)th item of Represents M c The (i,j)th item of ; The refined occluder mask extractor takes the coarse occluder mask as input, filters out the portion with pixel values greater than 0.5 as the main area of the occluder, then performs a dot product between the main area of the occluder and the feature map output by the last decoder of the coarse occluder mask extractor to calculate the main features of the occluder in the latent space. Simultaneously, the feature map is projected into the latent space to obtain the features of the coarse occluder mask in the latent space. Finally, similarity matching is performed between the main area of the occluder and the features of other positions of the coarse occluder mask, retaining the positively correlated areas to obtain the refined occluder mask. Step 4: Build a face generator. The input of the face generator is the occluded face image dataset obtained in step 1, the face segmentation map output in step 2, and the binary occlusion mask map output in step 3. The output is the unoccluded face image. Step 5: Construct a network model including an occlusion-robust face segmentation module, an occluder extractor, and a face generator, train the network model, and use the trained network model for face deocclusion.
2. The face deocclusion method based on feature reconstruction according to claim 1, characterized in that: In step 1, the CelebA-HQ dataset is selected as the original face image dataset, and several glasses images, mask images, and hand images are collected as occlusion datasets. The acquisition of the occluded face image dataset includes the following steps: First, we screen out unobstructed face images from the CelebA-HQ dataset. We use the dlib and OpenCV face detection modules to obtain the facial pose angles of the unobstructed face images. Based on the facial pose angles, we classify the unobstructed face images into three categories: frontal upward view, frontal downward view, left angle of 10° to 40°, left angle of 40° to 60°, left angle of more than 60°, right angle of 10° to 40°, right angle of 40° to 60°, and right angle of more than 60°. At the same time, we classify the images in the occluded dataset according to the same angles. Next, we use the face_alignment library to extract facial features and obtain 68 facial key points. Finally, we add the images in the occluded dataset to the unobstructed face images to obtain the occluded face image dataset.
3. The face deocclusion method based on feature reconstruction according to claim 2, characterized in that: In step 1, in order to further fit the real-life situation, random transparency is set for several of the glasses images; in order to further simulate the diverse types of occlusions in real life and the possibility of multiple occlusions occurring at the same time, when adding images in the occlusion dataset to the unobstructed face images, hand images at random positions are added to the face area of some of the unobstructed face images, and glasses images and mask images are added to the other unobstructed face images at the same time.
4. The face deocclusion method based on feature reconstruction according to claim 3, characterized in that: In step 2, the occlusion-robust face segmentation module uses the PP-LiteSeg semantic segmentation model. The input of the PP-LiteSeg semantic segmentation model is an occluded face image dataset, and the output is an occlusion segmentation map and a face segmentation map. The occlusion segmentation map contains the occlusion position information.
5. The face deocclusion method based on feature reconstruction according to claim 4, characterized in that: The step 3 further includes: using binary cross entropy loss and intersection over union loss (IoU) to train the refined occluder mask extractor: Binary Cross Entropy Loss is defined as: Among them, M r represents the output refined occluder mask map, Represents M r The (i,j)th item of ; The intersection-over-union loss IoU is defined as: Then we have: The objective function of the occluder mask extractor for: Among them, λ1, λ2 and λ3 are and The weight parameter of .
6. The face deocclusion method based on feature reconstruction according to claim 5, characterized in that: In step 4, the face generator includes a coarse face generator and a refined face generator. The coarse face generator includes eight encoders and eight decoders. Each encoder downsamples the refined occlusion mask map through a convolution layer, and the decoder upsamples the output of the encoder through a transposed convolution layer, restoring the feature map to the size of the input refined occlusion mask map, thereby obtaining a coarse face image. According to the refined binary occlusion mask map, a coarse synthetic face image is obtained. Among them, I f-occ is the input occluded face image, is the output rough face image, is the rough synthetic face image obtained; The rough face generator is trained using pixel reconstruction loss, adversarial loss, and gender consistency loss: Pixel reconstruction loss is defined as: in, Represents the pixel reconstruction loss of the valid area, represent The number of non-zero elements in I gt represents the original face image, Represents the pixel reconstruction loss of the occluded area, sum(M r ) represents M r The number of non-zero elements in , λ o is the weight parameter; The generative adversarial loss is defined as: represents the generative adversarial loss of the generator, represents the expectation of the distribution function, represents the distribution of generated data, I mer represents the samples sampled from the generating data distribution, Represents the discriminator to generate data The judgment result of represents the generative adversarial loss of the discriminator, P data represents the distribution of real data, I represents the sample obtained from the real data distribution, D(I gt ) represents the discriminator's response to the real data I gt The judgment result of Gender consistency loss Defined as: in, Represents the real image I gt The probability of being male, Represents the generated image Probability of being male; Then, the goal of the rough face generator is for: Among them, λ G and λ gen They are and The weight parameter of The refined face generator includes a local area refinement module and an important area refinement module: The local area refinement module extracts information from the neighboring area through the residual network; the important area refinement module uses the pixel similarity within the feature to identify the pixels related to the missing pixels from the valid pixels of the corresponding feature, fills the occluded area, and adopts the adaptive merging method to fuse the outputs of the local area refinement module and the important area refinement module to generate a refined binary occluder mask image. According to the refined binary occluder mask image, the input image and the generated refined face image are merged to obtain a refined synthetic face image. Among them, I f-occ is the input occluded face image, is the output refined face image, is the obtained refined synthetic face image; The refined face generator is trained using pixel reconstruction loss, perceptual loss, style loss, smoothness loss, and identity loss: Pixel reconstruction loss is defined as: in, Represents the pixel reconstruction loss of the valid area, represent The number of non-zero elements in I gt represents the original face image, Represents the pixel reconstruction loss of the occluded area, sum(M r ) represents M r The number of non-zero elements in , λ o is the weight parameter; Perceptual loss is defined as: Among them, φ i represents the feature map output by the i-th layer in the pre-trained VGG-16 network, i∈{5,10,17}; Style Loss is defined as: in, is the Gram matrix; Smoothing loss is defined as: identity loss is defined as: in, and I gt and The output vector of the input face recognition network, ∈, is a minimum value of 1e-8 to avoid division by zero; Ultimately, the goal of the refined face generator is: Among them, λ per ,λ sty ,λ tv and λ id They are and The weight parameter of .