Local face deocclusion method

Abstract

This invention discloses a local face deocclusion method, comprising: utilizing an improved pix2pixHD network to obtain an occluding mask and a fragmented face image corresponding to an occluded face image; utilizing a BiSeNet network to obtain a completed semantic segmentation map corresponding to the fragmented face image; and utilizing an improved SEAN network to restore the fragmented regions of the fragmented face under the guidance of the semantic segmentation map, thereby obtaining a deoccluded face image. This invention utilizes the face segmentation map to locally encode the face region, and combines it with unoccluded face regions to assist in the restoration of occluded regions, thereby improving the quality and accuracy of face deocclusion restoration.

Description

Technical Field

[0001] The present invention relates to the technical field of face deocclusion, and in particular to a local-based face deocclusion method. Background Art

[0002] Face deocclusion is a technique that, given an occluded face, recovers the unoccluded face. Occlusions are widely present in real-world scenarios, such as masks, sunglasses, and hands, and the presence of these occlusions can lead to performance degradation in downstream face-related tasks. Occlusions are diverse, involving different types of occluders with varying shapes and sizes. They can also appear in different locations on the face, including the eyes, mouth, and even half of the face, which increases the difficulty of face deocclusion. To address this challenge, face deocclusion technology has been proposed, which can be used as a preprocessing step for other face algorithms, such as face detection scenarios. Therefore, face deocclusion technology has a wide range of application scenarios.

[0003] Existing face deocclusion technologies have made many efforts to improve the face deocclusion effect, including the introduction of 3DMM, decoupled occlusion prediction, and multi-stage training methods. However, these methods usually rely on the prediction ability of neural networks to restore the complete face from the occluded face image, but ignore the more accurate restoration of the occluded area and the utilization of facial semantic information and unoccluded areas. Therefore, although they can achieve the goal of face deocclusion, the quality of the restored face is poor, which limits their application in downstream face technology; face restoration methods can also complete the incomplete areas of the face, but these methods are not designed for face deocclusion scenarios. They cannot automatically remove occluders and therefore cannot be directly applied to face deocclusion scenarios. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and propose a local face deocclusion method. Under the guidance of the semantic segmentation map, the encoding of local semantic regions and spatial correlation modulation are realized to take into account the semantic region information. At the same time, the complete face branch of the true face coding is introduced to improve the quality and accuracy of face restoration.

[0005] To achieve the above-mentioned purpose, the present invention provides a technical solution: a local face deocclusion method, comprising the following steps:

[0006] S1: Collect wild occluded face images, preprocess the wild occluded face data, and obtain occluded face images;

[0007] S2: The occluded face image is input into the trained improved pix2pixHD network. The improvement of the pix2pixHD network is to add a layer of Sigmoid activation function on the basis of the original pix2pixHD network to obtain the occlusion mask corresponding to the occluded face image between 0 and 1. The incomplete face image after deocclusion is calculated by the occlusion mask and the occluded face image.

[0008] S3: Input the incomplete face image into the trained BiSeNet network to obtain the completed semantic segmentation map corresponding to the incomplete face image;

[0009] S4: The completed semantic segmentation map and the incomplete face image are input into the trained improved SEAN network to obtain a high-quality de-occluded face image; wherein, the improvement of the SEAN network is as follows: during the training process, a true face image encoder is introduced in the feature extraction part, the true face image is input, and the true face image features are obtained through convolution to help the incomplete face image achieve more accurate encoding, and an incomplete face image branch is introduced in the generator feature modulation part, the incomplete face image is input, and the convolution encoding is used to intermediate embedding and corresponding modulation parameters to restore the existing areas in the incomplete face image.

[0010] Furthermore, in step S2, the occluded face image first undergoes downsampling, residual block and upsampling, and then the Sigmoid activation function is used to map the network output to (0,1) to obtain the soft mask of the occluder. Next, soft mask the occluder Thresholding is performed to obtain the occlusion mask. For each pixel, if the probability value is greater than 0.5, it will be set to 1, otherwise it will be set to 0, where 0 represents the current pixel being occluded by the occlusion object, and 1 represents the current pixel being unoccluded by the occlusion object. Finally, the occlusion mask is multiplied by the occluded face image at each pixel position to obtain the deoccluded incomplete face image. The loss function for the improved pix2pixHD network training is specifically set as:

[0011]

[0012] In the formula, BCE represents the binary cross entropy, L m represents the loss of the improved pix2pixHD network, represents the soft mask of the occluder output by the improved pix2pixHD network, S m Represents the true value of the occluder soft mask.

[0013] Furthermore, in step S3, the incomplete face image with a dimension of b×3×128×128 is input, and the completed semantic segmentation map with a dimension of b×19×512×512 is output, which is then resized to b×19×128×128, where b is the batch size. The loss function of the BiSeNet network training is specifically set to:

[0014]

[0015] Where CE represents cross entropy, L fs Represents the loss of the BiSeNet network, Represents the completed semantic segmentation map output by the BiSeNet network, S fs Represents the true value of the completed semantic segmentation map.

[0016] Furthermore, the specific operation steps of step S4 are:

[0017] S41: Use the incomplete face image encoder to extract the incomplete face image features corresponding to the incomplete face image. The incomplete face image encoder consists of 4 layers of 3×3 convolutions, uses LeakyReLU as the activation function, inputs an image of dimension b×3×128×128, and outputs incomplete face image features of dimension b×256×128×128. During the network training process, a true face image encoder will also be introduced. The true face image encoder takes the true face image corresponding to the incomplete face image as input and outputs the true face image features. The true face image encoder and the incomplete face image encoder have the same encoder architecture. During the training process, the features of the two encoders are each used with a probability of 50% to help the network better learn the feature representation of the incomplete face.

[0018] S42: Local semantic feature encoding is calculated using the incomplete face image features and the corresponding completed semantic segmentation map: the incomplete face image features obtained by the incomplete face image encoder are subjected to a 3×3 convolution, using LeakyReLU as the activation function to obtain a final face image feature with a dimension of b×128×128×128. Subsequently, the final face image feature and the completed semantic segmentation map are expanded to the same dimension b×19×128×128×128 through an extension operation and multiplied. Finally, global average pooling is performed on the dimensions representing height and width to obtain a local semantic feature encoding with a dimension of b×19×128. For any local semantic region, there is a corresponding vector describing the appearance content of the region.

[0019] S43: The local semantic feature encoding and the completed semantic segmentation map are used as conditions to inject into the generator to restore the de-occluded face image; the generator consists of 5 restoration blocks with resolutions of {16×16, 32×32, 32×32, 64×64, 128×128}. Each restoration block contains two 3×3 convolutions, two ReLUs and two restoration modulation blocks. The output features of the restoration block will be added to the input features of the restoration block and sent to the restoration block of the next resolution; for the restoration block, it passes the restoration conditions to the features through the restoration modulation block; referring to the completed semantic segmentation map, the semantic classification of each pixel position can be obtained, and then the local semantic feature encoding is filled according to the semantic classification of each pixel, thereby obtaining a semantic style feature map of dimension b×128×h×w, where b is the batch size, h is the height, w is the width, and 128 is the channel length of the local semantic feature encoding;

[0020] Each restoration and modulation block first resizes the semantic style feature map and the mutilated face image to the resolution size corresponding to each restoration block. Then, the mutilated face image is mapped to the mutilated face intermediate embedding using 3×3 convolution. Four 3×3 convolutions are then used to predict the scale and offset based on the resized semantic style feature map and the mutilated face intermediate embedding. The scale and offset are then combined using two learnable weights. Finally, spatial normalization is applied to achieve feature modulation.

[0021] S44: For the improved SEAN network, the adversarial approach is used to complete the training and the adversarial loss L is introduced. adv :

[0022]

[0023] Where D represents the discriminator, represents the deoccluded face image finally generated by the improved SEAN network, I represents the ground-truth face image, and E represents the expected operation. During the training process, the improved SEAN network attempts to generate more realistic deoccluded face images, while the discriminator attempts to correctly distinguish between real face images and generated face images.

[0024] S45: In order to further stabilize the training and help convergence, the feature matching loss L is introduced FM :

[0025]

[0026] Where N represents the number of intermediate layers, i is the corresponding index, and ||.||1 represents L1 loss;

[0027] S46: Introducing reconstruction loss L rec, the deoccluded face images generated by the improved SEAN network are further constrained at the image pixel and perception levels, encouraging the generated deoccluded face images to be close to the true value of the deoccluded face images:

[0028]

[0029] Where Φ represents the pre-trained VGG network, and k represents the selected network layer;

[0030] S47: The total loss function L of the final improved SEAN network total for:

[0031] L total =λ rec L rec +λ adv L adv +λ FM L FM

[0032] Where λ rec ,λ adv ,λ FM It is a hyperparameter used to adjust the reconstruction loss, adversarial loss, and feature matching loss.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. The present invention uses the face segmentation map to encode the local area. Compared with the existing direct global recovery technology, the present invention can perform more fine-grained recovery of the face.

[0035] 2. The present invention improves the restoration quality and accuracy of face deocclusion by locally encoding the semantic area and using the features of the unoccluded area as a hint for the occluded area.

[0036] 3. The present invention introduces a true face branch, which helps the network learn to supplement the missing facial appearance information by providing more complete and comprehensive facial features, and further achieves more accurate face deocclusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a framework diagram of the method of the present invention.

[0038] Figure 2 The framework diagram of the improved SEAN network.

[0039] Figure 3 Schematic diagram of the recovery block.

[0040] Figure 4 Schematic diagram of the recovery modulation block. DETAILED DESCRIPTION

[0041] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0042] like Figure 1 As shown, this embodiment provides a local face deocclusion method using a deep learning algorithm, which includes the following steps:

[0043] 1) Use a public face dataset and common face occluders collected from the Internet to create a face deocclusion dataset. The face deocclusion dataset contains true face images, occluder mask images, and synthetic occluded face images. The true face comes from the public face dataset CelebAHQ, with a resolution of 128×128. The occluders include four categories: hands, masks, cups, and glasses. Among them, there are 16 types of cups, 34 types of glasses, 29 types of hands, and 24 types of masks. The synthetic occluded face is the image obtained by placing the occluder image on the deoccluded face image. The occluder mask image is a black and white image used to indicate the size and position of the occluder.

[0044] Specifically, a publicly available facial landmark detector is first used to detect the ground-truth face image, generating facial landmarks, encompassing 68 feature points across the facial features and contours. Next, based on the type of occlusion and referring to the facial landmarks, the occlusion is randomly placed in the corresponding area of ​​the ground-truth face image. Glasses and sunglasses are placed in the eye area of ​​the ground-truth face; a water cup is placed in the mouth area of ​​the ground-truth face; a mask is placed in the lower facial area of ​​the ground-truth face; and a hand is placed anywhere on the face.

[0045] 2) The occluded face image is fed into a trained, improved pix2pixHD network. This network improves upon the original pix2pixHD network by adding a sigmoid activation function. This generates an occlusion mask between 0 and 1 corresponding to the occluded face image. Furthermore, the deoccluded face image is calculated using the occlusion mask and the occluded face image.

[0046] The occluded face image is first processed through downsampling, residual block, and upsampling. Then, the Sigmoid activation function is used to map the network output to (0, 1) to obtain a soft mask of the occluder with a dimension of b×2×128×128. Next, the occluder soft mask is thresholded. If the probability value is greater than 0.5, it will be set to 1, otherwise it will be set to 0, and the occluder mask corresponding to the occluded face image is obtained, with a dimension of b×1×128×128. For each pixel, the value is 0 or 1, 0 represents that the current pixel is occluded by the occluder, and 1 represents that the current pixel is not occluded by the occluder. Finally, the occluder mask is multiplied by the occluded face image at each pixel position to obtain the incomplete face image after deocclusion, with a dimension of b×3×128×128. Among them, b is the batch size. The loss function of the improved pix2pixHD network training is specifically set to:

[0047]

[0048] In the formula, BCE represents the binary cross entropy, L m represents the loss of the improved pix2pixHD network, represents the soft mask of the occluder output by the improved pix2pixHD network, S m Represents the true value of the occluder soft mask.

[0049] 3) Input the incomplete face image into the trained BiSeNet network to obtain the completed semantic segmentation map corresponding to the incomplete face image;

[0050] The input is a partial face image with a dimension of b×3×128×128, and the output is a completed semantic segmentation map with a dimension of b×19×512×512. The dimension is then resized to b×19×128×128. Where b is the batch size. The loss function for BiSeNet network training is specifically set to:

[0051]

[0052] Where CE represents cross entropy, L fs Represents the loss of the BiSeNet network, Represents the completed semantic segmentation map output by the BiSeNet network, S fs Represents the true value of the completed semantic segmentation map.

[0053] 4) Input the completed semantic segmentation map and the incomplete face image into the trained improved SEAN network to obtain a high-quality de-occluded face image. The improvement of the SEAN network is as follows: during the training process, a true face image encoder is introduced in the feature extraction part, the true face image is input, and the true face image features are obtained through convolution to help the incomplete face image achieve more accurate encoding. The incomplete face image branch is introduced in the generator feature modulation part, the incomplete face image is input, and the convolution encoding is used to embed the intermediate and the corresponding modulation parameters to restore the existing area in the incomplete face image. Figure 2As shown, the specific steps are:

[0054] The incomplete face image encoder is used to extract the incomplete face image features corresponding to the incomplete face image. The incomplete face image encoder consists of four layers of 3×3 convolutions, uses LeakyReLU as the activation function, takes an image of dimension b×3×128×128 as input, and outputs incomplete face image features of dimension b×256×128×128. During the network training process, a ground truth face image encoder is also introduced. This ground truth face image encoder takes the ground truth face image corresponding to the incomplete face image as input and outputs ground truth face image features. This ground truth face encoder and the incomplete face image encoder have the same encoder architecture. During training, the features of the two encoders are each used with a probability of 50% to help the network better learn the feature representation of the incomplete face.

[0055] The local semantic feature encoding is calculated using the incomplete facial image features and the corresponding completed semantic segmentation map. The incomplete facial image features obtained by the incomplete facial image encoder are subjected to a 3×3 convolution layer, using LeakyReLU as the activation function, to obtain a final facial image feature with a dimension of b×128×128×128. Subsequently, the final facial image features and the completed semantic segmentation map are expanded to the same dimension of b×19×128×128×128 through an expansion operation and multiplied together. Finally, a global average pooling is performed on the dimensions representing height and width to obtain a local semantic feature encoding with a dimension of b×19×128. In other words, for any local semantic region, there is a corresponding vector that describes the appearance content of the region.

[0056] Finally, the local semantic feature encoding and the completed semantic segmentation map are used as conditions to inject into the generator to restore the unoccluded face image. The generator consists of 5 restoration blocks with resolutions of {16×16, 32×32, 32×32, 64×64, 128×128}. Figure 3 As shown, each recovery block contains two 3×3 convolutions, two ReLUs, and two recovery modulation blocks. The output features of the recovery block are added to the input features of the recovery block and sent to the recovery block of the next resolution. For the recovery block, it passes the recovery conditions to the features by modulating the feature scale and bias through the recovery modulation block. Specifically, referring to the completed semantic segmentation map, the semantic classification of each pixel position can be obtained. Then, the local semantic feature encoding is filled according to the semantic classification of each pixel. Thus, a semantic style feature map of dimension b×128×h×w can be obtained, where b is the batch size, h is the height, w is the width, and 128 is the channel length of the local semantic feature encoding.

[0057] like Figure 4As shown in the figure, for each restoration modulation block, the semantic style feature map and the mutilated face image are first resized to the resolution size corresponding to each restoration block. The mutilated face image is then mapped to the mutilated face intermediate embedding using a 3×3 convolution. Four 3×3 convolutions are then used to predict the scale and offset based on the resized semantic style feature map and the mutilated face intermediate embedding. The scale and offset obtained above are then combined using two learnable weights. Finally, spatial normalization is applied to achieve feature modulation.

[0058] For the improved SEAN network, the training is completed in an adversarial manner, and the adversarial loss L is introduced. adv :

[0059]

[0060] Where D represents the discriminator, represents the deoccluded face image finally generated by the improved SEAN network, I represents the ground-truth face image, and E represents the expected operation. During the training process, the improved SEAN network attempts to generate more realistic deoccluded face images, while the discriminator attempts to correctly distinguish between real face images and generated face images;

[0061] In order to further stabilize the training and help convergence, the feature matching loss L is introduced FM :

[0062]

[0063] Where N represents the number of intermediate layers, i is the corresponding index, and ||.||1 represents L1 loss;

[0064] In addition, the reconstruction loss L is introduced rec , the deoccluded face images generated by the improved SEAN network are further constrained at the image pixel and perception levels, encouraging the generated deoccluded face images to be close to the true value of the deoccluded face images:

[0065]

[0066] Where Φ represents the pre-trained VGG network, and k represents the selected network layer;

[0067] The loss function L of the final improved SEAN network total for:

[0068] L total =λ rec L rec +λ adv L adv +λ FM L FM

[0069] Where λ rec ,λ adv ,λ FM It is a hyperparameter used to adjust the reconstruction loss, adversarial loss, and feature matching loss.

[0070] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A local face deocclusion method, characterized in that: The following steps are involved: S1: Collect wild occluded face images, preprocess the wild occluded face data, and obtain occluded face images; S2: The occluded face image is input into the trained improved pix2pixHD network. The improvement of the network is to add a layer of Sigmoid activation function on the basis of the original pix2pixHD network to obtain the occlusion mask corresponding to the occluded face image between 0 and 1. The incomplete face image after deocclusion is calculated by the occlusion mask and the occluded face image. S3: Input the incomplete face image into the trained BiSeNet network to obtain the completed semantic segmentation map corresponding to the incomplete face image; S4: Input the completed semantic segmentation map and the incomplete face image into the trained improved SEAN network to obtain a high-quality de-occluded face image, including: S41: Use the incomplete face image encoder to extract the incomplete face image features corresponding to the incomplete face image. The incomplete face image encoder consists of 4 layers of 3×3 convolutions, uses LeakyReLU as the activation function, takes an image of dimension b×3×128×128 as input, and outputs incomplete face image features of dimension b×256×128×128. Introduce the true face image encoder, which takes the true face image corresponding to the incomplete face image as input and outputs the true face image features. The true face image encoder and the incomplete face image encoder have the same encoder architecture, and the features of the two encoders are each used with a probability of 50%. S42: Local semantic feature encoding is calculated using the incomplete face image features and the corresponding completed semantic segmentation map: the incomplete face image features obtained by the incomplete face image encoder are subjected to a layer of 3×3 convolution, using LeakyReLU as the activation function to obtain a final face image feature with a dimension of b×128×128×128. Subsequently, the final face image feature and the completed semantic segmentation map are expanded to the same dimension b×19×128×128×128 through an extension operation and multiplied. Finally, global average pooling is performed on the dimensions representing height and width to obtain a local semantic feature encoding with a dimension of b×19×128. S43: The local semantic feature encoding and the completed semantic segmentation map are injected into the generator as conditions to restore the de-occluded face image; the generator consists of 5 restoration blocks with resolutions of {16×16, 32×32, 32×32, 64×64, 128×128}. Each restoration block contains two 3×3 convolutions, two ReLUs, and two restoration modulation blocks. The output features of the restoration block are added to the input features of the restoration block and sent to the restoration block of the next resolution; for the restoration block, it passes the restoration conditions to the features through the restoration modulation block; refer to the completed semantic segmentation map to obtain the semantic classification of each pixel position, and then fill the local semantic feature encoding according to the semantic classification of each pixel to obtain a semantic style feature map of dimension b×128×h×w, where b is the batch size, h is the height, w is the width, and 128 is the channel length of the local semantic feature encoding; Each restoration modulation block adjusts the semantic style feature map and the incomplete face image to the resolution size corresponding to each restoration block, maps the incomplete face image to the incomplete face intermediate embedding using 3×3 convolution, and then uses four 3×3 convolutions to predict the scale and offset based on the semantic style feature map after adjusting the resolution and the incomplete face intermediate embedding. The scale and offset obtained previously are combined through two learnable weights, and spatial-based normalization is applied to achieve feature modulation.

2. The local face deocclusion method according to claim 1, characterized in that: In step S2, the occluded face image first goes through three parts: downsampling, residual block and upsampling. Then the Sigmoid activation function is used to map the network output to (0,1) to obtain the soft mask of the occluder. ; Next, soft mask the occluder Thresholding is performed to obtain the occlusion mask. For each pixel, if the probability value is greater than 0.5, it will be set to 1, otherwise it will be set to 0, where 0 represents the current pixel being occluded by the occlusion object, and 1 represents the current pixel being unoccluded by the occlusion object. Finally, the occlusion mask is multiplied by the occluded face image at each pixel position to obtain the deoccluded incomplete face image. The loss function for the improved pix2pixHD network training is specifically set as: ； Where BCE represents the binary cross entropy, represents the loss of the improved pix2pixHD network, represents the soft mask of occluders output by the improved pix2pixHD network, Represents the true value of the occluder soft mask.

3. The local face deocclusion method according to claim 1, characterized in that: In step S3, the incomplete face image with the dimension of b×3×128×128 is input, and the completed semantic segmentation map with the dimension of b×19×512×512 is output, which is then resized to b×19×128×128, where b is the batch size. The loss function of BiSeNet network training is specifically set to: ； Where CE represents cross entropy, Represents the loss of the BiSeNet network, Represents the completed semantic segmentation map output by the BiSeNet network, Represents the true value of the completed semantic segmentation map.

4. The local face deocclusion method according to claim 1, characterized in that: The step S4 further includes: S44: For the improved SEAN network, an adversarial approach is used to complete the training and introduce adversarial loss : ； Where, represents the discriminator, represents the de-occluded face image finally generated by the improved SEAN network, represents the true face image, represents the expectation operation; during the training process, the improved SEAN network attempts to generate more realistic de-occluded face images, while the discriminator attempts to correctly distinguish between real face images and generated face images; S45: In order to further stabilize training and help convergence, feature matching loss is introduced : ； Where, Indicates the number of middle layers, is the corresponding index, express loss; S46: Introducing reconstruction loss , the deoccluded face images generated by the improved SEAN network are further constrained at the image pixel and perception levels, encouraging the generated deoccluded face images to be close to the true value of the deoccluded face images: ； Where, represents the pre-trained VGG network, Indicates the selected network layer; S47: The total loss function of the final improved SEAN network for: ； Where, 、 、 It is a hyperparameter used to adjust the reconstruction loss, adversarial loss, and feature matching loss.

Images