A Deep Learning-Based Method for Removing Shadows from Faces

Abstract

This invention discloses a deep learning-based method for removing shadows from faces. The method includes the following steps: constructing a self-sample illumination equalization network, including a downsampling layer, multiple cascaded self-sample illumination equalization blocks, and a decoder; the downsampling layer is used to downsample and obtain feature maps; the self-sample illumination equalization blocks are used to obtain facial features for removing shadows; the decoder is used to obtain a shadow-free face image; acquiring a training dataset of shadowed face images, inputting it into the self-sample illumination equalization network for end-to-end training, updating the network parameters to obtain a trained self-sample illumination equalization network; inputting the face image to be removed into the trained self-sample illumination equalization network, and outputting a shadow-free face image. This invention achieves the removal of facial shadows by establishing a correspondence between shadowed and shadow-free regions on the face and extracting illumination information to modulate the input facial feature image, thus ensuring the integrity of the facial structure while achieving uniform illumination.

Description

Technical Field

[0001] This invention belongs to the technical field of deep learning and image processing, specifically relating to a deep learning-based method for removing shadows from facial images to achieve facial illumination balance. Background Technology

[0002] Shadows are phenomena formed on the surface of objects due to occlusion or uneven lighting. Shadows affect the aesthetics of images, degrade image quality, and adversely impact many downstream image processing tasks. Shadow removal is a fundamental task in computer vision image processing, primarily aimed at removing shadowed areas from images to harmonize them with the surrounding lighting and colors. Image shadow removal algorithms have matured over the years, progressing from traditional methods to deep learning-based approaches. In recent years, with the increasing prevalence of technologies such as face recognition, face shadow removal has begun to attract researchers' attention as a subcategory of shadow removal. The paper "Portrait Shadow Manipulation" was the first to treat face shadow removal as an independent task.

[0003] Existing face shadow removal techniques are all implemented using deep learning. They synthesize a dataset of shadowed face images from real, shadow-free face images, and then train an end-to-end deep learning network using this dataset to remove facial shadows. For example, the paper "Portrait Shadow Manipulation" uses a real face dataset and different light source configurations to construct paired datasets of self-shadows, and then uses two GridNet networks to handle these two shadow types respectively. One GridNet network handles external shadows, and its output weights and biases are used to perform affine transformations on the original shadow image; while the other GridNet network handles self-shadows to lighten them. However, due to limitations in the method of training data synthesis, it sometimes struggles to handle symmetrical shadows and some smaller shadows (such as the shadows cast by hair on the skin), thus potentially failing to remove shadows effectively and resulting in poor performance. The paper "Unsupervised Portrait Shadow Removal via Generative Priors" proposes an unsupervised face shadow detection and removal method based on StyleGAN2. It uses a pre-trained StyleGAN2 model as the main network structure and combines it with a simple CNN to generate shadow masks. Then, a progressive optimization strategy is used to remove shadows from face images without the need to prepare pairs of real or synthetic data for training. Although this method avoids the difficulty of data collection and does not require additional training time and resources, if the face information in the test image is not in the hidden space of StyleGAN2, the generated shadowless face may have severe deformation or distortion, resulting in poor performance in practical applications. The paper "Blind Removal of Facial Foreign Shadows" proposes a novel shadow modeling method. The paper finds that converting a colored face image with shadows to grayscale simplifies shadow removal and improves the model's generalization performance on new test images. Therefore, it first converts the colored face image with shadows to grayscale, preserving the image's color information. Then, it uses the grayscale image with shadows removed and compares it with the original image to generate a shadow mask. Finally, it uses the shadow mask and color information to restore the grayscale image to a color image. However, because it cannot effectively establish the relationship between shadowed and non-shadowed areas on the face, this method is not very effective at removing shadows and preserving facial details.

[0004] It is evident that none of the above methods can simultaneously guarantee the effectiveness of shadow removal and the integrity of the facial structure after shadow removal. Therefore, it is necessary to study a method for removing shadows from the face that can effectively remove shadows while also preserving facial structural details relatively completely. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a deep learning-based method for removing shadows from faces. By establishing a correspondence between shadowed and unshadowed areas on the face and extracting illumination information to modulate the input facial feature image, the method achieves uniform illumination while ensuring the integrity of the facial structure, thereby removing facial shadows.

[0006] To achieve the above objectives, this invention provides a deep learning-based method for removing shadows from faces, comprising the following steps:

[0007] A self-sample illumination equalization network is constructed; the self-sample illumination equalization network includes a downsampling layer, multiple cascaded self-sample illumination equalization blocks, and a decoder; the downsampling layer is used to downsample to obtain feature maps; the self-sample illumination equalization blocks are used to obtain face features with shadows removed; the decoder is used to obtain shadow-free face images;

[0008] Obtain the training dataset of shadowed face images, input it into the self-sample illumination equalization network for end-to-end training, and update the network parameters to obtain the trained self-sample illumination equalization network.

[0009] The face image to be removed from the shadow is input into the trained self-sample illumination equalization network, and the output is a face image without shadow.

[0010] As a preferred technical solution, the downsampling layer consists of a series of convolutional layers, used to downsample and obtain a feature map; the convolutional layer reduces the size of the input image or feature map to half of its original size;

[0011] The self-sample illumination equalization block includes a self-sample illumination feature matching module and a feature-level illumination correction module; the self-sample illumination feature matching module is used to extract the correspondence map between shadow and non-shadow areas of the feature map and sample and re-match to obtain new facial features; the feature-level illumination correction module is used to predict the illumination factor and perform illumination correction on the new facial features to obtain facial features without shadows.

[0012] The decoder consists of a series of deconvolutional layers used to restore shadow-free facial features to a shadow-free facial image of the same size as the images in the training dataset.

[0013] As a preferred technical solution, the self-sample illumination feature matching module includes a correspondence predictor and a feature extractor; both the correspondence predictor and the feature extractor are composed of a series of convolutional layers; the correspondence predictor is used to obtain a correspondence map between shadow and non-shadow regions of the feature map; the feature extractor is used to extract facial features from the feature map, and sample and reassign new facial features based on the correspondence map; the correspondence map between shadow and non-shadow regions is used to represent the coordinates of non-shadow region pixels corresponding to each pixel in the input feature map learned by the correspondence predictor;

[0014] The feature-level illumination correction module includes two illumination factor predictors and an illumination correction module; each of the two illumination factor predictors consists of three convolutional layers and is used to predict illumination factors; the illumination correction module performs an affine transformation on the normalized feature map based on the two predicted illumination factors to obtain face features with shadows removed.

[0015] As a preferred technical solution, the end-to-end training specifically includes:

[0016] The training dataset is input into the downsampling layer for downsampling to obtain the feature map;

[0017] The feature map is input into the first self-sample illumination equalization block to obtain the first corrected face features;

[0018] The first corrected facial features are used as the feature map and input into the second self-sample illumination equalization block to obtain the second corrected facial features;

[0019] This process is repeated until the last self-sample lighting equalization block is reached. The last corrected facial feature output by this block is the facial feature with the shadows removed.

[0020] The facial features with the shadows removed are input into the decoder and deconvolution is performed to obtain a shadow-removed face image with the same size as the image in the training dataset.

[0021] Calculate the loss function, update the network parameters, and iterate until the self-sample illumination equalization network converges.

[0022] As a preferred technical solution, the step of inputting the feature map into the first self-sample illumination equalization block to obtain the first corrected facial features specifically involves:

[0023] The correspondence predictor is used to obtain the correspondence map between the shaded and unshaded regions of the feature map;

[0024] The feature extractor extracts features from the feature map to obtain facial features;

[0025] Based on the correspondence between shaded and unshaded areas in the feature map, the pixels in the face features are sampled and the pixel values ​​are reassigned to obtain the rematched features.

[0026] The rematched features are input into two illumination factor predictors to predict the illumination factor;

[0027] Based on the two predicted illumination factors, the illumination correction module performs an affine transformation on the normalized feature map to obtain the first corrected facial features.

[0028] As a preferred technical solution, the size of the correspondence map between the shadow and non-shadow regions is the same as the size of the input feature map, and the number of channels is 2.

[0029] As a preferred technical solution, the rematching features are obtained as follows:

[0030] Let P be the pixel value matrix of the pixels in the facial features. m×n =(p i,j ) m×n The correspondence between shaded and unshaded areas is represented by R. m×n =(r i,j ) m×n Where i≤m, j≤n; r i,j Let be the value of the pixel coordinates (i,j) in the correspondence graph, which is the coordinate of a pixel in the feature map; m is the length of the feature map, n is the width of the feature map, and p i,j It represents the pixel value at coordinates (i,j) of the pixel point in the face feature.

[0031] The pixel values ​​of each pixel in the facial features are redistributed based on the correspondence graph, i.e., rp i,j =p ri,j ;

[0032] This yields the rematch feature, denoted as RP. m×n =(rp i,j ) m×n .

[0033] As a preferred technical solution, the illumination factor consists of two feature maps whose size and number of channels are completely consistent with the input rematch features.

[0034] As a preferred technical solution, the step of performing illumination correction to obtain the first corrected facial features specifically involves:

[0035] Normalize the feature map channel by channel to obtain the normalized feature map, represented as:

[0036]

[0037] Where h is the normalized feature map, J is the number of pixels in the feature map, and M is the number of channels in the feature map. m,j Let μ be the pixel value corresponding to the j-th pixel in the m-th channel of the feature map. m σ is the average pixel value of the m-th channel in the feature map; m Let be the standard deviation of the pixel value of the m-th channel in the feature map;

[0038] Perform an affine transformation on the normalized feature map and apply illumination correction to obtain the first corrected facial features, represented as:

[0039] h′1=γh+β

[0040] Where h′1 is the first corrected facial feature, and γ and β are both illumination factors.

[0041] As a preferred technical solution, when the self-sample illumination equalization network is trained end-to-end, the weighted sum of the L1 loss function and the perceptual loss function is used as the final loss function, expressed as:

[0042] L total =L1+L P ;

[0043] Where L1 is the L1 loss function, expressed as:

[0044]

[0045] In the formula, To obtain the shadow-free face image through training, x f The image is the real, shadowless face image corresponding to the shadowed face image; ||·|1 represents the L1 norm;

[0046] L P The perceptual loss function is expressed as:

[0047]

[0048] In the formula, Φ i (·) represents the feature map output from the i-th layer of the pre-trained feature extractor inputting the image.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] Existing facial shadow removal technologies cannot simultaneously achieve effective shadow removal while preserving the integrity of facial structure and details after shadow removal. This invention, however, can remove facial shadows while maintaining the integrity of facial details and structure. Since the human face is inherently symmetrical, and the skin texture and color are roughly the same across different areas of the same face, this application's approach of using illumination information from shadow-free areas to guide facial shadow removal is clearly more reasonable than the conventional approach of detecting shadow areas and independently restoring their color. This allows the color of the original shadow area on the face to more closely match the surrounding skin color after shadow removal, avoiding obvious color unevenness and blurred boundaries, thus improving the visual effect. This invention first designs a self-sample illumination feature matching module to establish a correspondence between shadow and shadow-free areas. Then, it samples specific pixel values ​​from shadow-free areas in the facial features and redistributes them to corresponding positions in the shadow areas to guide shadow removal.

[0051] However, if we stop here and only use the upsampling of the facial features after sampling and redistribution to obtain the final shadow removal result, the same problem as previous techniques will occur, namely, the inability to guarantee the integrity of facial details and structure. This is because the previous sampling and redistribution process simply transferred pixel values ​​between regions without properly distinguishing the lighting information and facial structure information contained therein, which leads to some misalignment of facial information in the resulting image. To solve this problem, this invention creatively proposes a two-stage face shadow removal solution; the first stage is as described above; in the second stage, a feature-level lighting correction module is designed to extract the redistributed lighting information from the redistributed features and re-input it into the original shadowed face image after regression normalization. In this way, the misaligned facial structure information in the redistributed features will no longer affect the shadow removal result, ensuring the integrity of facial details and structure while removing facial shadows. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating a deep learning-based face shadow removal method in an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of the structure of a self-sample illumination equalization block in an embodiment of the present invention.

[0055] Figure 3 The shadowed face image in this embodiment of the invention is input from the sample illumination equalization network to obtain the shadow-removed face image. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0057] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0058] This invention, based on deep learning technology, uses a convolutional neural network to remove shadows from a human face. The main objective of this invention is to fully mine and utilize the internal lighting information of the face to guide and achieve accurate facial shadow removal while preserving complete facial details. Figure 1 As shown in the figure, this embodiment of a deep learning-based face shadow removal method includes the following steps:

[0059] S1. Construct a self-sample illumination equalization network, including a downsampling layer, multiple cascaded self-sample illumination equalization blocks, and a decoder; wherein, the downsampling layer is used to downsample to obtain feature maps; the self-sample illumination equalization blocks are used to obtain face features with shadows removed; and the decoder is used to obtain shadow-free face images.

[0060] S2. Obtain the training dataset of shadowed face images, input it into the self-sample illumination equalization network for end-to-end training, update the network parameters to obtain the trained self-sample illumination equalization network.

[0061] S3. Input the face image to be removed into the trained self-sample illumination equalization network, and output the face image without shadows.

[0062] Furthermore, since the face images input to the network are generally large in size and contain some redundant information that will not be used later, in order to reduce unnecessary computation in subsequent image processing, this invention first constructs a downsampling layer consisting of a series of convolutional layers. Each time the image passes through a convolutional layer, the image side length is reduced to half of its original size. The reduced feature map obtained by the downsampling layer is then input into the subsequent modules.

[0063] To fully utilize the illumination information of shadowless areas on the face for precise shadow removal, this invention proposes a series of cascaded self-sample illumination equalization blocks; such as... Figure 2 As shown, each self-sample illumination equalization block consists of a self-sample illumination feature matching module and a feature-level illumination correction module. The self-sample illumination feature matching module extracts the correspondence between shadow and non-shadow areas of the feature map, samples and re-matches them to obtain new facial features. This module can identify the illumination information of shadowless areas and transfer it to shadowed areas to help remove shadows. The feature-level illumination correction module predicts the illumination factor and performs illumination correction on the new facial features to obtain shadow-free facial features. This module can accurately extract the illumination information from the re-matched features and apply it to the shadowed facial features, avoiding the introduction of unnatural facial features. Figure 2 China F i F represents the i-th self-sample lighting equalization block. i+1 h represents the (i+1)th self-sample lighting equalization block. i Let represent the feature map obtained from the i-th self-sample illumination equalization block, where γ and β are illumination factors, and W represents the sampling and rematching process.

[0064] Finally, a decoder is connected to output a shadowless face image, which consists of a series of deconvolution layers. Its function is to restore the shadowless face features to a shadowless face image with the same size as the input image.

[0065] Furthermore, the self-sampled illumination feature matching module includes a correspondence predictor and a feature extractor. Both the correspondence predictor and feature extractor consist of a series of convolutional layers that only adjust the number of channels in the input feature map without changing its size. The correspondence predictor obtains the correspondence map between shadow and non-shadow regions of the feature map. The feature extractor extracts facial features from the feature map and samples and redistributes these features based on the correspondence map to obtain new facial features. The correspondence map between shadow and non-shadow regions is essentially a feature map itself, representing the coordinates of pixels in the non-shadow region corresponding to each pixel in the input feature map learned by the correspondence predictor. This coordinates guide the sampling of pixels and the redistribution of pixel values ​​in the input facial features.

[0066] Although the self-sample illumination feature matching module allows full utilization of prior illumination information within the face to compensate for shadow areas, the operation of transferring illumination information from shadowless areas to shadow areas through the learned matching correspondence map is essentially a pixel sampling and redistribution process. This may lead to undesirable facial feature distortion and artifacts in the redistributed facial features. Therefore, directly using the new facial features output by the self-sample illumination feature matching module to generate the final result is not feasible. To fully utilize the illumination information in the new facial features while avoiding the introduction of unnatural facial features, this invention also designs a feature-level illumination correction module, including two illumination factor predictors and an illumination correction module. The two illumination factor predictors have identical structures, each consisting of three convolutional layers used to predict illumination factors. The number of channels in the new facial feature input to the illumination factor predictor is first halved and then restored. During this compression and expansion of the channel number, useful information (illumination information) is preserved, while useless information is removed. By retaining only the illumination information in the new facial features and removing potentially distorted facial structural information, the uniformity of facial illumination can be effectively improved without damaging the original structure and details of the facial features. After obtaining the illumination factors, the illumination correction module performs an affine transformation on the normalized feature map based on the two predicted illumination factors to obtain the shadow-removed facial features.

[0067] Furthermore, the acquired training dataset is input into the self-sample illumination equalization network for end-to-end training, and the network parameters are updated to obtain the trained self-sample illumination equalization network, specifically as follows:

[0068] S201. Input the training dataset into the downsampling layer to perform downsampling and obtain the feature map;

[0069] S202. Input the feature map into the first self-sample illumination equalization block to obtain the first corrected face features, including:

[0070] S2021. Use the correspondence predictor to obtain the correspondence map between the shaded and unshaded regions of the feature map; wherein, the size of the correspondence map between the shaded and unshaded regions is the same as the size of the input feature map, and the number of channels is 2, which is not affected by the number of channels of the input feature map.

[0071] S2022, The feature extractor extracts features from the feature map to obtain facial features; the facial features are also the same size and have the same number of channels as the feature map;

[0072] S2023. Based on the correspondence between shaded and unshaded areas in the feature map, the pixels in the face features are sampled and their values ​​are reassigned to obtain rematching features; the specific steps are as follows:

[0073] Let P be the pixel value matrix of the pixels in the facial features. m×n =(p i,j ) m×n The correspondence between shaded and unshaded areas is represented by R. m×n =(r i,j ) m×n Where i≤m, j≤n, r i,j Let (i,j) be the coordinates of pixel (i,j) in the feature map corresponding to the pixel coordinates in the mapping graph, m be the length of the feature map, n be the width of the feature map, and p be the width of the pixel. i,j It represents the pixel value at coordinates (i,j) of the pixel point in the face feature.

[0074] The pixel values ​​of each pixel in the facial features are redistributed based on the correspondence graph, i.e., rp i,j =p ri,j ;

[0075] This yields the rematch feature, denoted as RP. m×n =(rp i,j ) m×n .

[0076] The following example illustrates the process of obtaining rematching features:

[0077] Let the pixel value matrix of the pixels in the facial features be... The correspondence diagram is as follows For the pixel at coordinates (1,1), its rematched pixel value is rp. 1,1 =p r1,1 =p 1,3 =5, rematch each pixel, and obtain the rematch feature representation as follows:

[0078] S2024. Input the rematched features into two illumination factor predictors respectively to predict the illumination factor;

[0079] The facial features output from the self-sample illumination feature matching module are passed through two illumination factor prediction modules to output two illumination factors. These two illumination factors are actually two feature maps with the same length, width, and number of channels as the input facial features. They are used here to perform pixel-by-pixel modulation on the normalized facial feature map.

[0080] S2025. Based on the two predicted illumination factors, the illumination correction module performs an affine transformation on the normalized feature map to obtain the first corrected facial features; specifically including:

[0081] Normalize the feature map channel by channel to obtain the normalized feature map, represented as:

[0082]

[0083] Where h is the normalized feature map, J is the number of pixels in the feature map, and M is the number of channels in the feature map. m,j Let μ be the pixel value corresponding to the j-th pixel in the m-th channel of the feature map. m σ is the average pixel value of the m-th channel in the feature map; m Let be the standard deviation of the pixel value of the m-th channel in the feature map;

[0084] Perform an affine transformation on the normalized feature map and apply illumination correction to obtain the first corrected facial features, represented as:

[0085] h′1=γh+β

[0086] Where h′1 is the first corrected facial feature, and γ and β are both illumination factors.

[0087] S203. Input the first corrected face features as a feature map into the second self-sample illumination equalization block to obtain the second corrected face features;

[0088] S204. This process continues until the last self-sample lighting equalizer block. The nth corrected facial feature output by this block is the facial feature with the shadow removed. The process of obtaining the corrected facial feature from the self-sample lighting equalizer block is the same as that of the first self-sample lighting equalizer block, and will not be described again here.

[0089] S205. Input the face features with the face shadow removed into the decoder and perform a deconvolution operation to obtain a face image with the same size as the image in the training dataset;

[0090] S206. Calculate the loss function, update the network parameters, and iterate until the self-sample illumination equalization network converges.

[0091] Furthermore, in the end-to-end training process of the self-sample illumination equalization network in this application, the weighted sum of the L1 loss function and the perceptual loss function is used as the final loss function, expressed as:

[0092] L total =L1+L P ;

[0093] Where L1 is the L1 loss function, used to compare the differences in pixel values ​​between the shaded face image output by the network and the real shadowless face image, that is, the average of the differences in pixel values ​​at all corresponding positions between the two images, expressed as:

[0094]

[0095] In the formula, To obtain the shadow-free face image through training, xf The ground truth is the real, shadowless face image corresponding to the shadowed face image, where ||·|1 represents the L1 norm.

[0096] L P The perceptual loss function is the average of the pixel values ​​at corresponding positions between the shadow-removed face image output by the network and the real shadow-free face image, both processed by a pre-trained feature extractor. This loss function is used to ensure the visual quality of the output image and is expressed as:

[0097]

[0098] In the formula, Φ i (·) represents the feature map output from the i-th layer of the pre-trained feature extractor inputting the image.

[0099] In this embodiment, the feature extractor uses the VGG19 convolutional neural network model, which is only used to calculate the loss function during training.

[0100] In summary, the self-sample illumination equalization network proposed in this invention has better task adaptability and strong generalization in removing shadows from different face images. Unlike existing technologies that directly remove shadows from face images, this invention adopts an innovative two-stage design approach. First, it extracts illumination information from shadow-free areas, and then uses the extracted illumination information to compensate the input face, thus completing shadow removal. This results in the color and illumination of the original shadowed area being more similar to the shadow-free area, leading to a more natural visual effect for the entire face. It can be used for preprocessing in beautification or other downstream face tasks. Figure 3 As shown, the shadowed face image is input into the self-sample lighting equalization network, and the shadow-free face image is obtained after passing through six cascaded self-sample lighting equalization blocks.

[0101] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A deep learning-based method for removing shadows from faces, characterized in that, Includes the following steps: A self-sample illumination equalization network is constructed. This network includes a downsampling layer, multiple cascaded self-sample illumination equalization blocks, and a decoder. The downsampling layer is used to downsample and obtain a feature map. The self-sample illumination equalization blocks are used to obtain shadow-free facial features. The decoder is used to obtain a shadow-free facial image. Each self-sample illumination equalization block includes a self-sample illumination feature matching module and a feature-level illumination correction module. The self-sample illumination feature matching module extracts the correspondence between shadow and non-shadow regions of the feature map, samples, and re-matches to obtain new facial features. The feature-level illumination correction module predicts illumination factors and corrects the new facial features to obtain shadow-free facial features. The self-sample illumination feature matching module includes a correspondence predictor and a feature extractor. The feature-level illumination correction module includes two illumination factor predictors and an illumination correction module. Obtain the training dataset of shadowed face images, input it into the self-sample illumination equalization network for end-to-end training, and update the network parameters to obtain the trained self-sample illumination equalization network. The face image to be removed from the shadow is input into the trained self-sample illumination equalization network, and the output is a face image without shadow.

2. The face shadow removal method based on deep learning according to claim 1, characterized in that, The downsampling layer consists of a series of convolutional layers used to downsample and obtain feature maps; the convolutional layers reduce the size of the input image or feature map to half of its original size. The decoder consists of a series of deconvolutional layers used to restore shadow-free facial features to a shadow-free facial image of the same size as the images in the training dataset.

3. The face shadow removal method based on deep learning according to claim 2, characterized in that, Both the correspondence predictor and the feature extractor consist of a series of convolutional layers. The correspondence predictor is used to obtain the correspondence map between the shadow and non-shadow regions of the feature map. The feature extractor is used to extract the face features of the feature map, and to sample and redistribute them based on the correspondence map to obtain new face features. The correspondence map between the shadow and non-shadow regions is used to represent the coordinates of the non-shadow region pixels corresponding to each pixel in the input feature map learned by the correspondence predictor. Both illumination factor predictors consist of three convolutional layers and are used to predict illumination factors. The illumination correction module performs an affine transformation on the normalized feature map based on two predicted illumination factors to obtain face features with shadows removed.

4. The face shadow removal method based on deep learning according to claim 3, characterized in that, The end-to-end training is specifically performed as follows: The training dataset is input into the downsampling layer for downsampling to obtain the feature map; The feature map is input into the first self-sample illumination equalization block to obtain the first corrected face features; The first corrected facial features are used as the feature map and input into the second self-sample illumination equalization block to obtain the second corrected facial features; This process is repeated until the last self-sample lighting equalization block is reached. The last corrected facial feature output by this block is the facial feature with the shadows removed. The facial features with the shadows removed are input into the decoder and deconvolution is performed to obtain a shadow-removed face image with the same size as the image in the training dataset. Calculate the loss function, update the network parameters, and iterate until the self-sample illumination equalization network converges.

5. A face shadow removal method based on deep learning according to claim 4, characterized in that, The step of inputting the feature map into the first self-sample illumination equalization block to obtain the first corrected facial features is specifically as follows: The correspondence predictor is used to obtain the correspondence map between the shaded and unshaded regions of the feature map; The feature extractor extracts features from the feature map to obtain facial features; Based on the correspondence between shaded and unshaded areas in the feature map, the pixels in the face features are sampled and the pixel values ​​are reassigned to obtain the rematched features. The rematched features are input into two illumination factor predictors to predict the illumination factor; Based on the two predicted illumination factors, the illumination correction module performs an affine transformation on the normalized feature map to obtain the first corrected facial features.

6. The face shadow removal method based on deep learning according to claim 5, characterized in that, The size of the mapping between shadowed and unshadowed regions is the same as the size of the input feature map, and the number of channels is 2.

7. A face shadow removal method based on deep learning according to claim 6, characterized in that, The obtained rematch features are specifically as follows: Let P be the pixel value matrix of the pixels in the facial features. m×n =(p i,j ) m×n The correspondence between shaded and unshaded areas is represented by R. m×n =(r i,j ) m×n Where i≤m, j≤n; r i,j Let be the value of the pixel coordinates (i,j) in the correspondence graph, which is the coordinate of a pixel in the feature map; m is the length of the feature map, n is the width of the feature map, and p i,j It represents the pixel value at coordinates (i,j) of the pixel point in the face feature. The pixel values ​​of each pixel in the facial features are redistributed based on the correspondence graph, i.e., rp i,j =p ri,j ; This yields the rematch feature, denoted as RP. m×n =(rp i,j ) m×n .

8. A face shadow removal method based on deep learning according to claim 6, characterized in that, The illumination factor consists of two feature maps whose size and number of channels are completely identical to the input rematch features.

9. A face shadow removal method based on deep learning according to claim 6, characterized in that, The process of obtaining the first corrected facial features through illumination correction is specifically as follows: Normalize the feature map channel by channel to obtain the normalized feature map, represented as: Where h is the normalized feature map, J is the number of pixels in the feature map, and M is the number of channels in the feature map. m,j Let μ be the pixel value corresponding to the j-th pixel in the m-th channel of the feature map. m σ is the average pixel value of the m-th channel in the feature map; m Let be the standard deviation of the pixel value of the m-th channel in the feature map; Perform an affine transformation on the normalized feature map and apply illumination correction to obtain the first corrected facial features, represented as: h′1=γh+β Where h′1 is the first corrected facial feature, and γ and β are both illumination factors.

10. A face shadow removal method based on deep learning according to claim 4, characterized in that, When the self-sample illumination equalization network is trained end-to-end, the weighted sum of the L1 loss function and the perceptual loss function is used as the final loss function, expressed as: L total =L1+L P ; Where L1 is the L1 loss function, expressed as: In the formula, To obtain the shadow-free face image through training, x f The image is the real, shadowless face image corresponding to the shadowed face image; ||·|1 represents the L1 norm; L P The perceptual loss function is expressed as: In the formula, Φ i (·) represents the feature map output from the i-th layer of the pre-trained feature extractor inputting the image.

Images