Face shadow elimination method, device and equipment based on background fusion

Abstract

The application provides a face shadow elimination method, device and equipment based on background fusion, which comprises the following steps: eliminating face shadows in an initial image to obtain a preliminary elimination image; dividing the preliminary elimination image into a plurality of preliminary image blocks with a preset size; encoding each preliminary image block to obtain a corresponding feature vector; for each first feature vector, combining all second feature vectors to obtain a first reference vector, and combining all third feature vectors to obtain a second reference vector; for each first feature vector, respectively combining the first reference vector and the second reference vector to decode to obtain a pair of intermediate image blocks; fusing each pair of intermediate image blocks to obtain a corresponding final image block; replacing each preliminary image block meeting a first condition in the preliminary elimination image with the final image block to obtain a final elimination image. Through the application, the face itself is in harmony with the background after the face shadow is eliminated.

Description

Technical Field

[0001] This application relates to the field of facial shadow removal technology, specifically to a method, apparatus, and device for facial shadow removal based on background fusion. Background Technology

[0002] As part of face editing, face shadow removal aims to eliminate complex shadows on the facial surface and restore high-quality facial images. Especially with the development of the live streaming industry and facial recognition systems, facial images are inevitably affected by shadows under complex ambient lighting conditions, reducing the aesthetic quality of the face and significantly impacting the performance of some computer vision tasks, such as face recognition and detection. Clearly, effectively removing facial shadows is an important but challenging task. However, current face shadow removal methods typically rely solely on information from the facial region, neglecting to consider background information, resulting in the face blending poorly with the background after shadow removal. Summary of the Invention

[0003] This application provides a method, apparatus, and device for eliminating facial shadows based on background fusion, which can solve the technical problem in the prior art where the face itself and the background are not in harmony after the facial shadows are eliminated.

[0004] In a first aspect, embodiments of this application provide a method for eliminating facial shadows based on background fusion, the method comprising:

[0005] The facial shadows in the initial image are removed to obtain a preliminary removed image;

[0006] The initially eliminated image is divided into multiple initial image blocks of preset sizes;

[0007] Each preliminary image block is encoded to obtain a corresponding feature vector. When the preliminary image block meets the first condition, the second condition, and the third condition, its feature vector is denoted as the first feature vector, the second feature vector, and the third feature vector, respectively. The first condition is that there are face shadow removal pixels in the image block. The second condition is that the image block is composed of background pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is greater than a preset proportion. The third condition is that the image block is composed of face shadow removal pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is less than or equal to a preset proportion.

[0008] For each first feature vector, a first reference vector is obtained by combining all second feature vectors with the first feature dictionary as the target domain and predicting. A second reference vector is obtained by combining all third feature vectors with the second feature dictionary as the target domain. The first reference vector is the average of the fourth feature vectors in the level with the highest prediction probability in the first feature dictionary, and the second reference vector is the average of the fifth feature vectors in the level with the highest prediction probability in the second feature dictionary. The first feature dictionary includes multiple fourth feature vectors divided into N levels, and the second feature dictionary includes multiple fifth feature vectors divided into N levels. The division is based on the similarity ranking of each feature vector with the global average. The fourth feature vector is obtained by encoding sample image blocks that meet the second condition and the fourth feature vector is obtained by encoding sample image blocks that meet the third condition. The sample image blocks are of a preset size and are taken from multiple sample images. The face areas of the sample images have shadows.

[0009] For each first feature vector, a pair of intermediate image blocks are obtained by decoding it in combination with the first reference vector and the second reference vector.

[0010] Each pair of intermediate image blocks is fused to obtain the corresponding final graphic block;

[0011] Replace each preliminary image block that meets the first condition in the preliminary eliminated image with a final image block to obtain the final eliminated image.

[0012] Furthermore, in one embodiment, the step of removing facial shadows in the initial image to obtain a preliminarily removed image includes:

[0013] The initial image is encoded to obtain the sixth feature vector;

[0014] The sixth feature vector is processed by a multi-level dilation layer to obtain the seventh feature vector. The multi-level dilation layer consists of multiple dilation blocks with different dilation rates to enhance the receptive field.

[0015] Decoding the seventh feature vector yields a preliminary image with the image removed.

[0016] Furthermore, in one embodiment, the multi-level expansion layer has three expansion blocks with expansion rates of 2, 4, and 8, respectively.

[0017] Furthermore, in one embodiment, the step of removing facial shadows in the initial image to obtain a preliminarily removed image includes:

[0018] Acquire multiple sample images;

[0019] Each sample image is divided into multiple sample image blocks of a preset size. Sample image blocks that meet the second condition are encoded to obtain the fourth feature vector, and sample image blocks that meet the third condition are encoded to obtain the fifth feature vector.

[0020] Global average pooling is performed on all fourth feature vectors to obtain the first global vector. The fourth feature vectors are then sorted according to their similarity to the first global vector, and divided into N similarity levels to form the first feature dictionary.

[0021] Global average pooling is performed on all fifth feature vectors to obtain the second global vector. The fifth feature vectors are then sorted according to their similarity to the second global vector, and divided into N similarity levels to form the second feature dictionary.

[0022] Furthermore, in one embodiment, the face shadow removal method further includes:

[0023] Using the training set and loss function, train the initial elimination network, the perceptual recovery network, and the global fusion network;

[0024] The preliminary removal network is used to remove facial shadows in the initial image to obtain a preliminary removed image;

[0025] The perceptual restoration network includes an encoder, an indexing module, and a decoder. The encoder encodes each preliminary image patch to obtain a corresponding feature vector. The indexing module predicts a first reference vector by combining all second feature vectors with the first feature dictionary as the target domain for each first feature vector, and predicts a second reference vector by combining all third feature vectors with the second feature dictionary as the target domain. The decoder decodes each first feature vector by combining the first reference vector and the second reference vector to obtain a pair of intermediate image patches.

[0026] The global fusion network is used to fuse each pair of intermediate image blocks to obtain the corresponding final image block;

[0027] The training set includes multiple target image pairs. Each target image pair includes a reference image and a shadow image. The face region of the reference image has no shadow, while the face region of the shadow image has a shadow. The pixel change region of the reference image relative to the reference image completely overlaps with the face shadow region in the shadow image.

[0028] During training, the shadow image is used as the initial image, and the reference image is used as the real result.

[0029] Furthermore, in one embodiment, the step of training the initial elimination network, the perceptual recovery network, and the global fusion network using the training set and the loss function includes:

[0030] The initial elimination network is trained using the training set and the first loss function;

[0031] The index module in the perceptual recovery network is trained using the training set, the second loss function, and the pre-trained preliminary elimination network.

[0032] The encoder and decoder in the perceptual recovery network are trained using the training set, the third loss function, the pre-trained initial elimination network, and the index module in the perceptual recovery network.

[0033] The global fusion network is trained using the training set, the fourth loss function, the pre-trained initial elimination network, and the perceptual recovery network.

[0034] Furthermore, in one embodiment, the loss term of the first loss function includes at least one of the following: SGAN loss of the initial image relative to the reference image, pixel loss, perceptual loss, and smoothness loss;

[0035] The formula for the second loss function is:

[0036]

[0037] Where n represents the number of preliminary image patches in the initially eliminated image that meet the first condition. This represents the first or second reference vector corresponding to the i-th preliminary image patch that meets the first condition. The feature vector representing an image patch at the same location in the reference image;

[0038] The loss term of the third loss function includes at least one of the following: SGAN loss, pixel loss, perceptual loss, and smoothness loss, relative to the intermediate image patch at the same position in the reference image.

[0039] The loss term of the fourth loss function includes at least one of the following: SGAN loss of the final eliminated image relative to the reference image, pixel loss, perceptual loss, and smoothness loss.

[0040] Furthermore, in one embodiment, the step of training the initial elimination network, the perceptual recovery network, and the global fusion network using the training set and the loss function further includes:

[0041] The target video is captured by a camera, in which the camera position is fixed and the obstruction moves between the light source and the face;

[0042] Extract multiple candidate shadow images and at least one candidate reference image from the target video, wherein the face regions of the candidate shadow images are shadowed, and the face regions of the candidate reference images are not shadowed;

[0043] For each candidate shadow image, determine its pixel variation region relative to each candidate reference image. If the pixel variation region does not contain facial edge texture and does not overlap with the background region, then the corresponding candidate shadow image and candidate reference image are determined as the target image pair.

[0044] Secondly, this application embodiment also provides a face shadow removal device based on background fusion. The face shadow removal device includes a preliminary removal network, a segmentation module, a perceptual restoration network, a global fusion network, and a replacement module. The perceptual restoration network includes an encoder, an indexing module, and a decoder.

[0045] The preliminary removal network is used to remove facial shadows from the initial image, resulting in a preliminary removed image;

[0046] The segmentation module is used to divide the initially eliminated image into multiple initial image blocks of preset sizes;

[0047] The encoder of the perceptual restoration network is used to encode each preliminary image block to obtain the corresponding feature vector. When the preliminary image block meets the first condition, the second condition, and the third condition, its feature vector is denoted as the first feature vector, the second feature vector, and the third feature vector, respectively. The first condition is that there are face shadow removal pixels in the image block. The second condition is that the image block is composed of background pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is greater than a preset proportion. The third condition is that the image block is composed of face shadow removal pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is less than or equal to a preset proportion.

[0048] The indexing module of the perceptual recovery network is used to predict a first reference vector for each first feature vector by combining all second feature vectors with the first feature dictionary as the target domain, and to predict a second reference vector by combining all third feature vectors with the second feature dictionary as the target domain. The first reference vector is the average of the fourth feature vectors in the level with the highest prediction probability in the first feature dictionary, and the second reference vector is the average of the fifth feature vectors in the level with the highest prediction probability in the second feature dictionary. The first feature dictionary includes multiple fourth feature vectors divided into N levels, and the second feature dictionary includes multiple fifth feature vectors divided into N levels. The division is based on the similarity ranking of each feature vector with the global average. The fourth feature vector is encoded by sample image blocks that meet the second condition and the fourth feature vector is encoded by sample image blocks that meet the third condition. The sample image blocks are of a preset size and are taken from multiple sample images. The face region of the sample images has shadows.

[0049] The decoder of the perceptual restoration network is used to decode each first feature vector by combining the first reference vector and the second reference vector to obtain a pair of intermediate image patches.

[0050] The global fusion network is used to fuse each pair of intermediate image blocks to obtain the corresponding final image block;

[0051] The replacement module is used to replace each preliminary image block that meets the first condition in the preliminary eliminated image with a final image block to obtain the final eliminated image.

[0052] Thirdly, this application also provides a face shadow removal device, which includes a processor, a memory, and a face shadow removal program stored in the memory and executable by the processor, wherein when the face shadow removal program is executed by the processor, it implements the steps of the face shadow removal method described above.

[0053] In this application, facial shadows in the initial image are removed to obtain a preliminary removed image. The preliminary removed image is divided into multiple preliminary image blocks of preset sizes. Each preliminary image block is encoded to obtain a corresponding feature vector. For each first feature vector, a first reference vector is predicted by combining all second feature vectors with a first feature dictionary as the target domain, and a second reference vector is predicted by combining all third feature vectors with a second feature dictionary as the target domain. For each first feature vector, a pair of intermediate image blocks is decoded by combining the first reference vector and the second reference vector respectively. Each pair of intermediate image blocks is fused to obtain a corresponding final image block. Each preliminary image block in the preliminary removed image that meets the first condition is replaced with a final image block to obtain the final removed image. Through this application, the preliminary removed image is repaired by utilizing information from the shadowless area of ​​the face and the background area. Compared with the preliminary removed image, the face in the final removed image blends more harmoniously with the background. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a face shadow removal method in one embodiment of this application;

[0055] Figure 2 This is a schematic diagram illustrating the principle of a face shadow removal method in one embodiment of this application;

[0056] Figure 3 This is a schematic diagram of the functional modules of a face shadow removal device in one embodiment of this application;

[0057] Figure 4 This is a schematic diagram of the hardware structure of the face shadow removal device involved in the embodiments of this application. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0060] In a first aspect, embodiments of this application provide a method for eliminating facial shadows based on background fusion.

[0061] Figure 1 This illustration shows a flowchart of a face shadow removal method according to an embodiment of this application. Figure 2 This illustration shows a schematic diagram of the principle of a face shadow removal method according to an embodiment of this application.

[0062] Reference Figure 1 and Figure 2 In one embodiment, the face shadow removal method includes the following steps:

[0063] S11. Remove the face shadows in the initial image to obtain a preliminary removed image.

[0064] In this embodiment, the initial image is the image that needs to undergo face shadow removal. The initial image consists of a face shadow area, a face shadow-free area, and a background area. After face shadow removal, the face shadow area becomes the face shadow removal area, while the face shadow-free area and the background area remain unchanged. That is, the initially removed image consists of the face shadow removal area, the face shadow-free area, and the background area. In the following text, pixels in the XX area will be simply referred to as XX pixels; for example, pixels in the background area will be simply referred to as background pixels.

[0065] It should be noted that the face shadow removal algorithm in step S11 can refer to existing technologies or be further optimized based on existing technologies.

[0066] S12. Divide the initially eliminated image into multiple initial image blocks of preset size.

[0067] In this embodiment, the image block is defined by type according to the type of pixels contained in the image block. An image block that meets the first condition is defined as a face shadow removal image block. The first condition is that the image block contains face shadow removal pixels. An image block that meets the second condition is defined as a background image block. The second condition is that the image block is composed of background pixels, or the image block is composed of background pixels and face shadow-free pixels, and the proportion of background pixels is greater than a preset ratio. An image block that meets the third condition is defined as a face shadow-free image block. The third condition is that the image block is composed of face shadow-free pixels, or the image block is composed of background pixels and face shadow-free pixels, and the proportion of background pixels is less than or equal to a preset ratio.

[0068] S13. Encode each preliminary image block to obtain the corresponding feature vector.

[0069] In this embodiment, the feature vector of the preliminary image block that meets the first condition is denoted as the first feature vector, the feature vector of the preliminary image block that meets the second condition is denoted as the second feature vector, and the feature vector of the preliminary image block that meets the third condition is denoted as the third feature vector. As can be seen from the condition description, the first feature vector mainly provides high-dimensional information about the face shadow removal region in the preliminary image block, the second feature vector mainly provides high-dimensional information about the background region in the preliminary image block, and the third feature vector mainly provides high-dimensional information about the face shadow-free region in the preliminary image block.

[0070] S14. For each first feature vector, combine all second feature vectors and use the first feature dictionary as the target domain to predict to obtain a first reference vector, and combine all third feature vectors and use the second feature dictionary as the target domain to predict to obtain a second reference vector.

[0071] In this embodiment, the first feature dictionary includes multiple fourth feature vectors divided into N levels, and the second feature dictionary includes multiple fifth feature vectors divided into N levels. The division is based on the similarity ranking of each feature vector with the global average. The fourth feature vectors are obtained by encoding sample image blocks that meet the second condition and the third condition. The sample image blocks are of a preset size and are taken from multiple sample images. The face regions of the sample images have shadows. That is, the sample images consist of face shadow regions, face unshadow regions, and background regions. As can be seen from the condition description, the fourth feature vector mainly provides high-dimensional information of the background region in the sample image block, and the fifth feature vector mainly provides high-dimensional information of the face unshadow regions in the sample image block.

[0072] By combining the high-dimensional information of a single first feature vector and all second feature vectors, and using the first feature dictionary as the target domain for prediction, prediction probabilities for N levels in the first feature dictionary are obtained. The first reference vector is the average of all fourth feature vectors corresponding to the level with the highest prediction probability. In this way, the first reference vector can predict the required degree of restoration of the face shadow removal area in the corresponding preliminary image patch based on the high-dimensional information of the background region.

[0073] By combining the high-dimensional information of a single first feature vector and all third feature vectors, and using the second feature dictionary as the target domain for prediction, prediction probabilities for N levels in the second feature dictionary are obtained. The second reference vector is the average of all fifth feature vectors corresponding to the level with the highest prediction probability. In this way, the first reference vector can predict the required degree of restoration of the face shadow removal area in the corresponding preliminary image patch based on the high-dimensional information of the non-shadow area of ​​the face.

[0074] S15. For each first feature vector, decode it by combining the first reference vector and the second reference vector to obtain a pair of intermediate image blocks.

[0075] In this embodiment, the first feature vector and the corresponding first reference vector are decoded together to obtain the first intermediate image block, and the first feature vector and the corresponding second reference vector are decoded together to obtain the second intermediate image block. The face shadow removal areas in the first intermediate image block and the second intermediate image block are repaired to different degrees.

[0076] S16. Fuse each pair of intermediate image blocks to obtain the corresponding final graphic block.

[0077] In this embodiment, by combining the repair results of the first intermediate image block and the second intermediate image block on the face shadow removal area, a better final image block is obtained.

[0078] S17. Replace each preliminary image block that meets the first condition in the preliminary eliminated image with a final image block to obtain the final eliminated image.

[0079] Therefore, through this embodiment, by using information from the shadowless area of ​​the face and the background area, the image that was initially removed is repaired. Compared with the image that was initially removed, the face in the final removed image blends more harmoniously with the background.

[0080] Furthermore, in one embodiment, step S11 specifically includes:

[0081] The initial image is encoded to obtain the sixth feature vector;

[0082] The sixth feature vector is processed by a multi-level dilation layer to obtain the seventh feature vector. The multi-level dilation layer consists of multiple dilation blocks with different dilation rates to enhance the receptive field.

[0083] Decoding the seventh feature vector yields a preliminary image with the image removed.

[0084] In this embodiment, an encoder-decoder network is used as the base network, and multi-level dilatation layers are used to improve the receptive field, thereby improving feature extraction and long-range feature collection.

[0085] For example, the multi-level expansion layer has three expansion blocks with expansion rates of 2, 4 and 8 respectively.

[0086] Furthermore, in one embodiment, the face shadow removal method further includes:

[0087] Acquire multiple sample images;

[0088] Each sample image is divided into multiple sample image blocks of a preset size. Sample image blocks that meet the second condition are encoded to obtain the fourth feature vector, and sample image blocks that meet the third condition are encoded to obtain the fifth feature vector.

[0089] Global average pooling is performed on all fourth feature vectors to obtain the first global vector. The fourth feature vectors are then sorted according to their similarity to the first global vector, and divided into N similarity levels to form the first feature dictionary.

[0090] Global average pooling is performed on all fifth feature vectors to obtain the second global vector. The fifth feature vectors are then sorted according to their similarity to the second global vector, and divided into N similarity levels to form the second feature dictionary.

[0091] In this embodiment, a first feature dictionary and a second feature dictionary are constructed in the early stage by analyzing a large number of sample images. The constructed feature dictionary can be used to process different face shadow removal tasks.

[0092] For example, N is 512.

[0093] Furthermore, in one embodiment, the face shadow removal method further includes:

[0094] Using the training set and loss function, train the initial elimination network, the perceptual recovery network, and the global fusion network;

[0095] The preliminary elimination network is used to perform step S11, the perceptual recovery network includes an encoder, an index module and a decoder, the encoder is used to perform step S13, the index module is used to perform step S14, the decoder is used to perform step S15, and the global fusion network is used to perform step S16.

[0096] The training set includes multiple target image pairs. Each target image pair includes a reference image and a shadow image. The face region of the reference image has no shadow, while the face region of the shadow image has a shadow. The pixel change region of the reference image relative to the reference image completely overlaps with the face shadow region in the shadow image.

[0097] During training, the shadow image is used as the initial image, and the reference image is used as the real result.

[0098] In this embodiment, steps S12 and S17 are simple image segmentation and stitching operations that do not require training. Steps S11, S13, S14, S15 and S16 have higher requirements, and the corresponding networks need to be trained extensively to ensure that they achieve the desired effect when used in practice.

[0099] Specifically, the pixel variation region is extracted using the following formula:

[0100]

[0101] Where R, G, and B represent the absolute values ​​of the pixel values ​​in the R, G, and B channels of pixels at the same position in the candidate shadow image and the candidate reference image, a represents the preset threshold, M=1 indicates that the pixel has changed, and M=0 indicates that the pixel has not changed.

[0102] For example, a is 0.05.

[0103] Furthermore, in one embodiment, the step of training the initial elimination network, the perceptual recovery network, and the global fusion network using the training set and the loss function includes:

[0104] The initial elimination network is trained using the training set and the first loss function;

[0105] The index module in the perceptual recovery network is trained using the training set, the second loss function, and the pre-trained preliminary elimination network.

[0106] The encoder and decoder in the perceptual recovery network are trained using the training set, the third loss function, the pre-trained initial elimination network, and the index module in the perceptual recovery network.

[0107] The global fusion network is trained using the training set, the fourth loss function, the pre-trained initial elimination network, and the perceptual recovery network.

[0108] In this embodiment, the initial elimination network, the index module in the perceptual recovery network, the encoder and decoder, and the global fusion network are trained in stages in sequence, which helps to shorten the training time and improve the training effect.

[0109] Furthermore, in one embodiment, the loss term of the first loss function includes at least one of the following: SGAN loss of the initial image relative to the reference image, pixel loss, perceptual loss, and smoothness loss;

[0110] The formula for the second loss function is:

[0111]

[0112] Where n represents the number of preliminary image patches in the initially eliminated image that meet the first condition. This represents the first or second reference vector corresponding to the i-th preliminary image patch that meets the first condition. The feature vector representing an image patch at the same location in the reference image;

[0113] The loss term of the third loss function includes at least one of the following: SGAN loss, pixel loss, perceptual loss, and smoothness loss, relative to the intermediate image patch at the same position in the reference image.

[0114] The loss term of the fourth loss function includes at least one of the following: SGAN loss of the final eliminated image relative to the reference image, pixel loss, perceptual loss, and smoothness loss.

[0115] Specifically, SGAN (Soft-GAN) loss typically uses a binary cross-entropy loss function to make the results more visually realistic. Pixel loss suppresses color and texture distortion by comparing the differences between the generated and real pixel values. Perceptual loss generates more realistic and detailed image results by comparing the differences between the generated and real results in a high-dimensional representation space. Smoothness loss is typically implemented by penalizing the gradient of the generated result to give it a smooth appearance.

[0116] Furthermore, prior to the step of training the initial elimination network, the perceptual recovery network, and the global fusion network using the training set and loss function, the following is also included:

[0117] The target video is captured by a camera, in which the camera position is fixed and the obstruction moves between the light source and the face;

[0118] Extract multiple candidate shadow images and at least one candidate reference image from the target video, wherein the face regions of the candidate shadow images are shadowed, and the face regions of the candidate reference images are not shadowed;

[0119] For each candidate shadow image, determine its pixel variation region relative to each candidate reference image. If the pixel variation region does not contain facial edge texture and does not overlap with the background region, then the corresponding candidate shadow image and candidate reference image are determined as the target image pair.

[0120] In this embodiment, by moving the occluder between the light source and the face, candidate shadow images and candidate reference images can be obtained. The shadow conditions in the candidate shadow images are different. When the pixel change area does not contain facial edge texture and does not overlap with the background area, it indicates that the difference between the two images is mainly concentrated in the face shadow area, and the pixel change area completely overlaps with the face shadow area. Thus, a method for obtaining target image pairs is provided. By performing the above steps on different backgrounds and different faces, a training set with a large number of target image pairs can be formed to solve the problem of scarce training data.

[0121] Optionally, the sample images used to construct the feature dictionary may be the shaded images described above.

[0122] Secondly, embodiments of this application also provide a face shadow removal device based on background fusion.

[0123] Figure 3 A schematic diagram of the functional modules of a face shadow removal device in one embodiment of this application is shown.

[0124] Reference Figure 3 In one embodiment, the face shadow removal device includes a preliminary removal network, a segmentation module, a perceptual restoration network, a global fusion network, and a replacement module. The perceptual restoration network includes an encoder, an indexing module, and a decoder.

[0125] The preliminary removal network is used to remove facial shadows from the initial image, resulting in a preliminary removed image;

[0126] The segmentation module is used to divide the initially eliminated image into multiple initial image blocks of preset sizes;

[0127] The encoder of the perceptual restoration network is used to encode each preliminary image block to obtain the corresponding feature vector. When the preliminary image block meets the first condition, the second condition, and the third condition, its feature vector is denoted as the first feature vector, the second feature vector, and the third feature vector, respectively. The first condition is that there are face shadow removal pixels in the image block. The second condition is that the image block is composed of background pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is greater than a preset proportion. The third condition is that the image block is composed of face shadow removal pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is less than or equal to a preset proportion.

[0128] The indexing module of the perceptual recovery network is used to predict a first reference vector for each first feature vector by combining all second feature vectors with the first feature dictionary as the target domain, and to predict a second reference vector by combining all third feature vectors with the second feature dictionary as the target domain. The first reference vector is the average of the fourth feature vectors in the level with the highest prediction probability in the first feature dictionary, and the second reference vector is the average of the fifth feature vectors in the level with the highest prediction probability in the second feature dictionary. The first feature dictionary includes multiple fourth feature vectors divided into N levels, and the second feature dictionary includes multiple fifth feature vectors divided into N levels. The division is based on the similarity ranking of each feature vector with the global average. The fourth feature vector is encoded by sample image blocks that meet the second condition and the fourth feature vector is encoded by sample image blocks that meet the third condition. The sample image blocks are of a preset size and are taken from multiple sample images. The face region of the sample images has shadows.

[0129] The decoder of the perceptual restoration network is used to decode each first feature vector by combining the first reference vector and the second reference vector to obtain a pair of intermediate image patches.

[0130] The global fusion network is used to fuse each pair of intermediate image blocks to obtain the corresponding final image block;

[0131] The replacement module is used to replace each preliminary image block that meets the first condition in the preliminary eliminated image with a final image block to obtain the final eliminated image.

[0132] Furthermore, in one embodiment, the initial elimination network is used for:

[0133] The initial image is encoded to obtain the sixth feature vector;

[0134] The sixth feature vector is processed by a multi-level dilation layer to obtain the seventh feature vector. The multi-level dilation layer consists of multiple dilation blocks with different dilation rates to enhance the receptive field.

[0135] Decoding the seventh feature vector yields a preliminary image with the image removed.

[0136] Furthermore, in one embodiment, the multi-level expansion layer has three expansion blocks with expansion rates of 2, 4, and 8, respectively.

[0137] Furthermore, in one embodiment, the face shadow removal device further includes a dictionary construction module for:

[0138] Acquire multiple sample images;

[0139] Each sample image is divided into multiple sample image blocks of a preset size. Sample image blocks that meet the second condition are encoded to obtain the fourth feature vector, and sample image blocks that meet the third condition are encoded to obtain the fifth feature vector.

[0140] Global average pooling is performed on all fourth feature vectors to obtain the first global vector. The fourth feature vectors are then sorted according to their similarity to the first global vector, and divided into N similarity levels to form the first feature dictionary.

[0141] Global average pooling is performed on all fifth feature vectors to obtain the second global vector. The fifth feature vectors are then sorted according to their similarity to the second global vector, and divided into N similarity levels to form the second feature dictionary.

[0142] Furthermore, in one embodiment, the face shadow removal device further includes a network training module for:

[0143] Using the training set and loss function, train the initial elimination network, the perceptual recovery network, and the global fusion network;

[0144] The preliminary removal network is used to remove facial shadows in the initial image to obtain a preliminary removed image;

[0145] The perceptual restoration network includes an encoder, an indexing module, and a decoder. The encoder encodes each preliminary image patch to obtain a corresponding feature vector. The indexing module predicts a first reference vector by combining all second feature vectors with the first feature dictionary as the target domain for each first feature vector, and predicts a second reference vector by combining all third feature vectors with the second feature dictionary as the target domain. The decoder decodes each first feature vector by combining the first reference vector and the second reference vector to obtain a pair of intermediate image patches.

[0146] The global fusion network is used to fuse each pair of intermediate image blocks to obtain the corresponding final image block;

[0147] The training set includes multiple target image pairs. Each target image pair includes a reference image and a shadow image. The face region of the reference image has no shadow, while the face region of the shadow image has a shadow. The pixel change region of the reference image relative to the reference image completely overlaps with the face shadow region in the shadow image.

[0148] During training, the shadow image is used as the initial image, and the reference image is used as the real result.

[0149] Furthermore, in one embodiment, the network training module is used for:

[0150] The initial elimination network is trained using the training set and the first loss function;

[0151] The index module in the perceptual recovery network is trained using the training set, the second loss function, and the pre-trained preliminary elimination network.

[0152] The encoder and decoder in the perceptual recovery network are trained using the training set, the third loss function, the pre-trained initial elimination network, and the index module in the perceptual recovery network.

[0153] The global fusion network is trained using the training set, the fourth loss function, the pre-trained initial elimination network, and the perceptual recovery network.

[0154] Furthermore, in one embodiment, the loss term of the first loss function includes at least one of the following: SGAN loss of the initial image relative to the reference image, pixel loss, perceptual loss, and smoothness loss;

[0155] The formula for the second loss function is:

[0156]

[0157] Where n represents the number of preliminary image patches in the initially eliminated image that meet the first condition. This represents the first or second reference vector corresponding to the i-th preliminary image patch that meets the first condition. The feature vector representing an image patch at the same location in the reference image;

[0158] The loss term of the third loss function includes at least one of the following: SGAN loss, pixel loss, perceptual loss, and smoothness loss, relative to the intermediate image patch at the same position in the reference image.

[0159] The loss term of the fourth loss function includes at least one of the following: SGAN loss of the final eliminated image relative to the reference image, pixel loss, perceptual loss, and smoothness loss.

[0160] Furthermore, in one embodiment, the face shadow removal device further includes an image pair acquisition module, used for:

[0161] The target video is captured by a camera, in which the camera position is fixed and the obstruction moves between the light source and the face;

[0162] Extract multiple candidate shadow images and at least one candidate reference image from the target video, wherein the face regions of the candidate shadow images are shadowed, and the face regions of the candidate reference images are not shadowed;

[0163] For each candidate shadow image, determine its pixel variation region relative to each candidate reference image. If the pixel variation region does not contain facial edge texture and does not overlap with the background region, then the corresponding candidate shadow image and candidate reference image are determined as the target image pair.

[0164] The functions of each module in the aforementioned face shadow removal device correspond to the steps in the aforementioned face shadow removal method embodiment, and their functions and implementation processes will not be described in detail here.

[0165] Thirdly, embodiments of this application provide a face shadow removal device based on background fusion. The face shadow removal device can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0166] Figure 4 A schematic diagram of the hardware structure of the face shadow removal device involved in the embodiment of this application is shown.

[0167] Reference Figure 4 In this embodiment of the application, the face shadow removal device may include a processor, a memory, a communication interface, and a communication bus.

[0168] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0169] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the face shadow removal device, as well as interfaces used for interconnecting the face shadow removal device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0170] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0171] The processor can be a general-purpose processor, which can call the face shadow removal program stored in memory and execute the face shadow removal method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the face shadow removal program is called can refer to the various embodiments of the face shadow removal method of this applicant, and will not be repeated here.

[0172] Those skilled in the art will understand that Figure 4 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0173] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0174] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0175] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0176] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0177] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0179] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for eliminating facial shadows based on background fusion, characterized in that, The method for eliminating facial shadows includes: The facial shadows in the initial image are removed to obtain a preliminary removed image; The initially eliminated image is divided into multiple initial image blocks of preset sizes; Each preliminary image block is encoded to obtain a corresponding feature vector. When the preliminary image block meets the first condition, the second condition, and the third condition, its feature vector is denoted as the first feature vector, the second feature vector, and the third feature vector, respectively. The first condition is that there are face shadow removal pixels in the image block. The second condition is that the image block is composed of background pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is greater than a preset proportion. The third condition is that the image block is composed of face shadow removal pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is less than or equal to a preset proportion. For each first feature vector, a first reference vector is obtained by combining all second feature vectors with the first feature dictionary as the target domain and predicting. A second reference vector is obtained by combining all third feature vectors with the second feature dictionary as the target domain. The first reference vector is the average of the fourth feature vectors in the level with the highest prediction probability in the first feature dictionary, and the second reference vector is the average of the fifth feature vectors in the level with the highest prediction probability in the second feature dictionary. The first feature dictionary includes multiple fourth feature vectors divided into N levels, and the second feature dictionary includes multiple fifth feature vectors divided into N levels. The division is based on the similarity ranking of each feature vector with the global average. The fourth feature vector is obtained by encoding sample image blocks that meet the second condition and the fourth feature vector is obtained by encoding sample image blocks that meet the third condition. The sample image blocks are of a preset size and are taken from multiple sample images. The face areas of the sample images have shadows. For each first feature vector, a pair of intermediate image blocks are obtained by decoding it in combination with the first reference vector and the second reference vector. Each pair of intermediate image blocks is fused to obtain the corresponding final graphic block; Replace each preliminary image block that meets the first condition in the preliminary eliminated image with a final image block to obtain the final eliminated image.

2. The face shadow removal method as described in claim 1, characterized in that, The step of removing facial shadows from the initial image to obtain a preliminarily eliminated image includes: The initial image is encoded to obtain the sixth feature vector; The sixth feature vector is processed by a multi-level dilation layer to obtain the seventh feature vector. The multi-level dilation layer consists of multiple dilation blocks with different dilation rates to enhance the receptive field. Decoding the seventh feature vector yields a preliminary image with the image removed.

3. The face shadow removal method as described in claim 2, characterized in that, The multi-level expansion layer has three expansion blocks with expansion rates of 2, 4 and 8, respectively.

4. The face shadow removal method as described in claim 1, characterized in that, The process includes the following steps before removing facial shadows from the initial image to obtain a preliminarily removed image: Acquire multiple sample images; Each sample image is divided into multiple sample image blocks of a preset size. Sample image blocks that meet the second condition are encoded to obtain the fourth feature vector, and sample image blocks that meet the third condition are encoded to obtain the fifth feature vector. Global average pooling is performed on all fourth feature vectors to obtain the first global vector. The fourth feature vectors are then sorted according to their similarity to the first global vector, and divided into N similarity levels to form the first feature dictionary. Global average pooling is performed on all fifth feature vectors to obtain the second global vector. The fifth feature vectors are then sorted according to their similarity to the second global vector, and divided into N similarity levels to form the second feature dictionary.

5. The face shadow removal method as described in claim 1, characterized in that, The method for eliminating facial shadows also includes: Using the training set and loss function, train the initial elimination network, the perceptual recovery network, and the global fusion network; The preliminary removal network is used to remove facial shadows in the initial image to obtain a preliminary removed image; The perceptual restoration network includes an encoder, an indexing module, and a decoder. The encoder encodes each preliminary image patch to obtain a corresponding feature vector. The indexing module predicts a first reference vector by combining all second feature vectors with the first feature dictionary as the target domain for each first feature vector, and predicts a second reference vector by combining all third feature vectors with the second feature dictionary as the target domain. The decoder decodes each first feature vector by combining the first reference vector and the second reference vector to obtain a pair of intermediate image patches. The global fusion network is used to fuse each pair of intermediate image blocks to obtain the corresponding final image block; The training set includes multiple target image pairs. Each target image pair includes a reference image and a shadow image. The face region of the reference image has no shadow, while the face region of the shadow image has a shadow. The pixel change region of the reference image relative to the reference image completely overlaps with the face shadow region in the shadow image. During training, the shadow image is used as the initial image, and the reference image is used as the real result.

6. The face shadow removal method as described in claim 5, characterized in that, The steps of training the initial elimination network, the perceptual recovery network, and the global fusion network using the training set and the loss function include: The initial elimination network is trained using the training set and the first loss function; The index module in the perceptual recovery network is trained using the training set, the second loss function, and the pre-trained preliminary elimination network. The encoder and decoder in the perceptual recovery network are trained using the training set, the third loss function, the pre-trained initial elimination network, and the index module in the perceptual recovery network. The global fusion network is trained using the training set, the fourth loss function, the pre-trained initial elimination network, and the perceptual recovery network.

7. The face shadow removal method as described in claim 6, characterized in that, The loss term of the first loss function includes at least one of the following: SGAN loss of the initial image relative to the reference image, pixel loss, perceptual loss, and smoothness loss; The formula for the second loss function is: Where n represents the number of preliminary image patches in the initially eliminated image that meet the first condition. This represents the first or second reference vector corresponding to the i-th preliminary image patch that meets the first condition. The feature vector representing an image patch at the same location in the reference image; The loss term of the third loss function includes at least one of the following: SGAN loss, pixel loss, perceptual loss, and smoothness loss, relative to the intermediate image patch at the same position in the reference image. The loss term of the fourth loss function includes at least one of the following: SGAN loss of the final eliminated image relative to the reference image, pixel loss, perceptual loss, and smoothness loss.

8. The face shadow removal method as described in claim 5, characterized in that, Prior to the step of training the initial elimination network, the perceptual recovery network, and the global fusion network using the training set and loss function, the following is also included: The target video is captured by a camera, in which the camera position is fixed and the obstruction moves between the light source and the face; Extract multiple candidate shadow images and at least one candidate reference image from the target video, wherein the face regions of the candidate shadow images are shadowed, and the face regions of the candidate reference images are not shadowed; For each candidate shadow image, determine its pixel variation region relative to each candidate reference image. If the pixel variation region does not contain facial edge texture and does not overlap with the background region, then the corresponding candidate shadow image and candidate reference image are determined as the target image pair.

9. A face shadow removal device based on background fusion, characterized in that, The face shadow removal device includes a preliminary removal network, a segmentation module, a perceptual restoration network, a global fusion network, and a replacement module. The perceptual restoration network includes an encoder, an indexing module, and a decoder. The preliminary removal network is used to remove facial shadows from the initial image, resulting in a preliminary removed image; The segmentation module is used to divide the initially eliminated image into multiple initial image blocks of preset sizes; The encoder of the perceptual restoration network is used to encode each preliminary image block to obtain the corresponding feature vector. When the preliminary image block meets the first condition, the second condition, and the third condition, its feature vector is denoted as the first feature vector, the second feature vector, and the third feature vector, respectively. The first condition is that there are face shadow removal pixels in the image block. The second condition is that the image block is composed of background pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is greater than a preset proportion. The third condition is that the image block is composed of face shadow removal pixels, or the image block is composed of background pixels and face shadow removal pixels, and the proportion of background pixels is less than or equal to a preset proportion. The indexing module of the perceptual recovery network is used to predict a first reference vector for each first feature vector by combining all second feature vectors with the first feature dictionary as the target domain, and to predict a second reference vector by combining all third feature vectors with the second feature dictionary as the target domain. The first reference vector is the average of the fourth feature vectors in the level with the highest prediction probability in the first feature dictionary, and the second reference vector is the average of the fifth feature vectors in the level with the highest prediction probability in the second feature dictionary. The first feature dictionary includes multiple fourth feature vectors divided into N levels, and the second feature dictionary includes multiple fifth feature vectors divided into N levels. The division is based on the similarity ranking of each feature vector with the global average. The fourth feature vector is encoded by sample image blocks that meet the second condition and the fourth feature vector is encoded by sample image blocks that meet the third condition. The sample image blocks are of a preset size and are taken from multiple sample images. The face region of the sample images has shadows. The decoder of the perceptual restoration network is used to decode each first feature vector by combining the first reference vector and the second reference vector to obtain a pair of intermediate image patches. The global fusion network is used to fuse each pair of intermediate image blocks to obtain the corresponding final image block; The replacement module is used to replace each preliminary image block that meets the first condition in the preliminary eliminated image with a final image block to obtain the final eliminated image.

10. A face shadow removal device based on background fusion, characterized in that, The face shadow removal device includes a processor, a memory, and a face shadow removal program stored in the memory and executable by the processor, wherein when the face shadow removal program is executed by the processor, it implements the steps of the face shadow removal method as described in any one of claims 1 to 8.

Images