Method and device for eliminating face shadow based on adaptive face symmetry

Abstract

The present application provides a method and device for removing face shadows based on adaptive face symmetry. The method comprises: flipping the face region in an initial image to obtain a flipped image; generating an initial symmetry mask based on the initial image and the flipped image, wherein the initial symmetry mask represents the region in the flipped image where the RGB pixel values ​​are higher than those of the initial image; inputting the flipped image and the initial symmetry mask into a trained symmetry feature extraction network for encoding to obtain shadow-free features of the flipped image and features of the initial symmetry mask; inputting the pure noise image, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetry mask into a trained dynamic symmetry perception network for stepwise reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. Through the present application, information about the symmetric regions of the face can be more effectively utilized, and face self-shadows can be retained while removing face shadows.

Description

Technical Field

[0001] The present application relates to the technical field of face shadow removal, and in particular to a face shadow removal method and device based on adaptive face symmetry. Background Art

[0002] With technological advancements, facial recognition and beautification technologies are becoming increasingly mature and are gradually being applied to fields such as security and live streaming. However, user environments are often complex and ever-changing, and facial shadows are inevitable, affecting both facial recognition accuracy and beautification results. Therefore, effectively removing facial shadows is a crucial task.

[0003] Related technologies use information from symmetrical areas of the face as a reference for shadow restoration. This is because facial shadows are typically asymmetrical. However, due to the randomness of light source position and facial orientation in real-world scenes, the face itself often experiences lighting changes. Directly leveraging symmetry often provides incorrect information to the network, causing face shadow removal to also remove self-shadows that should be preserved. This results in a loss of the original facial appearance and an unrealistic image after removal. Summary of the Invention

[0004] The present application provides a method and device for removing face shadows based on adaptive face symmetry, which can solve the technical problem in the prior art that images after face shadow removal are unrealistic.

[0005] In a first aspect, an embodiment of the present application provides a method for removing face shadows based on adaptive face symmetry, the method comprising:

[0006] Flip the face area in the initial image to obtain a flipped image;

[0007] generating an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0008] Input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetric mask;

[0009] The pure noise image, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetric mask are input into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1-th step, t∈[1,T]. The noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0010] Furthermore, in one embodiment, the training process of the dynamic symmetry perception network includes:

[0011] Get the initial image and its ground-truth shadow-free image from the training set;

[0012] Flip the face area in the initial image to obtain a flipped image;

[0013] Generate an initial symmetric mask based on the initial image and the flipped image;

[0014] Input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetric mask;

[0015] Randomly determine the step number t from [1, T];

[0016] Input the step number t, the real noise image of the t-th step, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetry mask into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain the single-step prediction result, where the single-step prediction result includes the predicted noise image of the t-1-th step;

[0017] Relevant parameters in the dynamic symmetric perception network are updated according to a first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the true noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the true shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the true noise image at the t-1th step.

[0018] Furthermore, in one embodiment, the single-step prediction result further includes a first prediction mask, the first prediction mask representing a predicted area that belongs to a face shadow in the original image and belongs to a non-face shadow in the flipped image;

[0019] The loss term of the first loss function also includes the loss of the first predicted mask relative to the first true mask, where the first true mask represents the real area belonging to the shadow of the face in the initial image and the real area belonging to the non-shadow of the face in the flipped image.

[0020] Furthermore, in one embodiment, the single-step prediction result further includes a second prediction mask, the second prediction mask representing the predicted area belonging to the face shadow in the initial image;

[0021] The loss term of the first loss function also includes the loss of the second predicted mask relative to the second true mask, where the second true mask represents the true area in the initial image that belongs to the shadow of the face.

[0022] Furthermore, in one embodiment, the step of flipping the face region in the initial image to obtain a flipped image includes:

[0023] Extract geometric information, key point information and surface texture information from the initial image through the facial feature extraction algorithm;

[0024] The surface texture information is flipped and combined with the geometric information and key point information to re-render the face area of ​​the initial image to obtain a flipped image.

[0025] Furthermore, in one embodiment, the training process of the symmetric feature extraction network includes:

[0026] Get the initial image and its ground-truth shadow-free image from the training set;

[0027] The initial image is input into the symmetric feature extraction network for encoding and decoding to obtain the predicted shadow-free image of the initial image;

[0028] Update relevant parameters in the symmetric feature extraction network according to the second loss function, wherein the loss term in the second loss function includes at least one of SGAN loss, pixel loss, perceptual loss and smoothness loss of the initial image relative to the true shadow-free image.

[0029] In a second aspect, an embodiment of the present application further provides another method for removing face shadows based on adaptive face symmetry, the method for removing face shadows based on adaptive face symmetry comprising:

[0030] Flip the face area in the initial image to obtain a flipped image;

[0031] generating an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0032] The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0033] The pure noise image, initial image, initial symmetry mask and shadow-free features of the flipped image are input into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to decode the noise image of the t-1-th step, t∈[1,T]. The noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0034] Furthermore, in one embodiment, the training process of the dynamic symmetry perception network includes:

[0035] Get the initial image and its ground-truth shadow-free image from the training set;

[0036] Flip the face area in the initial image to obtain a flipped image;

[0037] Generate an initial symmetric mask based on the initial image and the flipped image;

[0038] The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0039] Randomly determine the step number t from [1, T];

[0040] Input the step number t, the real noise image of the t-th step, the initial image, the initial symmetry mask, and the shadow-free features of the flipped image into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain the single-step prediction result, where the single-step prediction result includes the predicted noise image of the t-1-th step;

[0041] Relevant parameters in the dynamic symmetric perception network are updated according to a first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the true noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the true shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the true noise image at the t-1th step.

[0042] In a third aspect, an embodiment of the present application provides a face shadow removal device based on adaptive face symmetry, the face shadow removal device based on adaptive face symmetry comprising:

[0043] A face flipping module is used to flip the face area in the initial image to obtain a flipped image;

[0044] a mask generation module, configured to generate an initial symmetric mask based on the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0045] A feature extraction module is used to input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding, thereby obtaining the shadow-free features of the flipped image and the features of the initial symmetric mask;

[0046] The reverse denoising module is used to input the pure noise image, the initial image, the shadow-free features of the flipped image and the features of the initial symmetric mask into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain the shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the tth step to the t-1th step, the initial image and the noise image of the tth step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1th step, t∈[1,T]. The noise image of the Tth step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0047] In a fourth aspect, the embodiments of the present application further provide another face shadow removal device based on adaptive face symmetry, the face shadow removal device based on adaptive face symmetry comprising:

[0048] A face flipping module is used to flip the face area in the initial image to obtain a flipped image;

[0049] a mask generation module, configured to generate an initial symmetric mask based on the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0050] The feature extraction module is used to input the flipped image into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0051] The reverse denoising module is used to input the pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to decode the noise image of the t-1-th step, t∈[1,T]. The noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0052] In this application, a dynamic symmetry perception network is designed based on a diffusion model. A shadow-free image of the initial image is constructed from a pure noise image through stepwise reverse denoising. The dynamic symmetry perception network uses the features of the initial image to determine what shadow-free image to construct. By flipping the shadow-free features of the image and the features of the initial symmetry mask, the network obtains information that is beneficial for removing face shadows. This application makes it possible to more effectively utilize information from the symmetrical regions of the face, while removing face shadows while preserving self-shadows, thereby preserving the original appearance of the face and generating a more realistic shadow-free image. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the flow of a face shadow removal method based on adaptive face symmetry in one embodiment of the present application;

[0054] Figure 2 for Figure 1 Detailed schematic diagram of step S13 and step S14;

[0055] Figure 3 This is a flowchart of a method for removing face shadows based on adaptive face symmetry in another embodiment of the present application;

[0056] Figure 4 for Figure 3 Detailed schematic diagram of step S23 and step S24 in FIG. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0059] In a first aspect, an embodiment of the present application provides a method for removing face shadows based on adaptive face symmetry.

[0060] Figure 1 A flow chart of a method for removing face shadows based on adaptive face symmetry in an embodiment of the present application is shown.

[0061] Reference Figure 1 In one embodiment, a method for removing face shadows based on adaptive face symmetry includes the following steps:

[0062] S11, flipping the face region in the initial image to obtain a flipped image.

[0063] Specifically, in the face area, the pixel points with the same coordinates in the flipped image and the original image correspond to two symmetrically distributed pixel points in the original image. In the non-face area, the pixel points with the same coordinates in the flipped image and the original image correspond to the same pixel point in the original image.

[0064] Furthermore, in one embodiment, step S11 specifically includes:

[0065] Extract geometric information, key point information and surface texture information from the initial image through the facial feature extraction algorithm;

[0066] The surface texture information is flipped and combined with the geometric information and key point information to re-render the face area of ​​the initial image to obtain a flipped image.

[0067] S12. Generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image.

[0068] Specifically, pixel-level calculations are performed on the initial image and the flipped image to determine the pixels in the flipped image whose RGB pixel values ​​are higher than those in the initial image. The positional information of these pixels is then aggregated to form an initial symmetry mask. Since the RGB pixel values ​​in the shadowed area are lower than those in the non-shadowed area, the initial symmetry mask can be used to preliminarily predict the areas that belong to the face shadow in the initial image and the non-shadowed area in the flipped image. This area can provide information that is helpful for shadow removal. Since the actual facial structure is not completely symmetrical and asymmetric self-shadowing of the face occurs due to the influence of the light source, there will be a certain error between the initial symmetry mask and the actual areas that belong to the face shadow in the initial image and the non-shadowed area in the flipped image.

[0069] It should be noted that the “face shadow” and “face self-shadow” described in this application have completely different meanings and need to be distinguished. Face shadow is formed when an external light source is blocked by other objects and projected onto the face. For example, when a person stands in the sun and the buildings, trees, etc. next to him block part of the light, an external shadow will be formed on the face. Face self-shadow is determined by the structure and form of the face itself. The raised parts of the face (such as the nose, brow bone, etc.) and the sunken parts (such as the eye sockets, nasolabial folds, etc.) will naturally form shadows under specific light angles. It is an inherent feature of the face and has nothing to do with external obstructions.

[0070] S13. Input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image and the features of the initial symmetric mask.

[0071] In this embodiment, the symmetric feature extraction network uses an encoder-decoder structure to remove face shadows from the input image to produce a shadow-free image. The face shadow removal algorithm used by the symmetric feature extraction network can refer to existing technologies or be further optimized based on existing technologies. The shadow removal operation performed here is to provide more effective information for shadow removal of the initial image. The initial symmetric mask does not play a role in shadow removal of the flipped image; it only borrows the encoder of the symmetric feature extraction network for feature extraction.

[0072] It should be noted that the symmetric feature extraction network needs to use a decoder to obtain a predicted shadow-free image during the training process, and update the relevant parameters according to the loss of the predicted shadow-free image relative to the real shadow-free image. The trained symmetric feature extraction network does not need to use a decoder in actual application, and directly provides the features extracted by the encoder to subsequent steps.

[0073] S14. Input the pure noise image, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetric mask into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode to obtain the noise image of the t-1-th step, t∈[1,T]. The noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0074] The diffusion model process is bidirectional: the forward process involves gradual noise addition, and the reverse process involves gradual denoising, with a total of T steps. In the forward process, random noise is gradually added to the original data. Over time, the data is gradually overwhelmed by the noise, eventually becoming a pure noise distribution in the extreme case. This process generally follows the concept of a Markov chain, where each state change depends only on the previous state, and the state changes follow a specific probability distribution, which is a function of the latent variable. The reverse process is the opposite of the forward process, training a neural network to gradually remove the added noise from the data to restore the original data. By continuously learning the denoising process, the model acquires the ability to generate new data similar to the original data from the noise. Diffusion model training only focuses on the reverse process; the forward process remains unchanged.

[0075] The Dynamic Symmetry Perception Network improves upon the diffusion model by adding the initial image, the shadow-free features of the flipped image, and the features of the initial symmetry mask as additional inputs to the pure noise image. The initial image is encoded along with the noise image input at each step and then fused with the shadow-free features of the flipped image and the features of the initial symmetry mask for decoding. The encoder of the Dynamic Symmetry Perception Network shares a similar architecture to the encoder of the symmetry feature extraction network, but does not share weights, ensuring consistent feature structure and facilitating fusion.

[0076] For example, the dynamic symmetry perception network and the symmetric feature extraction network both use the UNet architecture, consisting of six downsample layers, one intermediate layer, and six upsample layers. Each downsample layer consists of two convolutional layers and one pooling layer, while the upsample layer consists of two convolutional layers and one interpolation layer.

[0077] Figure 2 Shown Figure 1 Detailed schematic diagram of step S13 and step S14 in FIG.

[0078] Reference Figure 2 In the symmetry feature extraction network, the flipped image and the initial symmetry mask are encoded to obtain the shadow-free features of the flipped image and the features of the initial symmetry mask. In the dynamic symmetry perception network, the initial image and the pure noise image are encoded and fused with the shadow-free features of the flipped image and the features of the initial symmetry mask to form a fusion feature. The fusion feature is decoded to obtain the noisy image of the T-1 step. The initial image and the noisy image of the T-1 step are encoded and fused with the shadow-free features of the flipped image and the features of the initial symmetry mask to form a fusion feature. The fusion feature is decoded to obtain the noisy image of the T-2 step. And so on. After obtaining the noisy image of the first step, the initial image and the noisy image of the first step are encoded and fused with the shadow-free features of the flipped image and the features of the initial symmetry mask to form a fusion feature. The fusion feature is decoded to obtain the shadow-free image of the initial image.

[0079] In this embodiment, a dynamic symmetry perception network is designed based on a diffusion model. A shadow-free image of the initial image is constructed from a pure noise image through stepwise reverse denoising. The dynamic symmetry perception network uses the features of the initial image to determine what shadow-free image to construct. By flipping the shadow-free features of the image and the features of the initial symmetry mask, the network obtains information that is beneficial for removing face shadows. This application makes it possible to more effectively utilize information about the symmetrical regions of the face, while removing face shadows while preserving self-shadows, thereby preserving the original appearance of the face and generating a more realistic shadow-free image.

[0080] Furthermore, in one embodiment, the training process of the dynamic symmetry perception network includes:

[0081] Get the initial image and its ground-truth shadow-free image from the training set;

[0082] Flip the face area in the initial image to obtain a flipped image;

[0083] Generate an initial symmetric mask based on the initial image and the flipped image;

[0084] Input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetric mask;

[0085] Randomly determine the step number t from [1, T];

[0086] Input the step number t, the real noise image of the t-th step, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetry mask into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain the single-step prediction result, where the single-step prediction result includes the predicted noise image of the t-1-th step;

[0087] Relevant parameters in the dynamic symmetric perception network are updated according to a first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the true noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the true shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the true noise image at the t-1th step.

[0088] In this embodiment, the dynamic symmetry perception network uses the initial image in the training set and its true shadow-free image. Forward denoising is performed on the true shadow-free image to obtain the true noise image of any step and the added true noise. The same operations are performed on the initial image during the training process and the application process. The difference is that the training process randomly executes one of the denoising steps each time, and the input noise image is the true noise image. The loss is calculated based on the single-step prediction result and the corresponding true result.

[0089] The single-step prediction result includes at least the predicted noise image at step t-1. To improve training effectiveness, it can also include other content. There are two ways to calculate the loss for the predicted noise image at step t-1: one is to calculate the loss of the predicted noise relative to the true noise, and the other is to calculate the loss of the predicted noise image relative to the true noise image. The two losses are equivalent, but the first method is more efficient.

[0090] For example, the calculation formula for adding noise is as follows:

[0091]

[0092] Among them, x t is the real noise image of step t, x0 is the real shadow-free image, α t =1-β t , β t is the noise level,

[0093] For example, the loss of predicted noise relative to real noise is calculated as follows:

[0094]

[0095] Among them, e t represents the prediction noise at step t, ∈ t represents the true noise at step t.

[0096] Furthermore, in one embodiment, the single-step prediction result further includes a first prediction mask, the first prediction mask representing a predicted area that belongs to a face shadow in the original image and belongs to a non-face shadow in the flipped image;

[0097] The loss term of the first loss function also includes the loss of the first predicted mask relative to the first true mask, where the first true mask represents the real area belonging to the shadow of the face in the initial image and the real area belonging to the non-shadow of the face in the flipped image.

[0098] In this embodiment, a first prediction mask is added to the single-step prediction result. Compared with the initial symmetric mask, the first prediction mask refers to more information and makes more accurate predictions for the areas belonging to the face shadow in the initial image and the non-shadow areas of the face in the flipped image. Since the face shadow area and the non-shadow area of ​​the face in the initial image are known information during the training process, the first true mask can be determined in combination with the flipped image. Increasing the loss of the first prediction mask relative to the first true mask in the first loss function helps to improve the training effect.

[0099] For example, the loss of the first predicted mask relative to the first true mask is calculated as follows:

[0100]

[0101] in, represents the first prediction mask, Represents the first true mask.

[0102] Furthermore, in one embodiment, the single-step prediction result further includes a second prediction mask, the second prediction mask representing the predicted area belonging to the face shadow in the initial image;

[0103] The loss term of the first loss function also includes the loss of the second predicted mask relative to the second true mask, where the second true mask represents the true area in the initial image that belongs to the shadow of the face.

[0104] In this embodiment, the second prediction mask is added to the single-step prediction result, and the loss of the second prediction mask relative to the second true mask is increased in the first loss function, which helps to improve the training effect.

[0105] For example, the loss of the second predicted mask relative to the second true mask is calculated as follows:

[0106]

[0107] in, represents the second prediction mask, Represents the second true mask.

[0108] Furthermore, in one embodiment, the training process of the symmetric feature extraction network includes:

[0109] Get the initial image and its ground-truth shadow-free image from the training set;

[0110] The initial image is input into the symmetric feature extraction network for encoding and decoding to obtain the predicted shadow-free image of the initial image;

[0111] Update relevant parameters in the symmetric feature extraction network according to the second loss function, wherein the loss term in the second loss function includes at least one of SGAN loss, pixel loss, perceptual loss and smoothness loss of the initial image relative to the true shadow-free image.

[0112] Specifically, SGAN (Soft-GAN) loss typically uses a binary cross-entropy loss function to make the results more visually realistic. Pixel loss compares the difference between the pixel values ​​of the generated results and the pixel values ​​of the real results to suppress color and texture distortion. Perceptual loss compares the difference between the generated results and the real results in a high-dimensional representation space to generate more realistic and detailed image results. Smoothness loss is usually implemented by penalizing the gradient of the generated results to make the generated results have a smooth appearance.

[0113] In a second aspect, an embodiment of the present application provides another method for removing facial shadows based on adaptive facial symmetry.

[0114] Figure 3 A schematic flow chart of a method for removing face shadows based on adaptive face symmetry in another embodiment of the present application is shown; Figure 4 Shown Figure 3 Detailed schematic diagram of step S23 and step S24 in FIG.

[0115] Reference Figure 3 and Figure 4 In one embodiment, a method for removing face shadows based on adaptive face symmetry includes the following steps:

[0116] S21, flipping the face region in the initial image to obtain a flipped image;

[0117] S22. Generate an initial symmetric mask based on the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0118] S23, inputting the flipped image into the trained symmetric feature extraction network for encoding to obtain a shadow-free feature of the flipped image;

[0119] S24. Input the pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to be decoded to obtain the noise image of the t-1-th step, t∈[1,T]. The noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0120] The second method differs from the first only in that the initial symmetry mask in the first method is encoded in the symmetry feature extraction network and then fed into the dynamic symmetry perception network, while the initial symmetry mask in the second method is encoded in the dynamic symmetry perception network. Both methods are theoretically feasible and can produce more realistic shadow-free images, but in practice, the first method has been found to produce better overall results.

[0121] Accordingly, the training process of the dynamic symmetry perception network includes:

[0122] Get the initial image and its ground-truth shadow-free image from the training set;

[0123] Flip the face area in the initial image to obtain a flipped image;

[0124] Generate an initial symmetric mask based on the initial image and the flipped image;

[0125] The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0126] Randomly determine the step number t from [1, T];

[0127] Input the step number t, the real noise image of the t-th step, the initial image, the initial symmetry mask, and the shadow-free features of the flipped image into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain the single-step prediction result, where the single-step prediction result includes the predicted noise image of the t-1-th step;

[0128] Relevant parameters in the dynamic symmetric perception network are updated according to a first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the true noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the true shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the true noise image at the t-1th step.

[0129] Other detailed embodiments of the second method are consistent with the first method and will not be described in detail here.

[0130] In a third aspect, an embodiment of the present application also provides a facial shadow removal device based on adaptive facial symmetry.

[0131] In one embodiment, a face shadow removal device based on adaptive face symmetry includes:

[0132] A face flipping module is used to flip the face area in the initial image to obtain a flipped image;

[0133] a mask generation module, configured to generate an initial symmetric mask based on the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0134] A feature extraction module is used to input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding, thereby obtaining the shadow-free features of the flipped image and the features of the initial symmetric mask;

[0135] The reverse denoising module is used to input the pure noise image, the initial image, the shadow-free features of the flipped image and the features of the initial symmetric mask into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain the shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the tth step to the t-1th step, the initial image and the noise image of the tth step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1th step, t∈[1,T]. The noise image of the Tth step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0136] Furthermore, in one embodiment, the face shadow removal apparatus based on adaptive face symmetry further includes a first training module for:

[0137] Get the initial image and its ground-truth shadow-free image from the training set;

[0138] Flip the face area in the initial image to obtain a flipped image;

[0139] Generate an initial symmetric mask based on the initial image and the flipped image;

[0140] Input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetric mask;

[0141] Randomly determine the step number t from [1, T];

[0142] Input the step number t, the real noise image of the t-th step, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetry mask into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain the single-step prediction result, where the single-step prediction result includes the predicted noise image of the t-1-th step;

[0143] Relevant parameters in the dynamic symmetric perception network are updated according to a first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the true noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the true shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the true noise image at the t-1th step.

[0144] Furthermore, in one embodiment, the single-step prediction result further includes a first prediction mask, the first prediction mask representing a predicted area that belongs to a face shadow in the original image and belongs to a non-face shadow in the flipped image;

[0145] The loss term of the first loss function also includes the loss of the first predicted mask relative to the first true mask, where the first true mask represents the real area belonging to the shadow of the face in the initial image and the real area belonging to the non-shadow of the face in the flipped image.

[0146] Furthermore, in one embodiment, the single-step prediction result further includes a second prediction mask, the second prediction mask representing the predicted area belonging to the face shadow in the initial image;

[0147] The loss term of the first loss function also includes the loss of the second predicted mask relative to the second true mask, where the second true mask represents the true area in the initial image that belongs to the shadow of the face.

[0148] Furthermore, in one embodiment, the face flipping module is used to:

[0149] Extract geometric information, key point information and surface texture information from the initial image through the facial feature extraction algorithm;

[0150] The surface texture information is flipped and combined with the geometric information and key point information to re-render the face area of ​​the initial image to obtain a flipped image.

[0151] Furthermore, in one embodiment, the face shadow removal apparatus based on adaptive face symmetry further includes a second training module for:

[0152] Get the initial image and its ground-truth shadow-free image from the training set;

[0153] The initial image is input into the symmetric feature extraction network for encoding and decoding to obtain the predicted shadow-free image of the initial image;

[0154] Update relevant parameters in the symmetric feature extraction network according to the second loss function, wherein the loss term in the second loss function includes at least one of SGAN loss, pixel loss, perceptual loss and smoothness loss of the initial image relative to the true shadow-free image.

[0155] Fourthly, the embodiments of the present application also provide another face shadow removal device based on adaptive face symmetry.

[0156] In one embodiment, a face shadow removal device based on adaptive face symmetry includes:

[0157] A face flipping module is used to flip the face area in the initial image to obtain a flipped image;

[0158] a mask generation module, configured to generate an initial symmetric mask based on the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0159] The feature extraction module is used to input the flipped image into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0160] The reverse denoising module is used to input the pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to decode the noise image of the t-1-th step, t∈[1,T]. The noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0161] Furthermore, in one embodiment, the face shadow removal apparatus based on adaptive face symmetry further includes a first training module for:

[0162] Get the initial image and its ground-truth shadow-free image from the training set;

[0163] Flip the face area in the initial image to obtain a flipped image;

[0164] Generate an initial symmetric mask based on the initial image and the flipped image;

[0165] The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0166] Randomly determine the step number t from [1, T];

[0167] Input the step number t, the real noise image of the t-th step, the initial image, the initial symmetry mask, and the shadow-free features of the flipped image into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain the single-step prediction result, where the single-step prediction result includes the predicted noise image of the t-1-th step;

[0168] Relevant parameters in the dynamic symmetric perception network are updated according to a first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the true noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the true shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the true noise image at the t-1th step.

[0169] Other detailed embodiments of the second device are consistent with those of the first device and will not be described in detail here.

[0170] Among them, the functional implementation of each module in the above-mentioned face shadow removal device based on adaptive face symmetry corresponds to the various steps in the above-mentioned face shadow removal method embodiment based on adaptive face symmetry, and its functions and implementation processes will not be repeated here one by one.

[0171] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0172] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.

[0173] In the description of the embodiments of this application, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0174] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0175] In some processes described in the embodiments of the present application, multiple operations or steps are included that appear in a specific order. However, it should be understood that these operations or steps may not be performed in the order in which they appear in the embodiments of the present application or may be performed in parallel. The sequence numbers of the operations are only used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be performed in sequence or in parallel, and these operations or steps may be combined.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of the present application.

[0177] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A face shadow removal method based on adaptive face symmetry, characterized in that: The face shadow elimination method based on adaptive face symmetry includes: Flip the face area in the initial image to obtain a flipped image; generating an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image; Input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetric mask; The pure noise image, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetric mask are input into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1-th step, t∈[1,T]. The noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

2. The method for removing face shadows based on adaptive face symmetry according to claim 1, wherein: The training process of the dynamic symmetry perception network includes: Get the initial image and its ground-truth shadow-free image from the training set; Flip the face area in the initial image to obtain a flipped image; Generate an initial symmetric mask based on the initial image and the flipped image; Input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetric mask; Randomly determine the step number t from [1, T]; Input the step number t, the real noise image of the t-th step, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetry mask into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain the single-step prediction result, where the single-step prediction result includes the predicted noise image of the t-1-th step; Relevant parameters in the dynamic symmetric perception network are updated according to a first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the true noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the true shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the true noise image at the t-1th step.

3. The method for removing face shadows based on adaptive face symmetry according to claim 2, wherein: The single-step prediction result also includes a first prediction mask, which represents the predicted area that belongs to the shadow of the face in the original image and belongs to the non-shadow of the face in the flipped image; The loss term of the first loss function also includes the loss of the first predicted mask relative to the first true mask, where the first true mask represents the real area belonging to the shadow of the face in the initial image and the real area belonging to the non-shadow of the face in the flipped image.

4. The method for removing face shadows based on adaptive face symmetry according to claim 2, wherein: The single-step prediction result also includes a second prediction mask, which represents the predicted area belonging to the face shadow in the initial image; The loss term of the first loss function also includes the loss of the second predicted mask relative to the second true mask, where the second true mask represents the true area in the initial image that belongs to the shadow of the face.

5. The method for removing face shadows based on adaptive face symmetry according to claim 1, wherein: The step of flipping the face area in the initial image to obtain a flipped image comprises: Extract geometric information, key point information and surface texture information from the initial image through the facial feature extraction algorithm; The surface texture information is flipped and combined with the geometric information and key point information to re-render the face area of ​​the initial image to obtain a flipped image.

6. The method for removing face shadows based on adaptive face symmetry according to claim 1, wherein: The training process of the symmetric feature extraction network includes: Get the initial image and its ground-truth shadow-free image from the training set; The initial image is input into the symmetric feature extraction network for encoding and decoding to obtain the predicted shadow-free image of the initial image; Update relevant parameters in the symmetric feature extraction network according to the second loss function, wherein the loss term in the second loss function includes at least one of SGAN loss, pixel loss, perceptual loss and smoothness loss of the initial image relative to the true shadow-free image.

7. A face shadow removal method based on adaptive face symmetry, characterized in that: The face shadow elimination method based on adaptive face symmetry includes: Flip the face area in the initial image to obtain a flipped image; generating an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image; The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image; The pure noise image, initial image, initial symmetry mask and shadow-free features of the flipped image are input into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to decode the noise image of the t-1-th step, t∈[1,T]. The noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

8. The method for removing face shadows based on adaptive face symmetry according to claim 7, wherein: The training process of the dynamic symmetry perception network includes: Get the initial image and its ground-truth shadow-free image from the training set; Flip the face area in the initial image to obtain a flipped image; Generate an initial symmetric mask based on the initial image and the flipped image; The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image; Randomly determine the step number t from [1, T]; Input the step number t, the real noise image of the t-th step, the initial image, the initial symmetry mask, and the shadow-free features of the flipped image into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain the single-step prediction result, where the single-step prediction result includes the predicted noise image of the t-1-th step; Relevant parameters in the dynamic symmetric perception network are updated according to a first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the true noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the true shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the true noise image at the t-1th step.

9. A face shadow removal device based on adaptive face symmetry, characterized in that: The face shadow removal device based on adaptive face symmetry comprises: A face flipping module is used to flip the face area in the initial image to obtain a flipped image; a mask generation module, configured to generate an initial symmetric mask based on the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image; A feature extraction module is used to input the flipped image and the initial symmetric mask into the trained symmetric feature extraction network for encoding, thereby obtaining the shadow-free features of the flipped image and the features of the initial symmetric mask; The reverse denoising module is used to input the pure noise image, the initial image, the shadow-free features of the flipped image and the features of the initial symmetric mask into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain the shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the tth step to the t-1th step, the initial image and the noise image of the tth step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1th step, t∈[1,T]. The noise image of the Tth step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

10. A face shadow removal device based on adaptive face symmetry, characterized in that: The face shadow removal device based on adaptive face symmetry comprises: A face flipping module is used to flip the face area in the initial image to obtain a flipped image; a mask generation module, configured to generate an initial symmetric mask based on the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image; The feature extraction module is used to input the flipped image into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image; The reverse denoising module is used to input the pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to decode the noise image of the t-1-th step, t∈[1,T]. The noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

Images