Face shadow elimination method and device based on adaptive face symmetry

Abstract

The invention provides a face shadow elimination method and device based on self-adaptive face symmetry. The method comprises the following steps: overturning a face region in an initial image to obtain an overturned image; an initial symmetric mask is generated according to the initial image and the flipped image, and the initial symmetric mask represents an area with the RGB pixel value higher than that of the initial image in the flipped image; inputting the flipped image and the initial symmetric mask into a trained symmetric feature extraction network for coding to obtain a shadow-free feature of the flipped image and a feature of the initial symmetric mask; and inputting the shadow-free features of the pure noise image, the initial image and the flipped image and the features of the initial symmetric mask into a trained dynamic symmetric sensing network for gradual reverse denoising to obtain a shadow-free image of the initial image, the dynamic symmetric sensing network being based on a diffusion model. According to the method and the device, the information of the face symmetric region can be more effectively utilized, and the face self-shadow is reserved while the face shadow is eliminated.

Description

Technical Field

[0001] The present application relates to the technical field of face shadow removal, and in particular to a face shadow removal method and device based on adaptive face symmetry. Background Art

[0002] With the development of technology, face recognition and beautification technology has become more and more mature, and has gradually penetrated into the fields of security, live broadcast, etc. However, the user's usage environment is often complex and changeable, and face shadows will inevitably appear, affecting the accuracy of face recognition and the beautification effect. Therefore, effectively removing face shadows is a crucial task.

[0003] In related technologies, information about symmetrical areas of the face is used as a reference for shadow restoration because facial shadows are usually asymmetrical. However, due to the randomness of the light source position and facial direction in real scenes, the face itself often undergoes lighting changes. Directly using symmetry often provides incorrect information to the network, resulting in the removal of face shadows while also removing the face's self-shadows that should have been retained. This makes it impossible to retain the original appearance of the face, and the removed image is unrealistic. Summary of the invention

[0004] The present application provides a method and device for removing face shadows based on adaptive face symmetry, which can solve the technical problem in the prior art that the image after face shadow removal is unrealistic.

[0005] In a first aspect, an embodiment of the present application provides a method for removing face shadows based on adaptive face symmetry, and the method for removing face shadows based on adaptive face symmetry includes:

[0006] Flip the face area in the initial image to obtain a flipped image;

[0007] Generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in which the RGB pixel value of the flipped image is higher than that of the initial image;

[0008] Input the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetry mask;

[0009] The pure noise image, the initial image, the shadow-free features of the flipped image and the features of the initial symmetric mask are input into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0010] Furthermore, in one embodiment, the training process of the dynamic symmetry perception network includes:

[0011] Get the initial image and its ground-truth shadow-free image from the training set;

[0012] Flip the face area in the initial image to obtain a flipped image;

[0013] Generate an initial symmetric mask based on the initial image and the flipped image;

[0014] Input the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetry mask;

[0015] Randomly determine the step number t from [1, T];

[0016] Input the step number t, the real noise image of the t-th step, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetric mask into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain a single-step prediction result, wherein the single-step prediction result includes the predicted noise image of the t-1-th step;

[0017] Update relevant parameters in the dynamic symmetric perception network according to the first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the real noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the real shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the real noise image at the t-1th step.

[0018] Furthermore, in one embodiment, the single-step prediction result further includes a first prediction mask, the first prediction mask representing a prediction area that belongs to a face shadow in the initial image and a face non-shadow in the flipped image;

[0019] The loss term of the first loss function also includes the loss of the first predicted mask relative to the first real mask, where the first real mask represents the real area belonging to the shadow of the face in the initial image and the non-shadow of the face in the flipped image.

[0020] Furthermore, in one embodiment, the single-step prediction result further includes a second prediction mask, and the second prediction mask represents a predicted area belonging to a face shadow in the initial image;

[0021] The loss term of the first loss function also includes a loss of the second predicted mask relative to the second true mask, where the second true mask represents the true area in the initial image that belongs to the shadow of the face.

[0022] Furthermore, in one embodiment, the step of flipping the face region in the initial image to obtain a flipped image includes:

[0023] Extract geometric information, key point information and surface texture information from the initial image through the face feature extraction algorithm;

[0024] The surface texture information is flipped, and the geometric information and key point information are combined to re-render the face area of ​​the initial image to obtain a flipped image.

[0025] Furthermore, in one embodiment, the training process of the symmetric feature extraction network includes:

[0026] Get the initial image and its ground-truth shadow-free image from the training set;

[0027] The initial image is input into the symmetric feature extraction network for encoding and decoding to obtain a predicted shadow-free image of the initial image;

[0028] Update relevant parameters in the symmetric feature extraction network according to the second loss function, wherein the loss term in the second loss function includes at least one of SGAN loss, pixel loss, perceptual loss and smoothness loss of the initial image relative to the true shadow-free image.

[0029] In a second aspect, the embodiment of the present application also provides another method for removing face shadows based on adaptive face symmetry, and the method for removing face shadows based on adaptive face symmetry includes:

[0030] Flip the face area in the initial image to obtain a flipped image;

[0031] Generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in which the RGB pixel value of the flipped image is higher than that of the initial image;

[0032] The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0033] The pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image are input into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to be decoded to obtain the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0034] Furthermore, in one embodiment, the training process of the dynamic symmetry perception network includes:

[0035] Get the initial image and its ground-truth shadow-free image from the training set;

[0036] Flip the face area in the initial image to obtain a flipped image;

[0037] Generate an initial symmetric mask based on the initial image and the flipped image;

[0038] The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0039] Randomly determine the step number t from [1, T];

[0040] Input the step number t, the real noise image of the t-th step, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain a single-step prediction result, wherein the single-step prediction result includes the predicted noise image of the t-1-th step;

[0041] Update relevant parameters in the dynamic symmetric perception network according to the first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the real noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the real shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the real noise image at the t-1th step.

[0042] In a third aspect, an embodiment of the present application provides a face shadow removal device based on adaptive face symmetry, and the face shadow removal device based on adaptive face symmetry includes:

[0043] A face flipping module is used to flip the face area in the initial image to obtain a flipped image;

[0044] A mask generation module, used to generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0045] A feature extraction module is used to input the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, so as to obtain the shadow-free features of the flipped image and the features of the initial symmetry mask;

[0046] The reverse denoising module is used to input the pure noise image, the initial image, the shadow-free features of the flipped image and the features of the initial symmetric mask into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0047] In a fourth aspect, the embodiment of the present application further provides another face shadow removal device based on adaptive face symmetry, the face shadow removal device based on adaptive face symmetry comprising:

[0048] A face flipping module is used to flip the face area in the initial image to obtain a flipped image;

[0049] A mask generation module, used to generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0050] A feature extraction module is used to input the flipped image into a trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0051] The reverse denoising module is used to input the pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to decode the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0052] In this application, a dynamic symmetry perception network is designed based on a diffusion model. A shadow-free image of the initial image is constructed from a pure noise image by stepwise reverse denoising. The dynamic symmetry perception network learns what image's shadow-free image to construct through the features of the initial image, and learns information that is helpful for removing face shadows by flipping the shadow-free features of the image and the features of the initial symmetry mask. Through this application, the information of the symmetrical area of ​​the face can be more effectively utilized, and the self-shadow of the face can be retained while removing the face shadow, thereby retaining the original appearance of the face and generating a more realistic shadow-free image. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a method for removing face shadows based on adaptive face symmetry in one embodiment of the present application;

[0054] Figure 2 for Figure 1 Detailed schematic diagram of step S13 and step S14;

[0055] Figure 3 This is a flow chart of a method for removing face shadows based on adaptive face symmetry in another embodiment of the present application;

[0056] Figure 4 for Figure 3 Detailed schematic diagram of step S23 and step S24 in FIG. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0059] In a first aspect, an embodiment of the present application provides a method for removing face shadows based on adaptive face symmetry.

[0060] Figure 1 A flow chart of a method for removing face shadows based on adaptive face symmetry in one embodiment of the present application is shown.

[0061] Reference Figure 1 In one embodiment, a method for removing face shadows based on adaptive face symmetry includes the following steps:

[0062] S11, flipping the face region in the initial image to obtain a flipped image.

[0063] Specifically, in the face area, the pixel points with the same coordinates in the flipped image and the initial image correspond to two symmetrically distributed pixel points in the initial image. In the non-face area, the pixel points with the same coordinates in the flipped image and the initial image correspond to the same pixel point in the initial image.

[0064] Furthermore, in one embodiment, step S11 specifically includes:

[0065] Extract geometric information, key point information and surface texture information from the initial image through the face feature extraction algorithm;

[0066] The surface texture information is flipped, and the geometric information and key point information are combined to re-render the face area of ​​the initial image to obtain a flipped image.

[0067] S12. Generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image.

[0068] Specifically, pixel-level calculations are performed on the initial image and the flipped image to determine the pixels whose RGB pixel values ​​in the flipped image are higher than those in the initial image, and the position information of these pixels is aggregated to form an initial symmetric mask. Since the RGB pixel values ​​in the shadow area are lower than those in the non-shadow area, the initial symmetric mask can be used to preliminarily predict the area that belongs to the face shadow in the initial image and the area that belongs to the face non-shadow in the flipped image. This area can provide information that is helpful for shadow elimination. Since the actual face structure is not completely symmetrical and there are asymmetric face self-shadows due to the influence of the light source, there will be a certain error between the initial symmetric mask and the actual area that belongs to the face shadow in the initial image and the face non-shadow in the flipped image.

[0069] It should be noted that the "face shadow" and "face self-shadow" described in this application have completely different meanings and need to be distinguished. Face shadow is formed when an external light source is blocked by other objects and projected onto the face. For example, when a person stands in the sun and the buildings, trees, etc. next to him block part of the light, an external shadow will be formed on the face. Face self-shadow is determined by the structure and shape of the face itself. The raised parts of the face (such as the nose, brow bones, etc.) and the sunken parts (such as the eye sockets, nasolabial folds, etc.) will naturally form shadows under specific light angles. It is an inherent feature of the human face and has nothing to do with external obstructions.

[0070] S13, inputting the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, and obtaining the shadow-free features of the flipped image and the features of the initial symmetry mask.

[0071] In this embodiment, the symmetric feature extraction network adopts an encoder-decoder structure to remove face shadows from the input image to obtain a shadow-free image. The face shadow removal algorithm adopted by the symmetric feature extraction network can refer to the prior art, or be further optimized on the basis of the prior art. The shadow removal operation performed here is to provide more effective information for the shadow removal of the initial image. The initial symmetric mask does not play a role in the shadow removal of the flipped image, but only uses the encoder of the symmetric feature extraction network to extract features.

[0072] It should be noted that the symmetric feature extraction network needs to use a decoder to obtain a predicted shadow-free image during the training process, and update relevant parameters according to the loss of the predicted shadow-free image relative to the real shadow-free image. The trained symmetric feature extraction network does not need to use a decoder in actual application, and directly provides the features extracted by the encoder to subsequent steps.

[0073] S14. Input the pure noise image, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetry mask into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetry mask to decode the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0074] The process of the diffusion model is bidirectional, with the forward process being a step-by-step noise addition process and the reverse process being a step-by-step denoising process, with a total of T steps. In the forward process, the diffusion model gradually adds random noise to the original data. As time goes by, the data is gradually overwhelmed by the noise and eventually becomes a pure noise distribution in the limit. This process usually follows the concept of Markov chain, that is, the change of each state depends only on the previous state, and the change of state follows a specific probability distribution, which is a function of the latent variable. The reverse process is the opposite of the forward process, training a neural network to gradually remove the noise added to the data to restore the original data. By continuously learning the denoising process, the model can master the ability to generate new data similar to the original data from the noise. The training of the diffusion model is only for the reverse process, and the forward process is not changed.

[0075] The improvements made by the dynamic symmetry perception network on the basis of the diffusion model include: adding the initial image, the shadow-free features of the flipped image, and the features of the initial symmetry mask as additional inputs on the basis of the pure noise image, where the initial image needs to be encoded together with the noise image input at each step, and decoded by fusion with the shadow-free features of the flipped image and the features of the initial symmetry mask. The encoder of the dynamic symmetry perception network has a similar architecture to the encoder of the symmetry feature extraction network, but does not share weights, ensuring that the feature structures extracted by the two are consistent and easy to merge.

[0076] For example, the dynamic symmetry perception network and the symmetric feature extraction network are both UNet network architectures, consisting of six downsample layers, one intermediate layer, and six upsample layers. Each downsample layer consists of two convolution layers and one pooling layer, and the upsample layer consists of two convolution layers and one interpolation layer.

[0077] Figure 2 Shows Figure 1 Detailed schematic diagram of step S13 and step S14 in FIG.

[0078] Reference Figure 2 In the symmetry feature extraction network, the flipped image and the initial symmetry mask are encoded to obtain the shadow-free features of the flipped image and the features of the initial symmetry mask. In the dynamic symmetry perception network, the initial image and the pure noise image are encoded and fused with the shadow-free features of the flipped image and the features of the initial symmetry mask to form a fused feature. The fused feature is decoded to obtain the noisy image of the T-1th step. The initial image and the noisy image of the T-1th step are encoded and fused with the shadow-free features of the flipped image and the features of the initial symmetry mask to form a fused feature. The fused feature is decoded to obtain the noisy image of the T-2th step. And so on. After obtaining the noisy image of the first step, the initial image and the noisy image of the first step are encoded and fused with the shadow-free features of the flipped image and the features of the initial symmetry mask to form a fused feature. The fused feature is decoded to obtain the shadow-free image of the initial image.

[0079] In this embodiment, a dynamic symmetry perception network is designed based on a diffusion model, and a shadow-free image of the initial image is constructed from a pure noise image by stepwise reverse denoising. The dynamic symmetry perception network learns what image's shadow-free image to construct through the features of the initial image, and learns information that is helpful for eliminating face shadows by flipping the shadow-free features of the image and the features of the initial symmetry mask. Through this application, the information of the symmetrical area of ​​the face can be more effectively utilized, and the self-shadow of the face can be retained while the face shadow is eliminated, thereby retaining the original appearance of the face and generating a more realistic shadow-free image.

[0080] Furthermore, in one embodiment, the training process of the dynamic symmetry perception network includes:

[0081] Get the initial image and its ground-truth shadow-free image from the training set;

[0082] Flip the face area in the initial image to obtain a flipped image;

[0083] Generate an initial symmetric mask based on the initial image and the flipped image;

[0084] Input the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetry mask;

[0085] Randomly determine the step number t from [1, T];

[0086] Input the step number t, the real noise image of the t-th step, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetric mask into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain a single-step prediction result, wherein the single-step prediction result includes the predicted noise image of the t-1-th step;

[0087] Update relevant parameters in the dynamic symmetric perception network according to the first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the real noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the real shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the real noise image at the t-1th step.

[0088] In this embodiment, the dynamic symmetry perception network uses the initial image in the training set and its true shadow-free image. Forward denoising is performed on the true shadow-free image to obtain the true noise image of any step and the added true noise. The same operation is performed on the initial image during the training process and the application process. The difference is that the training process randomly executes one of the denoising steps each time, and the input noise image is a true noise image. The loss is calculated based on the single-step prediction result and the corresponding true result.

[0089] The single-step prediction result includes at least the predicted noise image of the t-1th step, and other contents may be included to improve the training effect. There are two ways to calculate the loss for the predicted noise image of the t-1th step. One is to calculate the loss of the predicted noise relative to the real noise, and the other is to calculate the loss of the predicted noise image relative to the real noise image. The two losses are equivalent, but the first one is more efficient.

[0090] For example, the calculation formula for adding noise is as follows:

[0091]

[0092] Among them, x t is the real noise image of the tth step, x0 is the real shadow-free image, α t =1-β t , β t is the noise level,

[0093] For example, the loss of predicted noise relative to real noise is calculated as follows:

[0094]

[0095] Among them, e t represents the prediction noise at step t, ∈ t represents the real noise at the tth step.

[0096] Furthermore, in one embodiment, the single-step prediction result further includes a first prediction mask, the first prediction mask representing a prediction area that belongs to a face shadow in the initial image and a face non-shadow in the flipped image;

[0097] The loss term of the first loss function also includes the loss of the first predicted mask relative to the first real mask, where the first real mask represents the real area belonging to the shadow of the face in the initial image and the non-shadow of the face in the flipped image.

[0098] In this embodiment, a first prediction mask is added to the single-step prediction result. Compared with the initial symmetric mask, the first prediction mask refers to more information and makes more accurate predictions for the areas belonging to the face shadow in the initial image and the non-shadow areas of the face in the flipped image. Since the face shadow areas and non-shadow areas in the initial image are known information during the training process, the first true mask can be determined in combination with the flipped image. Increasing the loss of the first prediction mask relative to the first true mask in the first loss function helps to improve the training effect.

[0099] For example, the loss of the first predicted mask relative to the first true mask is calculated as follows:

[0100]

[0101] in, represents the first prediction mask, Represents the first true mask.

[0102] Furthermore, in one embodiment, the single-step prediction result further includes a second prediction mask, and the second prediction mask represents a predicted area belonging to a face shadow in the initial image;

[0103] The loss term of the first loss function also includes a loss of the second predicted mask relative to the second true mask, where the second true mask represents the true area belonging to the face shadow in the initial image.

[0104] In this embodiment, the second prediction mask is added to the single-step prediction result, and the loss of the second prediction mask relative to the second true mask is increased in the first loss function, which helps to improve the training effect.

[0105] For example, the loss of the second predicted mask relative to the second true mask is calculated as follows:

[0106]

[0107] in, represents the second prediction mask, Represents the second real mask.

[0108] Furthermore, in one embodiment, the training process of the symmetric feature extraction network includes:

[0109] Get the initial image and its ground-truth shadow-free image from the training set;

[0110] Input the initial image into the symmetric feature extraction network for encoding and decoding to obtain a predicted shadow-free image of the initial image;

[0111] Update relevant parameters in the symmetric feature extraction network according to the second loss function, wherein the loss term in the second loss function includes at least one of SGAN loss, pixel loss, perceptual loss and smoothness loss of the initial image relative to the true shadow-free image.

[0112] Specifically, SGAN (Soft-GAN) loss usually uses a binary cross entropy loss function to make the results more visually realistic. Pixel loss suppresses color and texture distortion by comparing the difference between the pixel values ​​of the generated results and the pixel values ​​of the real results. Perceptual loss generates more realistic and detailed image results by comparing the difference between the generated results and the real results in a high-dimensional representation space. Smoothness loss is usually implemented by penalizing the gradient of the generated results to make the generated results have a smooth appearance.

[0113] In a second aspect, an embodiment of the present application provides another method for removing face shadows based on adaptive face symmetry.

[0114] Figure 3 A schematic flow chart of a method for removing face shadows based on adaptive face symmetry in another embodiment of the present application is shown; Figure 4 Shows Figure 3 Detailed schematic diagram of step S23 and step S24 in FIG.

[0115] Reference Figure 3 and Figure 4 In one embodiment, a method for removing face shadows based on adaptive face symmetry includes the following steps:

[0116] S21, flipping the face region in the initial image to obtain a flipped image;

[0117] S22, generating an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0118] S23, inputting the flipped image into the trained symmetric feature extraction network for encoding to obtain a shadow-free feature of the flipped image;

[0119] S24. Input the pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to be decoded to obtain the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0120] The difference between the second method and the first method is that the initial symmetry mask of the first method is encoded in the symmetry feature extraction network and the encoded features are input into the dynamic symmetry perception network, while the initial symmetry mask of the second method is encoded in the dynamic symmetry perception network. Both methods are theoretically feasible and can generate more realistic shadow-free images, but in practical applications, it is found that the first method has better overall effect.

[0121] Accordingly, the training process of the dynamic symmetry perception network includes:

[0122] Get the initial image and its ground-truth shadow-free image from the training set;

[0123] Flip the face area in the initial image to obtain a flipped image;

[0124] Generate an initial symmetric mask based on the initial image and the flipped image;

[0125] The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0126] Randomly determine the step number t from [1, T];

[0127] Input the step number t, the real noise image of the t-th step, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain a single-step prediction result, wherein the single-step prediction result includes the predicted noise image of the t-1-th step;

[0128] Update relevant parameters in the dynamic symmetric perception network according to the first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the real noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the real shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the real noise image at the t-1th step.

[0129] Other detailed embodiments of the second method are consistent with the first method and will not be described in detail here.

[0130] In a third aspect, an embodiment of the present application also provides a face shadow removal device based on adaptive face symmetry.

[0131] In one embodiment, a face shadow removal device based on adaptive face symmetry includes:

[0132] A face flipping module is used to flip the face area in the initial image to obtain a flipped image;

[0133] A mask generation module, used to generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0134] A feature extraction module is used to input the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, so as to obtain the shadow-free features of the flipped image and the features of the initial symmetry mask;

[0135] The reverse denoising module is used to input the pure noise image, the initial image, the shadow-free features of the flipped image and the features of the initial symmetric mask into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0136] Furthermore, in one embodiment, the face shadow removal device based on adaptive face symmetry further includes a first training module for:

[0137] Get the initial image and its ground-truth shadow-free image from the training set;

[0138] Flip the face area in the initial image to obtain a flipped image;

[0139] Generate an initial symmetric mask based on the initial image and the flipped image;

[0140] Input the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetry mask;

[0141] Randomly determine the step number t from [1, T];

[0142] Input the step number t, the real noise image of the t-th step, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetric mask into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain a single-step prediction result, wherein the single-step prediction result includes the predicted noise image of the t-1-th step;

[0143] Update relevant parameters in the dynamic symmetric perception network according to the first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the real noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the real shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the real noise image at the t-1th step.

[0144] Furthermore, in one embodiment, the single-step prediction result further includes a first prediction mask, the first prediction mask representing a prediction area that belongs to a face shadow in the initial image and a face non-shadow in the flipped image;

[0145] The loss term of the first loss function also includes the loss of the first predicted mask relative to the first real mask, where the first real mask represents the real area belonging to the shadow of the face in the initial image and the non-shadow of the face in the flipped image.

[0146] Furthermore, in one embodiment, the single-step prediction result further includes a second prediction mask, and the second prediction mask represents a predicted area belonging to a face shadow in the initial image;

[0147] The loss term of the first loss function also includes a loss of the second predicted mask relative to the second true mask, where the second true mask represents the true area in the initial image that belongs to the shadow of the face.

[0148] Furthermore, in one embodiment, the face flipping module is used to:

[0149] Extract geometric information, key point information and surface texture information from the initial image through the face feature extraction algorithm;

[0150] The surface texture information is flipped, and the geometric information and key point information are combined to re-render the face area of ​​the initial image to obtain a flipped image.

[0151] Furthermore, in one embodiment, the face shadow removal device based on adaptive face symmetry further includes a second training module for:

[0152] Get the initial image and its ground-truth shadow-free image from the training set;

[0153] Input the initial image into the symmetric feature extraction network for encoding and decoding to obtain a predicted shadow-free image of the initial image;

[0154] Update relevant parameters in the symmetric feature extraction network according to the second loss function, wherein the loss term in the second loss function includes at least one of SGAN loss, pixel loss, perceptual loss and smoothness loss of the initial image relative to the true shadow-free image.

[0155] In a fourth aspect, the embodiments of the present application also provide another face shadow removal device based on adaptive face symmetry.

[0156] In one embodiment, a face shadow removal device based on adaptive face symmetry includes:

[0157] A face flipping module is used to flip the face area in the initial image to obtain a flipped image;

[0158] A mask generation module, used to generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image;

[0159] A feature extraction module is used to input the flipped image into a trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0160] The reverse denoising module is used to input the pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to decode the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

[0161] Furthermore, in one embodiment, the face shadow removal device based on adaptive face symmetry further includes a first training module for:

[0162] Get the initial image and its ground-truth shadow-free image from the training set;

[0163] Flip the face area in the initial image to obtain a flipped image;

[0164] Generate an initial symmetric mask based on the initial image and the flipped image;

[0165] The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image;

[0166] Randomly determine the step number t from [1, T];

[0167] Input the step number t, the real noise image of the t-th step, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain a single-step prediction result, wherein the single-step prediction result includes the predicted noise image of the t-1-th step;

[0168] Update relevant parameters in the dynamic symmetric perception network according to the first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the real noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the real shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the real noise image at the t-1th step.

[0169] Other detailed embodiments of the second device are consistent with those of the first device and will not be described in detail here.

[0170] Among them, the functional implementation of each module in the above-mentioned face shadow removal device based on adaptive face symmetry corresponds to the steps in the above-mentioned face shadow removal method embodiment based on adaptive face symmetry, and its functions and implementation process will not be repeated here one by one.

[0171] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0172] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0173] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0174] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0175] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0177] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A face shadow removal method based on adaptive face symmetry, characterized in that: The face shadow elimination method based on adaptive face symmetry comprises: Flip the face area in the initial image to obtain a flipped image; Generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in which the RGB pixel value of the flipped image is higher than that of the initial image; Input the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetry mask; The pure noise image, the initial image, the shadow-free features of the flipped image and the features of the initial symmetric mask are input into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

2. The method for removing face shadows based on adaptive face symmetry as claimed in claim 1, characterized in that: The training process of the dynamic symmetry perception network includes: Get the initial image and its ground-truth shadow-free image from the training set; Flip the face area in the initial image to obtain a flipped image; Generate an initial symmetric mask based on the initial image and the flipped image; Input the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, and obtain the shadow-free features of the flipped image and the features of the initial symmetry mask; Randomly determine the step number t from [1, T]; Input the step number t, the real noise image of the t-th step, the initial image, the shadow-free features of the flipped image, and the features of the initial symmetric mask into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain a single-step prediction result, wherein the single-step prediction result includes the predicted noise image of the t-1-th step; Update relevant parameters in the dynamic symmetric perception network according to the first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the real noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the real shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the real noise image at the t-1th step.

3. The method for removing face shadows based on adaptive face symmetry as claimed in claim 2, characterized in that: The single-step prediction result also includes a first prediction mask, which represents a prediction area that belongs to a face shadow in the initial image and a non-face shadow in the flipped image; The loss term of the first loss function also includes the loss of the first predicted mask relative to the first real mask, where the first real mask represents the real area belonging to the shadow of the face in the initial image and the non-shadow of the face in the flipped image.

4. The method for removing face shadows based on adaptive face symmetry as claimed in claim 2, characterized in that: The single-step prediction result also includes a second prediction mask, which represents a predicted area in the initial image that belongs to a face shadow; The loss term of the first loss function also includes a loss of the second predicted mask relative to the second true mask, where the second true mask represents the true area belonging to the face shadow in the initial image.

5. The method for removing face shadows based on adaptive face symmetry as claimed in claim 1, characterized in that: The step of flipping the face area in the initial image to obtain a flipped image comprises: Extract geometric information, key point information and surface texture information from the initial image through the face feature extraction algorithm; The surface texture information is flipped, and the geometric information and key point information are combined to re-render the face area of ​​the initial image to obtain a flipped image.

6. The method for removing face shadows based on adaptive face symmetry as claimed in claim 1, characterized in that: The training process of the symmetric feature extraction network includes: Get the initial image and its ground-truth shadow-free image from the training set; Input the initial image into the symmetric feature extraction network for encoding and decoding to obtain a predicted shadow-free image of the initial image; Update relevant parameters in the symmetric feature extraction network according to the second loss function, wherein the loss term in the second loss function includes at least one of SGAN loss, pixel loss, perceptual loss and smoothness loss of the initial image relative to the true shadow-free image.

7. A method for removing face shadows based on adaptive face symmetry, characterized in that: The face shadow elimination method based on adaptive face symmetry comprises: Flip the face area in the initial image to obtain a flipped image; Generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in which the RGB pixel value of the flipped image is higher than that of the initial image; The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image; The pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image are input into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image. The dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to be decoded to obtain the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

8. The method for removing face shadows based on adaptive face symmetry as claimed in claim 7, characterized in that: The training process of the dynamic symmetry perception network includes: Get the initial image and its ground-truth shadow-free image from the training set; Flip the face area in the initial image to obtain a flipped image; Generate an initial symmetric mask based on the initial image and the flipped image; The flipped image is input into the trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image; Randomly determine the step number t from [1, T]; Input the step number t, the real noise image of the t-th step, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the dynamic symmetry perception network for reverse denoising from the t-th step to the t-1-th step, and obtain a single-step prediction result, wherein the single-step prediction result includes the predicted noise image of the t-1-th step; Update relevant parameters in the dynamic symmetric perception network according to the first loss function, wherein the loss term of the first loss function includes the loss of the predicted noise at the t-1th step relative to the real noise at the t-1th step, and the predicted noise at the t-1th step is calculated by the predicted noise image at the t-1th step and the real shadow-free image of the initial image, or the loss term of the first loss function includes the loss of the predicted noise image at the t-1th step relative to the real noise image at the t-1th step.

9. A face shadow removal device based on adaptive face symmetry, characterized in that: The face shadow removal device based on adaptive face symmetry comprises: A face flipping module is used to flip the face area in the initial image to obtain a flipped image; A mask generation module, used to generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image; A feature extraction module is used to input the flipped image and the initial symmetry mask into the trained symmetry feature extraction network for encoding, so as to obtain the shadow-free features of the flipped image and the features of the initial symmetry mask; The reverse denoising module is used to input the pure noise image, the initial image, the shadow-free features of the flipped image and the features of the initial symmetric mask into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image and the features of the initial symmetric mask to decode the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

10. A face shadow removal device based on adaptive face symmetry, characterized in that: The face shadow removal device based on adaptive face symmetry comprises: A face flipping module is used to flip the face area in the initial image to obtain a flipped image; A mask generation module, used to generate an initial symmetric mask according to the initial image and the flipped image, wherein the initial symmetric mask represents an area in the flipped image where the RGB pixel value is higher than that of the initial image; A feature extraction module is used to input the flipped image into a trained symmetric feature extraction network for encoding to obtain the shadow-free features of the flipped image; The reverse denoising module is used to input the pure noise image, the initial image, the initial symmetry mask and the shadow-free features of the flipped image into the trained dynamic symmetry perception network for step-by-step reverse denoising to obtain a shadow-free image of the initial image, wherein the dynamic symmetry perception network is based on a diffusion model. In the reverse denoising from the t-th step to the t-1-th step, the initial image, the initial symmetry mask and the noise image of the t-th step are encoded, and the encoded features are fused with the shadow-free features of the flipped image to decode the noise image of the t-1-th step, t∈[1,T], the noise image of the T-th step is a pure noise image, and the noise image of the 0th step is a shadow-free image of the initial image.

Images