An image shadow removal method based on a double-branch network

Abstract

The application belongs to the technical field of image processing, and particularly relates to a kind of image shadow removal methods based on double branch network;The method comprises: obtaining to-be-processed shadow image and its corresponding mask image, pre-processing shadow image and mask image, obtaining shadow area image and non-shadow area image;Shadow area image and non-shadow area image are respectively sent into shadow removal branch network and identity mapping branch network for processing, and shadow-removed area image and mapping image are obtained;Shadow-removed area image and mapping image are added pixel by pixel, and preliminary shadow-removed image is obtained;Preliminary shadow-removed image is sent into color repair module for color repair, and clear image after shadow removal is obtained;The application can effectively realize shadow removal, and can reduce color deviation problem in shadow removal result, and has good application prospect.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an image shadow removal method based on a dual-branch network. Background Technology

[0002] In computer vision, shadows in images degrade image quality to varying degrees, posing significant challenges to many downstream computer tasks. In object detection, shadows blur object boundaries, making accurate contour recognition difficult. Accurate object boundaries are crucial for object localization and segmentation; shadows increase the difficulty. In object tracking, shadows obscure surface texture and details, hindering texture-based tracking algorithms from accurately extracting features and reducing performance. In face detection, shadows can occlude or cover parts of faces, preventing complete detection. Blurred boundaries between shadowed and face regions can lead to incorrect results or missed faces. Therefore, image shadow removal is generally considered an upstream task in computer vision, significantly impacting many downstream tasks. Thus, image shadow removal is a highly challenging and valuable task.

[0003] Traditional shadow removal methods typically focus on the low-level features of shadows, aiming to recover information such as material, brightness, and gradient in the shadowed area. Some methods solve for shadow-free images using Gaussian mixture models or physically based lighting models; others use gradient fields to calculate the Poisson equation; and still others eliminate shadows by matching shadowed and non-shadowed regions. Overall, these traditional shadow removal methods heavily rely on low-level features, which have inherent limitations in many situations. For example, these feature-based methods often struggle to understand the global semantic context of the scene, thus failing to address shadow removal in complex backgrounds or lighting conditions.

[0004] In recent years, with the release of shadow removal datasets and the development of deep learning research and computer hardware, shadow removal methods based on convolutional neural networks (CNNs) have made significant progress. A CNN is a deep neural network with a convolutional structure. Its main idea is based on the operation of the human visual system: extracting local features of an image using local receptive fields and combining these local features into global features at a higher level to achieve tasks such as image classification, recognition, and segmentation. CNNs can automatically learn features in images and perform feature extraction and dimensionality reduction through operations such as convolution, pooling, and activation. They also possess advantages such as translation invariance and parameter sharing, making them widely used in the field of computer vision.

[0005] Utilizing deep learning for shadow removal simplifies the physical modeling and manual feature extraction processes, achieving good removal results. The application of convolutional neural networks (CNNs) in shadow removal leverages their deep feature extraction capabilities, thereby improving the accuracy and efficiency of shadow removal. Generally, the deeper the neural network, the more complex the extracted features. However, many deep learning-based shadow removal methods suffer from insufficient interpretability, making it difficult to effectively interpret and understand the results. Although some methods attempt to use lighting models for shadow removal, modeling the transformation between shadowed and unshadowed pixels as a linear relationship, their design is relatively simplistic and fails to fully consider the nonlinear response characteristics of photoelectric sensors. Furthermore, current methods consider the same convolution operations for both shadowed and unshadowed regions, ignoring the significant differences in color mapping between shadowed and unshadowed regions, leading to poor image quality and a heavy computational burden. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes an image shadow removal method based on a dual-branch network. The method includes: acquiring the shadow image to be processed and its corresponding mask image, and inputting them into a trained dual-branch network for processing to obtain a clear image after shadow removal.

[0007] The training process of a dual-branch network includes:

[0008] S1: Obtain the shadow image to be processed and its corresponding mask image, preprocess the shadow image and mask image to obtain the shadow area image and the non-shadow area image;

[0009] S2: The shadow region image and the non-shadow region image are fed into the shadow removal branch network and the identity mapping branch network respectively for processing to obtain the shadow-removed region image and the mapped image;

[0010] S3: Add the image of the removed shadow area and the mapped image pixel by pixel to obtain the preliminary image with the shadow removed;

[0011] S4: The image with the initial shadow removal is sent to the color restoration module for color restoration to obtain a clear image after the shadow removal;

[0012] S5: Calculate the total loss of the dual-branch network and adjust the network parameters according to the total loss to obtain the trained dual-branch network.

[0013] Preferably, the preprocessing of the shadow image and the mask image includes:

[0014] Both the shadow image and the mask image are scaled to 256×256. The shadow image and the mask image are multiplied pixel by pixel to obtain the shadow area image. The mask image is then inverted and multiplied pixel by pixel with the shadow image to obtain the non-shadow area image.

[0015] Preferably, the process of feeding the image of the shaded region and the image of the unshaded region into the shadow removal branch network for processing includes:

[0016] The image of the shadowed area is input into a relative light intensity estimation network for processing to obtain the relative light intensity.

[0017] The non-shaded area image is input into the attenuation factor estimation network for processing to obtain the attenuation factor;

[0018] A response attenuation model is established based on the relationship between pixel values ​​and illumination. The response attenuation model is then fitted based on relative light intensity and attenuation factor to obtain the image of the shaded area.

[0019] Furthermore, both the relative light intensity estimation network and the attenuation factor estimation network adopt a symmetric encoder-decoder structure;

[0020] The encoder consists of 8 convolutional layers with 4 kernels, a stride of 2, and padding of 1. The first convolutional layer has 4 input channels and 64 output channels. The second to seventh convolutional layers have the following input channel counts: 64, 128, 256, 512, 512, 512, and 128, 256, 512, 512, 512, 512, respectively. LeakyReLU and BatchNorm layers are added before and after the second to seventh convolutional layers, respectively. The eighth convolutional layer has 512 input channels and 512 output channels, and a LeakyReLU layer is added before it.

[0021] The decoder consists of 8 deconvolutional layers. The number of input channels in the first deconvolutional layer is twice the number of output channels in the last convolutional layer of the encoder. The parameters of each deconvolutional layer are set to a kernel size of 4, a stride of 2, and padding of 1. The number of input channels in the 1st to 7th convolutional layers are 1024, 2048, 2048, 2048, 2048, 1024, and 512, respectively, and the number of output channels are 1024, 1024, 1024, 1024, 512, 256, and 128, respectively. ReLU layers and BatchNorm layers are added before and after the 1st to 7th convolutional layers, respectively. The 8th convolutional layer has 256 input channels and 3 output channels, and ReLU layers and sigmoid layers are added before and after the 8th convolutional layer, respectively.

[0022] Furthermore, the response decay model is expressed as:

[0023]

[0024] Where p represents the pixel value of the image after fitting the model, P s represents the shadow pixel value, k represents the ratio of incident light intensity between the shadowless area and the shadowed area, a represents the attenuation factor corresponding to the object, and R represents the reflectivity.

[0025] Preferably, the identity mapping branch network consists of an encoder and a decoder;

[0026] The encoder is based on the SR-Net architecture and consists of 6 convolutional layers. The first to fourth convolutional layers are followed by an instance normalization module and a LeakyReLU activation function, respectively. The fifth convolutional layer is followed by a BatchNorm layer and a LeakyReLU activation function. The parameters of all convolutional layers are set to a kernel size of 4×4, a stride of 2, and padding of 1. The number of input channels is 4, 64, 128, 256, 512, and 512, respectively, and the number of output channels is 64, 128, 256, 512, 512, and 512, respectively.

[0027] The decoder consists of 6 deconvolutional layers. LeakyReLU activation layers and BatchNorm layers are added before and after the first deconvolutional layer, respectively. LeakyReLU activation layers and instance normalization modules are added before and after the second to fifth deconvolutional layers, respectively. LeakyReLU activation layers and tanh activation layers are added before and after the sixth deconvolutional layer, respectively. The parameters of all deconvolutional layers are set to kernel size 4×4, stride 2, and padding 1. The number of input channels of the 6 deconvolutional layers are 512, 1024, 1024, 512, 256, and 128, respectively, and the number of output channels are 512, 512, 256, 128, 64, and 3, respectively.

[0028] Preferably, the process of sending the initially shadow-removed image to the color restoration module for color restoration is represented as follows:

[0029]

[0030] in, p represents the normalized pixel value. h,w,c This represents the initial pixel values ​​of the image after initial shadow removal; This represents the average value of the pixels in the shaded area. The variance of the pixels in the shadow area. Represents the variance of pixels in the non-shaded area. This represents the average value of pixels in the non-shaded area.

[0031] Preferably, the formula for calculating the total loss of a two-branch network is:

[0032] L=λ1L STF +λ2L FTF +λ3L coarse +λ4L fine

[0033] L STF =||I RES_S -I SF_S ||

[0034] L FTF =||I RES_SF -I SF_SF ||

[0035] L coarse =||I course -I SF ||

[0036] L fine =||I fine -I SF ||

[0037] Where L represents the total loss, L STF To remove branch loss for shadows, L FTF For the identity mapping branch loss, For rough loss, To refine the loss, λ1~λ4 are the first~fourth hyperparameters, ||·||1 represents the L1 distance, and I RES_S Represents the image with the shaded area removed, I RES_SF For the mapped image, I course To initially remove shadows from the image, I fine To obtain a clear image after removing shadows, I SF For a true shadow-free image, I SF_S and I SF_SFThese are the shadowed and unshadowed areas of a real, shadowless image, respectively.

[0038] The beneficial effects of this invention are as follows:

[0039] This invention proposes a dual-branch network method for image shadow removal. Addressing the task conflicts and computational resource waste inherent in previous shadow removal methods, this method employs a strategy of using different networks to process shadowed and unshadowed regions separately. For shadowed regions such as… Figure 2 This paper proposes an image shadow removal method based on an excitation attenuation model, addressing the nonlinear response of photoelectric sensors during color acquisition, within the SRNet_STF branch. The excitation attenuation model divides the image into two components: relative intensity and attenuation factor. Relative intensity is related to the reflected light intensity of the scene, while the attenuation factor is related to the material of the object. Since relative intensity and attenuation factor cannot be directly separated from the image, a dual encoder structure is used to extract features of different components. A dual decoder is then used to predict the two components of the image, with the decoder output constrained by corresponding physical equations. The method adaptively learns the complex lighting and color mapping relationship to fit the nonlinear transformation from shadow to non-shadow, thereby achieving accurate processing of shadow areas. In non-shadow areas, such as… Figure 2 The SRNet_FTF branch utilizes an identity mapping module and a lightweight convolution module for low-cost computation, effectively saving computational resources. Considering that non-shaded regions contain richer background color information, a novel normalization strategy is proposed for color restoration to further enhance the information flow from non-shaded to shaded regions and make them more coordinated. The implementation of this strategy will further optimize the overall performance of the dual-branch network in the shadow removal task and improve the image shadow removal effect. Qualitative and quantitative experimental results show that the network of this invention, based on the illumination model and response decay model, has good effectiveness in shadow removal, exhibiting excellent performance under various natural lighting scenes and evaluation metrics, and can flexibly calculate pixel values ​​under different illumination conditions.

[0040] Given that shadowed and unshadowed areas in a shadowed image should have similar texture and color distribution, and that the unshadowed portion contains important prior information, this invention proposes a color restoration module. This module aims to transfer color features from unshadowed areas to shadowed areas to mitigate color shift issues in shadow removal results. Attached Figure Description

[0041] Figure 1 This is a flowchart of the image shadow removal method based on a dual-branch network in this invention;

[0042] Figure 2 This is a framework diagram of the dual-branch network structure in this invention;

[0043] Figure 3 This is a visual comparison diagram of the present invention on the ISTD+ test set;

[0044] Figure 4 This is a graph showing the quantitative comparison results of the present invention on the ISTD+ test set. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] This invention proposes an image shadow removal method based on a dual-branch network. The method includes the following steps: acquiring the shadow image to be processed and its corresponding mask image, and inputting them into a trained dual-branch network for processing to obtain a clear image after shadow removal.

[0047] The dual-branch network of this invention simplifies the Lambert lighting model in the shadow removal branch, using relative light intensity to describe the illumination relationship between shadowed and non-shadowed areas. Based on the characteristics of the nonlinear response of the photoelectric sensor, a reciprocal form of the attenuation function is selected. It uses only one parameter (attenuation factor) to describe the excitation attenuation trend. The attenuation factor is only related to the object's material and is independent of the scene's illumination. By modeling pixel values ​​as the integral of the attenuation factor over the reflected light intensity, a response attenuation model can be derived. This model divides the image into a material component (attenuation factor) and an illumination component (relative light intensity). A dual encoder-dual decoder structure is used to predict these two components. These two components are then combined into a coarse shadow-removed image through corresponding physical constraints. The coarse shadow-removed image and the mapped image are then concatenated and fed into the color restoration module. The mean and variance of pixel values ​​in the non-shadowed areas are calculated to aid in color restoration of the shadowed areas.

[0048] like Figure 1 As shown, the training process of a dual-branch network includes:

[0049] S1: Obtain the shadow image to be processed and its corresponding mask image, preprocess the shadow image and mask image to obtain the shadow area image and the non-shadow area image.

[0050] Both the shadow image and the mask image are scaled to 256×256. Specifically, after scaling the shadow image and the mask image to 286×286, they are randomly cropped to obtain a cropped image of size 256×256. The cropped image is then rotated and flipped with a 50% probability to obtain the pre-processed shadow image and mask image. Next, the shadow image and the mask image are multiplied pixel by pixel to obtain the shadow area image. The mask image is then inverted and multiplied pixel by pixel with the shadow image to obtain the non-shadow area image.

[0051] S2: The shadow region image and the non-shadow region image are fed into the shadow removal branch network and the identity mapping branch network respectively for processing to obtain the shadow-removed region image and the mapped image.

[0052] The image of the shaded region and the image of the unshaded region are fed into the shadow removal branch network for processing to obtain the image of the removed shaded region. For example... Figure 2 As shown, the shadow removal branch network, namely the SRNet_STF branch, includes a relative intensity estimation network and an attenuation factor estimation network. The shadow region image is input into the relative intensity estimation network to obtain the relative intensity component, which is related to the reflected light intensity of the scene. The shadow region image is then fed into the attenuation factor estimation network to obtain the attenuation factor component, which is related to the material of the object. The two components of the image are constrained by corresponding physical equations for reconstruction, as shown in Equation 17. The identity mapping branch network, namely the SRNet_FTF branch, reconstructs the image using a more lightweight convolutional SRNet.

[0053] In this invention, both the relative light intensity estimation network and the attenuation factor estimation network adopt a symmetric encoder-decoder structure, but their task objectives and gradient update paths are different, generating relative light intensity and attenuation factor respectively; wherein:

[0054] The encoders are all 8-layer convolutional neural networks with 4 kernels, a stride of 2, and padding of 1, halving the width and height of the image after each convolution. The first convolutional layer has 4 input channels and 64 output channels; the second through seventh convolutional layers have the following input channels: 64, 128, 256, 512, 512, 512, and 128, 256, 512, 512, 512, 512, respectively, and the following output channels: 512, 128, 256, 512, 512, 512, 512. Furthermore, LeakyReLU layers and BatchNorm layers are added before and after the second through seventh convolutional layers, respectively; the eighth convolutional layer has 512 input channels and 512 output channels, and a LeakyReLU layer is added before it.

[0055] The decoder consists of eight deconvolutional layers. The first deconvolutional layer has twice the number of input channels as the last convolutional layer of the encoder, enabling the decoder to decode features encoded by both encoders. Each deconvolutional layer is configured with a kernel size of 4, a stride of 2, and padding of 1, doubling the image width and height after each convolution. The input channels of the first through seventh convolutional layers are 1024, 2048, 2048, 2048, 1024, and 512, respectively, and the output channels are 1024, 1024, and 512, respectively.

[0056] 1024, 1024, 512, 256, 128. In addition, ReLU layers and BatchNorm layers are added before and after the first to seventh convolutional layers respectively; the eighth convolutional layer has 256 input channels and 3 output channels, and ReLU layers and sigmoid layers are added before and after the eighth convolutional layer respectively.

[0057] The encoder input size is 256×256. During downsampling, the intermediate feature size is halved after each convolution. After eight convolutions, the size changes from 256×256 to 1×1. During upsampling, the intermediate feature size doubles after each deconvolution. By using skip connections, the feature maps from the downsampling and upsampling processes are concatenated, allowing low-level and high-level features to be integrated, which helps to better consider the global and local features of the image.

[0058] Next, a response attenuation model is established based on the relationship between pixel values ​​and illumination. This model is then fitted using relative light intensity and an attenuation factor to obtain the image of the shaded region. The process of establishing the response attenuation model includes:

[0059] In the real world, most electronic cameras exhibit nonlinear response characteristics during photoelectric conversion. Specifically, as light intensity increases, the change in pixel value is not directly proportional to the change in light intensity, but rather exhibits a nonlinear relationship.

[0060] Under normal circumstances, the reflectivity of an object is relatively unaffected by light intensity and can be approximated as linear. For example, equation (1):

[0061] I R =I L ×R (1)

[0062] In the formula, I R I represents the intensity of reflected light. L R is the incident light intensity; R is the reflectivity, which is independent of the light intensity.

[0063] The intensity of reflected light is proportional to the incident light, but the pixel value is not proportional to the reflected light. Therefore, the change in pixel value is smaller than the change in incident light.

[0064] The relationship between pixel value and reflected light intensity is modeled as an integral relationship, as shown in equation (2):

[0065] P=∫0 x f(x)dx (2)

[0066] In the formula, P represents the pixel value, x represents the current reflected light intensity, dx represents the small change in reflected light intensity, and f(x) is the response function, which represents the pixel value increment caused by a unit increase in reflected light intensity when the reflected light intensity is x.

[0067] Due to the nonlinear response characteristics of photoelectric sensors, the response value gradually decreases with increasing reflected light intensity, eventually approaching zero. This invention defines this phenomenon as response decay. Assuming that the illumination intensity is a continuous variable, the relationship between reflected light intensity and the response value can be represented by a response curve.

[0068] Assuming complete darkness, the response value is 1. This means that without attenuation, one unit of illumination intensity causes an increment of 1 in the pixel value, as shown in equation (3):

[0069] f(0)=1 (3)

[0070] Because objects made of different materials have different light reflection characteristics, their response curves are usually different as well. When the same incident light shines on different objects, the spectral characteristics of the reflected light will differ. Furthermore, increasing light intensity does not lead to a decrease in pixel value; therefore, the lower limit of the response curve is 0.

[0071] In order to meet the initial value and lower bound of the response curve defined in equation (3), this invention uses a reciprocal function to model the decay of the response value, as shown in (4):

[0072]

[0073] In the formula, f(x) is the response value corresponding to the intensity of reflected light x; a is the attenuation factor, which is related to the material of the object.

[0074] The relationship between the attenuation factor and the attenuation curve is that the response value decreases as the light intensity increases, with a lower limit of 0. When the light intensity is the same, the larger the attenuation factor, the smaller the response value.

[0075] From equations (2) and (4), it can be seen that the relationship between reflected light intensity and pixel value is as shown in equation (5):

[0076]

[0077] In the formula: p is the pixel value (brightness); x is the reflected light intensity; a is the attenuation factor.

[0078] This invention proposes a simplified lighting model that considers the incident light intensity in the shadowless region as the superposition of ambient light intensity and direct light intensity, so that the reflected light intensity can be expressed as a linear combination of ambient light intensity and direct light intensity, as shown in equation (6):

[0079] I r =R a *I a +R d *I d (6)

[0080] In the formula, I r Indicates the intensity of reflected light; R a R d These represent the reflectance of ambient light and direct light, respectively; I a I d These represent the light intensity of ambient light and direct sunlight, respectively.

[0081] Assuming the reflectivity of the object remains constant, and that the reflectivity of direct light is equal to that of diffuse reflection, it can be expressed as equation (7):

[0082] I r =R(I a +I d (7)

[0083] Assuming that the shadow area is only illuminated by ambient light, and the non-shadow area is illuminated by both ambient light and direct light, the pixel values ​​of the shadow area can be obtained according to formula (5) as shown in formula (8):

[0084]

[0085] In the formula, P s Indicates the pixel value of the shadow area; I a This represents the ambient light intensity; 'a' is the attenuation factor, which is related to the material of the object; and 'R' is the reflectivity of the object.

[0086] The pixel values ​​for the unshaded areas are shown in formula (9):

[0087]

[0088] In the formula, P sf I represents the pixel value of the area without shadow. a I represents ambient light intensity. d Let represent the intensity of direct light, 'a' be the attenuation factor, and 'R' be the reflectivity of the object.

[0089] The incident light intensity in the shadowless region can be expressed as the sum of the ambient light intensity and the direct light intensity, as shown in formula (10):

[0090] I sf =Ia +I d =kI a ,k>1 (10)

[0091] In the formula, k represents the ratio of incident light intensity in the unshaded area to that in the shaded area.

[0092] According to equation (10), the formula for converting shadow to unshadowed pixel values ​​is derived as shown in equations (11)-(12):

[0093]

[0094] Converting both sides of equation (11) into exponential form, we obtain equation (12):

[0095]

[0096] From equations (8) and (9), we can obtain:

[0097]

[0098] Substituting equation (13) into equation (12), we get:

[0099]

[0100] Therefore, the transformation from shadow to unshadowed pixel values ​​is shown in equation (16):

[0101]

[0102] In the formula, P sf P represents the pixel value without shadows. s is the shadow pixel value; k is the ratio of incident light intensity (relative light intensity) between the shadowless area and the shadowed area; a is the attenuation factor corresponding to the object.

[0103] As shown in equation (16), the conversion from shadow to shadowless pixel value requires two parameters: the object's attenuation factor a and the relative light intensity k. The object's attenuation factor a is independent of illumination and depends on the object's differential absorption of light of different wavelengths. The relative light intensity k is related to the scene's illumination conditions; the larger k is, the greater the proportion of direct light intensity to total illumination intensity.

[0104] Based on equations (8)-(9), the pixel values ​​of any region (shadowed region and unshadowed region) can be summarized, i.e., the response attenuation model is shown in equation (17):

[0105]

[0106] In the formula, p represents the pixel value of the image after fitting the model, a is the attenuation factor, which is related to the material; k is the relative light intensity, which is related to the lighting conditions. In the shadowless area, k is the ratio of total light intensity to ambient light intensity; in the shadowed area, k = 1; in the penumbra, k varies uniformly; R is the reflectivity, which is related to the material.

[0107] The relative light intensity estimation network and the attenuation factor estimation network adopt the same symmetric encoder-decoder structure. Both the encoder and decoder are 8-layer convolutional neural networks with 4 convolutional kernels and a stride of 2. The input has 4 channels (3 channels of the shadow image + 1 channel of the shadow mask), and the output has 3 channels, representing that the excitation attenuation model considers the R, G, and B channels respectively.

[0108] The images of the shaded and unshaded regions are fed into the identity mapping branch network for processing to obtain the mapped image. The identity mapping branch network consists of an encoder and a decoder, wherein:

[0109] The encoder is based on the SR-Net architecture and consists of 6 convolutional layers. The first to fourth convolutional layers are followed by an instance normalization module and a LeakyReLU activation function, respectively. The fifth convolutional layer is followed by a BatchNorm layer and a LeakyReLU activation function. The parameters of all convolutional layers are set to a kernel size of 4×4, a stride of 2, and padding of 1. The number of input channels is 4, 64, 128, 256, 512, and 512, respectively, and the number of output channels is 64, 128, 256, 512, 512, and 512, respectively.

[0110] The decoder consists of 6 deconvolutional layers. LeakyReLU activation layers and BatchNorm layers are added before and after the first deconvolutional layer, respectively. LeakyReLU activation layers and instance normalization modules are added before and after the second to fifth deconvolutional layers, respectively. LeakyReLU activation layers and tanh activation layers are added before and after the sixth deconvolutional layer, respectively. The parameters of all deconvolutional layers are set to kernel size 4×4, stride 2, and padding 1. The number of input channels of the 6 deconvolutional layers are 512, 1024, 1024, 512, 256, and 128, respectively, and the number of output channels are 512, 512, 256, 128, 64, and 3, respectively.

[0111] S3: Add the image of the removed shadow area and the mapped image pixel by pixel to obtain the preliminary image with the shadow removed.

[0112] S4: Send the initially shadow-removed image to the color restoration module. Figure 2 The RefineNet module in the image is used for color restoration to obtain a clear image after removing shadows.

[0113] In the color restoration module, the mean and variance of pixels in the non-shadow and shadow areas are calculated first. Specifically, as shown in equations (18)-(20):

[0114] n total =n s +n n (18)

[0115]

[0116] In the formula n total This indicates that an image contains a total of pixels, n. s n represents the number of pixels in the shadow area. n μ represents pixels in the non-shaded area. s and μ n These represent the mean values ​​of pixels in the shaded and unshaded areas, respectively. and These represent the pixel variance in the non-shaded area and the pixel variance in the shaded area, respectively. total This represents the average pixel value of the entire image. It represents the variance of the entire image pixels.

[0117] Since the RGB channel values ​​in the shaded area are usually much lower than those in the unshaded area, we can observe from the above formula that... μ s μ n compared to, and μ total There is a larger offset. Therefore, the mean and variance of the shadowed areas should be made closer to those of the unshaded areas to make the overall image more harmonious.

[0118] The pixel mean and variance of the shaded region can be adjusted using the pixel mean and normalization mapping of the non-shaded region. Specifically, the normalized value of the shaded pixel p located in (h, w, c) can be calculated using the following equations (21)-(23):

[0119]

[0120] In the formula p h,w,c and These are the initial and normalized pixel values ​​of a pixel with height h, width w, and channel c (c∈R, G, B), respectively. ∈ is a minimum value to prevent division by zero. This represents the average value of pixels in the shadow region of channel c. This represents the variance of pixels in the shadow region of the c channel. This represents the variance of pixels in the non-shaded area of ​​the c channel. Num represents the mean value of pixels in the non-shaded area of ​​the c channel.Region This represents the sum of the pixel values ​​of all pixels within the Region. The Region can be a shaded area or a non-shaded area.

[0121] S5: Calculate the total loss of the dual-branch network and adjust the network parameters according to the total loss to obtain the trained dual-branch network.

[0122] The formula for calculating the total loss of a two-branch network is:

[0123] L=λ1L STF +λ2L FTF +λ3L coarse +λ4L fine

[0124] L STF =||I RES_S -I SF_S ||

[0125] L FTF =||I RES_SF -I SF_SF ||

[0126] L coarse =||I course -I SF ||

[0127] L fine =||I fine -I SF ||

[0128] Where L represents the total loss, L STF To remove branch loss for shadows, L FTF For the identity mapping branch loss, For rough loss, To refine the loss, λ1~λ4 are the first~fourth hyperparameters, ||·||1 represents the L1 distance, and I RES_S Represents the image with the shaded area removed, I RES_SF For the mapped image, I course To initially remove shadows from the image, I fine To obtain a clear image after removing shadows, I SF For a true shadow-free image, I SF_S and I SF_SF These are the shadowed and unshadowed areas of a real, shadowless image, respectively.

[0129] The network parameters are adjusted according to the loss function. Training stops when the loss function converges or reaches the maximum preset number of iterations, and the network parameters are maintained, resulting in a trained dual-branch network. The shadow image to be processed, along with the image mask, is input into the trained dual-branch network for processing, resulting in a clear image after shadow removal.

[0130] Evaluation of the present invention:

[0131] To quantify and compare the effectiveness of this invention, it was compared with algorithms such as DSC, AUTO-Exposure, SG-ShadowNet, and BMNet on two publicly available real-world datasets (SRD and ISTD). Three objective evaluation metrics were selected: root mean square error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM) in the LAB color space. Given a shadow removal result and a corresponding shadow-free image, RMSE measures their average pixel error, PSNR measures their average pixel similarity, and SSIM measures their structural similarity. A lower RMSE value indicates a better generated result, while higher PSNR and SSIM values ​​indicate a better generated result.

[0132] Figure 3 This paper presents a visual comparison of the processing results of the present invention and five other shadow removal methods on real-world shadowed images. From left to right, the first column shows the shadowed image to be processed, followed by different shadow removal methods: Fu et al., DC-ShadowNet, Zhu et al., BMNet, ShadowDiffusion, and the method proposed in this invention, as well as a real shadowless image. The other methods produce noticeable artifacts at the edges of shadows or recover incorrect color and brightness information, while the dual-branch network of the present invention is able to reconstruct a higher-quality shadowless image.

[0133] Figure 4 This paper presents a quantitative comparison of the image shadow removal method (DSR-Net) of this invention with nine other shadow removal methods on the real shadow removal dataset SRD, including non-deep learning methods by Guo et al. and deep learning-based methods DHAN, DC-ShadowNet, Fu et al., Zhu et al., BMNet, ShadowDiffusion, and ShadowFormer. Each image with shadow removal is divided into shadow regions (S), non-shadow regions (NS), and the entire image region (ALL), and the shadow removal performance of different methods is measured using the three quantitative evaluation metrics mentioned above. Figure 4 The results show that the present invention has achieved comprehensive and excellent shadow removal results in terms of PSNR, SSIM and RMSE.

[0134] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image shadow removal method based on a dual-branch network, characterized in that, include: The shadow image to be processed and its corresponding mask image are obtained and input into a trained dual-branch network for processing to obtain a clear image after removing the shadow; The training process of a dual-branch network includes: S1: Obtain the shadow image to be processed and its corresponding mask image, preprocess the shadow image and mask image to obtain the shadow area image and the non-shadow area image; S2: The shadow region image and the non-shadow region image are fed into the shadow removal branch network and the identity mapping branch network respectively for processing to obtain the shadow-removed region image and the mapped image; S3: Add the image of the removed shadow area and the mapped image pixel by pixel to obtain the preliminary image with the shadow removed; S4: The image with the initial shadow removal is sent to the color restoration module for color restoration to obtain a clear image after the shadow removal; S5: Calculate the total loss of the dual-branch network and adjust the network parameters according to the total loss to obtain the trained dual-branch network.

2. The image shadow removal method based on a dual-branch network according to claim 1, characterized in that, The preprocessing of the shadow image and the mask image includes: Both the shadow image and the mask image are scaled to 256×256. The shadow image and the mask image are multiplied pixel by pixel to obtain the shadow area image. The mask image is then inverted and multiplied pixel by pixel with the shadow image to obtain the non-shadow area image.

3. The image shadow removal method based on a dual-branch network according to claim 1, characterized in that, The process of feeding the image of the shadowed region into the shadow removal branch network for processing includes: the shadow removal branch network includes a relative light intensity estimation network and an attenuation factor estimation network; The image of the shadowed area is input into a relative light intensity estimation network for processing to obtain the relative light intensity. The image of the shaded region is input into the attenuation factor estimation network for processing to obtain the attenuation factor; A response attenuation model is established based on the relationship between pixel values ​​and illumination. The response attenuation model is then fitted based on relative light intensity and attenuation factor to obtain the image of the shaded area.

4. The image shadow removal method based on a dual-branch network according to claim 3, characterized in that, Both the relative light intensity estimation network and the attenuation factor estimation network adopt a symmetric encoder-decoder structure; The encoder consists of 8 convolutional layers with 4 kernels, a stride of 2, and padding of 1. The first convolutional layer has 4 input channels and 64 output channels. The second to seventh convolutional layers have the following input channel counts: 64, 128, 256, 512, 512, 512, and 128, 256, 512, 512, 512, 512, respectively. LeakyReLU and BatchNorm layers are added before and after the second to seventh convolutional layers, respectively. The eighth convolutional layer has 512 input channels and 512 output channels, and a LeakyReLU layer is added before it. The decoder consists of 8 deconvolutional layers. The number of input channels in the first deconvolutional layer is twice the number of output channels in the last convolutional layer of the encoder. The parameters of each deconvolutional layer are set to a kernel size of 4, a stride of 2, and padding of 1. The number of input channels in the 1st to 7th convolutional layers are 1024, 2048, 2048, 2048, 2048, 1024, and 512, respectively, and the number of output channels are 1024, 1024, 1024, 1024, 512, 256, and 128, respectively. ReLU layers and BatchNorm layers are added before and after the 1st to 7th convolutional layers, respectively. The 8th convolutional layer has 256 input channels and 3 output channels, and ReLU layers and sigmoid layers are added before and after the 8th convolutional layer, respectively.

5. The image shadow removal method based on a dual-branch network according to claim 3, characterized in that, The response decay model is expressed as: Where p represents the pixel value of the image after fitting the model, P s represents the shadow pixel value, k represents the ratio of incident light intensity between the shadowless area and the shadowed area, a represents the attenuation factor corresponding to the object, and R represents the reflectivity.

6. The image shadow removal method based on a dual-branch network according to claim 1, characterized in that, The identity mapping branch network consists of an encoder and a decoder; The encoder is based on the SR-Net architecture and consists of 6 convolutional layers. The first to fourth convolutional layers are followed by an instance normalization module and a LeakyReLU activation function, respectively. The fifth convolutional layer is followed by a BatchNorm layer and a LeakyReLU activation function. The parameters of all convolutional layers are set to a kernel size of 4×4, a stride of 2, and padding of 1. The number of input channels is 4, 64, 128, 256, 512, and 512, respectively, and the number of output channels is 64, 128, 256, 512, 512, and 512, respectively. The decoder consists of 6 deconvolutional layers. LeakyReLU activation layers and BatchNorm layers are added before and after the first deconvolutional layer, respectively. LeakyReLU activation layers and instance normalization modules are added before and after the second to fifth deconvolutional layers, respectively. LeakyReLU activation layers and tanh activation layers are added before and after the sixth deconvolutional layer, respectively. The parameters of all deconvolutional layers are set to kernel size 4×4, stride 2, and padding 1. The number of input channels of the 6 deconvolutional layers are 512, 1024, 1024, 512, 256, and 128, respectively, and the number of output channels are 512, 512, 256, 128, 64, and 3, respectively.

7. The image shadow removal method based on a dual-branch network according to claim 1, characterized in that, The process of sending the initially shadow-removed image to the color restoration module for color restoration is represented as follows: in, p represents the normalized pixel value. h,w,c This represents the initial pixel values ​​of the image after initial shadow removal; This represents the average value of the pixels in the shaded area. The variance of the pixels in the shadow area. Represents the variance of pixels in the non-shaded area. This represents the average value of pixels in the non-shaded area.

8. The image shadow removal method based on a dual-branch network according to claim 1, characterized in that, The formula for calculating the total loss of a two-branch network is: L=λ1L STF +λ2L FTF +λ3L coarse +λ4L fine L STF =||I RES_S -I SF_S || L FTF =||I RES_SF -I SF_SF || L coarse =||I course -I SF || L fine =||I fine -I SF || Where L represents the total loss, L STF To remove branch loss for shadows, L FTF For the identity mapping branch loss, For rough loss, To refine the loss, λ1~λ4 are the first~fourth hyperparameters, ||·||1 represents the L1 distance, and I RES_S Represents the image with the shadow removed, I RES_SF For the mapped image, I course To initially remove shadows from the image, I fine To obtain a clear image after removing shadows, I SF For a true shadow-free image, I SF_S and I SF_SF These are the shadowed and unshadowed areas of a real, shadowless image, respectively.

Images