An image shadow removal method based on illumination compensation
By decomposing the shadow removal process into two stages—direct light compensation and ambient light compensation—and using an encoder-decoder network model to process direct light and ambient light, the problems of brightness distortion and texture inconsistency in existing methods are solved. The resulting shadow-free images have a more natural visual effect and retain more complete details.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-03-11
- Publication Date
- 2026-06-19
AI Technical Summary
Existing deep learning methods fail to adequately consider the independence of direct light and ambient light during shadow removal, leading to problems such as brightness distortion, color distortion, and texture inconsistency, and lacking explicit modeling of the physical processes of lighting.
A light compensation-based approach is adopted, which decomposes the shadow removal process into two physical stages: direct light compensation and ambient light compensation. Direct light and ambient light are processed by direct light compensation network and ambient light compensation network, respectively. Local ambient light is used as guiding information to construct an encoder-decoder network model for training.
It improves the interpretability and credibility of the model, and the generated shadowless images are more visually natural and retain more complete details, especially in areas with complex textures and color transitions.
Smart Images
Figure CN122243774A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and in particular to image shadow removal, specifically relating to an image shadow removal method based on illumination compensation. Background Technology
[0002] Shadows are dark areas formed when light is blocked by objects, preventing even illumination of a surface. They are an unavoidable visual phenomenon in natural scenes. Shadows are a result of changes in illumination and often interfere with the texture and color information of an image, thus affecting image quality and the accuracy of subsequent visual tasks. For example, in tasks such as object detection, object tracking, and image segmentation, images without shadow removal may lead to algorithmic misjudgments or performance degradation.
[0003] Traditional shadow removal methods primarily rely on various prior image knowledge. For example, gradient consistency-based methods assume that gradient changes at shadow boundaries are continuous; color constancy-based methods utilize the variation patterns of different color channels under shadows; and texture similarity-based methods assume that the texture patterns of shadow regions are similar to those of non-shadow regions. While these methods are effective in specific scenarios, their performance is highly dependent on hand-designed rules, resulting in limited generalization ability and difficulty in handling complex and varied real-world scenarios.
[0004] In recent years, deep learning techniques, especially convolutional neural networks, have greatly advanced shadow removal technology. These methods typically model the shadow removal problem as an end-to-end image-to-image translation task, directly learning a mapping function from a shadowed image to a shadowless image. For example, some studies utilize GAN frameworks to improve the realism of generated images; others introduce attention mechanisms to focus on shadow regions. Although these methods have achieved significant progress in quantitative metrics, they still have significant shortcomings in qualitative evaluation, especially in terms of visual realism.
[0005] The core limitation of most existing deep learning-based shadow removal methods lies in their failure to adequately consider the combined effects of direct light and the environment during shadow formation. In shadow regions, direct light is blocked by occlusions, thus its contribution approaches zero, while ambient light becomes the dominant factor. However, existing methods generally treat the shadow image as an indivisible whole, attempting to process all lighting information simultaneously with a single, complex network. They ignore the independence of direct light and ambient light, leading to problems such as brightness distortion, color distortion, and texture inconsistencies when the network "guesses" shadowless images.
[0006] The root cause of these problems is that existing methods lack explicit modeling of the physical processes of illumination, cannot explain their internal decision-making logic, and cannot provide reliable prior knowledge to guide the recovery process. Summary of the Invention
[0007] To address the aforementioned problems in the prior art, this invention employs an image shadow removal method based on illumination compensation, comprising the following steps: acquiring the image to be processed; inputting the image to be processed into a trained image shadow removal model to obtain a shadow-free image; the image shadow removal model includes: a direct light compensation network and an ambient light compensation network; the training process of the image shadow removal model includes:
[0008] S1. Obtain the training dataset. The training dataset contains multiple image triplets. Each image triplet includes: the image to be removed from the shadow. and its shadowless realistic image and shadow mask image The image to be shaded in the image triplet. and its shadowless realistic image Perform grayscale transformation to obtain the image triplet image of the shadow to be removed. and its shadowless realistic image grayscale image , ;
[0009] S2, Image to be removed of shadows grayscale image and its shadow mask image The image is input into a direct light compensation network to obtain the image of the shadow to be removed. Direct light compensation coefficient matrix ;
[0010] S3, using the direct light compensation coefficient matrix Removing shadows from images Perform direct light compensation to obtain the image of the shadow to be removed. Direct light compensation results ;
[0011] S4. Using the direct light compensation coefficient matrix and the image to be removed of shadows Calculate the image to be removed of shadows Ambient light ; Calculate ambient light Compensation results with direct light The difference between them yields ambient light difference guidance information.
[0012] S5. Image to be removed of shadows Direct light compensation results Shadow mask image and guidance information Input the ambient light compensation network to obtain the image of the shadow to be removed. Ambient light compensation coefficient matrix ;
[0013] S6. Utilizing the ambient light compensation coefficient matrix Compensation results for direct light Ambient light compensation is performed to obtain the image of the shadow to be removed. Shadowless image ;
[0014] S7. Based on the calculated loss function value, update the image shadow removal model. When the loss function value is minimized, the trained image shadow removal model is obtained.
[0015] The beneficial effects of this invention are:
[0016] (1) This invention decomposes the shadow removal process into two distinct physical stages: “direct light compensation” and “ambient light compensation”, making the working mechanism of the entire network transparent and in line with physical laws, which greatly improves the interpretability and credibility of the model.
[0017] (2) This invention utilizes the local ambient light extracted from the non-shaded area and uses the difference between the ambient light and direct light compensation results as guiding information to more clearly guide the network to learn how to compensate for ambient light, thereby improving the model's ability to adapt to different scenarios and recover the color difference caused by ambient light.
[0018] (3) Thanks to the explicit modeling of the physical lighting model and the two-stage compensation strategy, the shadowless images generated by this invention are more natural and realistic in appearance, with more complete detail retention, especially in complex texture and color transition areas. Attached Figure Description
[0019] Figure 1 This is a flowchart of an image shadow removal method based on illumination compensation according to the present invention;
[0020] Figure 2 This is a framework diagram of the image shadow removal model of the present invention;
[0021] Figure 3 This is a schematic diagram of the channel attention layer of the present invention;
[0022] Figure 4 This is a schematic diagram of the structure of adjacent attention layers in this invention;
[0023] Figure 5 This is a visualization comparing the performance of this invention with other methods on the ISTD+ test set. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] Specifically, existing methods generally treat shadow images as an indivisible whole, attempting to use a single, complex network to simultaneously process direct light and ambient light information. The degradation problem in shadow regions mainly includes reduced brightness, color distortion, and texture degradation. Modeling the shadow effect using spatial coordinates... and wavelength Observed pixel intensity at It can be represented as illumination. With albedo The product of: In a single light source scene, pixels in non-shadow areas Light can be decomposed into direct light components. and ambient light component : ,in This represents the spatial variation attenuation coefficient. Therefore, the pixel intensity in the non-shaded area... It can be represented as: Conversely, when an object blocks the main light source, creating a shadow area, the pixel intensity of the shadow area... It can be represented as The shaded areas are affected by both the lack of direct light and the attenuation of ambient light.
[0026] Based on the above analysis, embodiments of the present invention provide an image shadow removal method based on illumination compensation, such as... Figure 1-4 As shown, the process includes the following steps: acquiring the image to be processed, inputting the image to be processed into a trained image shadow removal model to obtain a shadow-free image; the image shadow removal model includes: a direct light compensation network and an ambient light compensation network; the training process of the image shadow removal model includes:
[0027] S1. Obtain the training dataset. The training dataset contains multiple image triplets. Each image triplet includes: the image to be removed from the shadow. and its shadowless realistic image and shadow mask image The image to be shaded in the image triplet. and its shadowless realistic image Perform a standard grayscale transformation to obtain the image triplet of the image to be shaded. and its shadowless realistic image grayscale image , ;
[0028] S2, Image to be removed of shadows grayscale image and its shadow mask image Input the direct light compensation network to obtain the image of the shadow to be removed. Direct light compensation coefficient matrix ;
[0029] The direct light compensation network and the ambient light compensation network are encoder-decoder structures, which include, in sequence, an encoder, an adjacent attention layer, and a decoder.
[0030] The encoder consists of a three-layer network structure, with the first two layers being channel attention layers and the last layer being an adjacent attention layer, with downsampling layers added between each layer;
[0031] The input channels of the three-layer network are the same as the output channels in sequence; the parameters of each downsampling layer are set as follows: kernel size 4, stride 2, padding 1; the number of input channels of the four downsampling layers are InputChannel, 32, 64, 128 in sequence, and the number of output channels are 32, 64, 128, 256 in sequence.
[0032] An adjacent attention layer is added between the encoder and decoder; the number of input and output channels is 256.
[0033] The decoder consists of a three-layer network structure: the first layer is the adjacent attention layer, and the last two layers are the channel attention layers. An upsampling layer is added between each layer.
[0034] The input channels of the three-layer network are the same as the output channels in sequence; the parameters of each upsampling layer are set as follows: kernel size 4, stride 2, padding 1; the number of input channels of the four upsampling layers are 256, 128, 64, and 32 in sequence, and the number of output channels are 128, 64, and 32 in sequence, and InputChannel.
[0035] The direct light compensation network has 2 input channels and the ambient light compensation network has 7 input channels.
[0036] Direct light compensation network for removing shadows from images grayscale image and its shadow mask image The processing includes: the image to be removed of shadows grayscale image and its shadow mask image Channel overlay is performed, the result of channel overlay is input into the encoder, the encoder output is input into the adjacent attention layer, the output of the adjacent attention layer is input into the decoder, and the decoder output is compared with the image to be removed from the shadow. grayscale image Adding them together yields the direct light compensation coefficient matrix. .
[0037] The channel attention layer comprises, in sequence, a first LayerNorm layer, a channel attention module, a second LayerNorm layer, and a fully connected layer. The channel attention layer processes its input data by: inputting the input data into the first LayerNorm layer; inputting the output of the first LayerNorm layer into the channel attention module; fusing the output of the channel attention module with the input data to obtain a fusion result; inputting the fusion result into the second LayerNorm layer; inputting the output of the second LayerNorm layer into the fully connected layer; and fusing the output of the fully connected layer with the fusion result to obtain the output data.
[0038] The adjacent attention layer sequentially comprises: a first LayerNorm layer, an adjacent attention module, a channel attention module, a second LayerNorm layer, and a fully connected layer. The adjacent attention layer processes its input data by: inputting the input data into the first LayerNorm layer; inputting the output of the first LayerNorm layer into the adjacent attention module; inputting the output of the adjacent attention module into the channel attention module; fusing the output of the channel attention module with the input data to obtain a fusion result; inputting the fusion result into the second LayerNorm layer; inputting the output of the second LayerNorm layer into the fully connected layer; and fusing the output of the fully connected layer with the fusion result to obtain the output data.
[0039] S3, using the direct light compensation coefficient matrix Removing shadows from images Perform direct light compensation to obtain the image of the shadow to be removed. Direct light compensation results ;
[0040] Direct light compensation results The calculation formula is as follows: .
[0041] S4. Using the direct light compensation coefficient matrix and the image to be removed of shadows Calculate the image to be removed of shadows Ambient light ; Calculate ambient light Compensation results with direct light The difference between them yields ambient light difference guidance information. ;
[0042] Calculate the image to be removed of shadows Ambient light include:
[0043] S41, Compensation coefficient matrix for direct light The image is truncated within the range (0,1), and the truncated direct light compensation coefficient matrix is then subjected to max pooling to obtain the dilated mask image. ;
[0044] Direct light compensation coefficient matrix Truncation within the range (0,1) includes: for the direct light compensation coefficient matrix For any element x in the set, if x is less than or equal to 0, then let x = 0; if x is greater than 0 and less than 1, then let x remain unchanged; if x is greater than or equal to 1, then let x = 1.
[0045] Max pooling was used with parameters set to kernel size 5, stride 1, and padding 2.
[0046] S42. Apply the mask image Subtract shadow mask image The prior mask image of the local non-shaded area is obtained. ;
[0047] S43. Use the prior mask image of the local non-shaded area. Multiply by the image to be removed This yields an image of the shaded adjacent region;
[0048] S44. Calculate the average value of all pixels in the image of the adjacent shadow region, and multiply the average value by the shadow mask image. Ambient light .
[0049] S5. Image to be removed of shadows Direct light compensation results Shadow mask image and guidance information Input the ambient light compensation network to obtain the image of the shadow to be removed. Ambient light compensation coefficient matrix ;
[0050] Image with shadows to be removed Direct light compensation results Shadow mask image and guidance information Channel overlay is performed before inputting into the ambient light compensation network.
[0051] S6. Utilizing the ambient light compensation coefficient matrix Compensation results for direct light Ambient light compensation is performed to obtain the image of the shadow to be removed. Shadowless image ;
[0052] Image without shadow .
[0053] S7. Based on the calculated loss function value, update the image shadow removal model. When the loss function value is minimized, the trained image shadow removal model is obtained.
[0054] loss function ;in, To recover the loss from direct sunlight, For the recovery loss of ambient light, The hyperparameters of the loss function are 1 and 0.001, respectively;
[0055] Direct light recovery loss is achieved by optimizing the grayscale image of the image to be shadowed after direct light compensation. Grayscale image of a real image without shadows The gap between them prompts the direct light compensation network to correctly compensate for the lack of direct light in the shadow area; and features are extracted from the pre-trained model to measure the perceptual similarity between the two, and the direct light recovery loss is calculated, including:
[0056]
[0057] Ambient light recovery loss After optimizing ambient light compensation Compared to shadowless images, real images The difference between them prompts the ambient light compensation network to adjust the ambient light difference. Under the guidance of [unclear], the influence of ambient light in the shadow area is correctly compensated; and the perceptual similarity between the two is measured by extracting features through a pre-trained model, and the ambient light restoration loss is calculated. include:
[0058]
[0059] in, For a constant 10 -6 , Indicates the L1 distance. Let conv1_2, conv2_2, conv3_2, conv5_2, and conv5_2 represent different layers i in the pre-trained model VGG-19. These represent the weights of different layers in the pre-trained model VGG-19, which are 0.1, 0.1, 1, 1, and 1, respectively. This indicates the input image in the pre-trained model VGG-19. The output results of different layers i.
[0060] During the model training phase, the AdamW optimizer with an initial learning rate of 0.0004 was used, and then gradually reduced to 10 using a cosine annealing strategy. -6 The parameters of the deshading model are updated based on the loss through a backpropagation mechanism.
[0061] This invention aims to explicitly separate and process the two key components of direct light and ambient light by introducing prior knowledge from the physical lighting model. By constructing a dedicated sub-network to estimate the direct light compensation coefficient matrix and the ambient light compensation coefficient matrix, and by using local ambient light as prior information, this invention can effectively suppress brightness and color distortion, significantly improve the visual quality and physical realism of shadow removal results, and enhance the interpretability of the model.
[0062] like Figure 5As shown in Table 1, this invention (Ours) is combined with the following frameworks: Param+M+D (a joint framework consisting of a parameter estimation network, a mask generation network, and a refinement network); Dual Hierarchical Aggregation Network (DHAN); Auto-Exposure Fusion (AEF); Lightness-Guided Network (LG Net); Domain-Classifier Guided Network (DC Net); Mask-Guided Residual Learning Network (MGRLN-Net); Decoupled Multi-Task Network (DMTN); Three-Branch Residual Network (TBRNet); De-Self and Soft Shadows (DeS3); and Shadow Multi-intensity Generation and Diffusion Modeling-Guided Shadow Removal Enhancement Network. This invention compares the shadow removal method with that of SMGDM-SRA (Shadow Diffusion Model) and Omni-directional Shadow Removal (OmniSR). The proposed method demonstrates superior shadow removal performance across various scenarios, effectively controlling color differences within and outside the shadow area and significantly reducing artifacts.
[0063] Table 1 Quantitative evaluation of the method on the AISTD dataset
[0064]
[0065] Among them, the lower the RMSE, the better; the higher the SSIM and PSNR, the better; S represents the shadow area; NS represents the non-shadow area; ALL represents the entire image.
[0066] Using this invention, direct light and ambient light in shadow images can be effectively and specifically processed, improving the overall performance of the network. At the same time, due to the reasonable introduction of non-shadow priors, the colors of shadow areas are accurately restored, thereby achieving better shadow removal performance.
[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for removing image shadows based on illumination compensation, characterized in that, Includes the following steps: Obtain the image to be processed, input the image to be processed into the trained image shadow removal model, and obtain a shadow-free image; The image shadow removal model includes: a direct light compensation network and an ambient light compensation network; the training process of the image shadow removal model includes: S1. Obtain the training dataset. The training dataset contains multiple image triplets. Each image triplet includes: the image to be removed from the shadow. and its shadowless realistic image and shadow mask image The image to be shaded in the image triplet. and its shadowless realistic image Perform grayscale transformation to obtain the image triplet image of the shadow to be removed. and its shadowless realistic image grayscale image , ; S2, Image to be removed of shadows grayscale image and its shadow mask image The image is input into a direct light compensation network to obtain the image of the shadow to be removed. Direct light compensation coefficient matrix ; S3, using the direct light compensation coefficient matrix Removing shadows from images Perform direct light compensation to obtain the image of the shadow to be removed. Direct light compensation results ; S4. Using the direct light compensation coefficient matrix and the image to be removed of shadows Calculate the image to be removed of shadows Ambient light ; Calculate ambient light Compensation results with direct light The difference between them yields ambient light difference guidance information. ; S5. Image to be removed of shadows Direct light compensation results Shadow mask image and guidance information Input the ambient light compensation network to obtain the image of the shadow to be removed. Ambient light compensation coefficient matrix ; S6. Utilizing the ambient light compensation coefficient matrix Compensation results for direct light Ambient light compensation is performed to obtain the image of the shadow to be removed. Shadowless image ; S7. Calculate the loss function value and update the image shadow removal model based on the loss function value. When the loss function value is minimized, the trained image shadow removal model is obtained.
2. The image shadow removal method based on illumination compensation according to claim 1, characterized in that, The direct light compensation network and ambient light compensation network consist of an encoder, adjacent attention layers, and a decoder; the direct light compensation network is used to remove shadows from the image. grayscale image and its shadow mask image The processing includes: the image to be removed of shadows grayscale image and its shadow mask image Channel overlay is performed, the result of channel overlay is input into the encoder, the encoder output is input into the adjacent attention layer, the output of the adjacent attention layer is input into the decoder, and the decoder output is compared with the image to be removed from the shadow. grayscale image Adding them together yields the direct light compensation coefficient matrix. .
3. The image shadow removal method based on illumination compensation according to claim 2, characterized in that, The encoder consists of a three-layer network structure, with the first two layers being channel attention layers and the last layer being an adjacent attention layer, with downsampling layers added between each layer; the decoder consists of a three-layer network structure, with the first layer being an adjacent attention layer and the last two layers being channel attention layers, with upsampling layers added between each layer.
4. The image shadow removal method based on illumination compensation according to claim 3, characterized in that, The channel attention layer consists of, in sequence: a first LayerNorm layer, a channel attention module, a second LayerNorm layer, and a fully connected layer. The channel attention layer processes its input data by: inputting the input data into the first LayerNorm layer; inputting the output of the first LayerNorm layer into the channel attention module; fusing the output of the channel attention module with the input data to obtain a fusion result; inputting the fusion result into the second LayerNorm layer; inputting the output of the second LayerNorm layer into the fully connected layer; and fusing the output of the fully connected layer with the fusion result to obtain the output data.
5. The image shadow removal method based on illumination compensation according to claim 3, characterized in that, The adjacent attention layer sequentially comprises: a first LayerNorm layer, an adjacent attention module, a channel attention module, a second LayerNorm layer, and a fully connected layer. The adjacent attention layer processes its input data by: inputting the input data into the first LayerNorm layer; inputting the output of the first LayerNorm layer into the adjacent attention module; inputting the output of the adjacent attention module into the channel attention module; fusing the output of the channel attention module with the input data to obtain a fusion result; inputting the fusion result into the second LayerNorm layer; inputting the output of the second LayerNorm layer into the fully connected layer; and fusing the output of the fully connected layer with the fusion result to obtain the output data.
6. The image shadow removal method based on illumination compensation according to claim 1, characterized in that, Direct light compensation results The calculation formula is as follows: .
7. The image shadow removal method based on illumination compensation according to claim 1, characterized in that, Calculate the image to be removed of shadows Ambient light include: S41, Compensation coefficient matrix for direct light The image is truncated within the range (0,1), and the truncated direct light compensation coefficient matrix is then subjected to max pooling to obtain the dilated mask image. ; S42. Apply the mask image Subtract shadow mask image The prior mask image of the local non-shaded area is obtained. ; S43. Use the prior mask image of the local non-shaded area. Multiply by the image to be removed This yields an image of the shaded adjacent region; S44. Calculate the average value of all pixels in the image of the adjacent shadow region, and multiply the average value by the shadow mask image. Ambient light .
8. The image shadow removal method based on illumination compensation according to claim 1, characterized in that, loss function ;in, To recover the loss from direct sunlight, For the recovery loss of ambient light, This is a hyperparameter.
9. The image shadow removal method based on illumination compensation according to claim 8, characterized in that, Direct light recovery loss ;in, It is a constant. Indicates the L1 distance. This represents the i-th layer in the pre-trained model. This represents the weight of the i-th layer in the pre-trained model.
10. The image shadow removal method based on illumination compensation according to claim 8, characterized in that, Ambient light recovery loss ;in, It is a constant. Indicates the L1 distance. This represents the i-th layer in the pre-trained model. This represents the weight of the i-th layer in the pre-trained model.