A method and apparatus for removing shadows from an image

By improving the Transformer encoder-decoder and coordinate attention mechanism, and combining a weighted combination of Charbonnier loss and PSNR loss, the problems of poor model generalization ability and artifacts in image shadow removal methods are solved, achieving high-quality shadow removal and image detail preservation.

CN118134823BActive Publication Date: 2026-11-10NORTHWEST UNIV +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410287500.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-13
Publication Date
2026-11-10
Estimated Expiration
2044-03-13

AI Technical Summary

Technical Problem

Existing image shadow removal methods suffer from poor model generalization ability and insufficient robustness. Furthermore, deep learning-based methods are prone to artifacts and image style inconsistencies after shadow removal.

Method used

An improved Transformer codec is adopted, a coordinate attention mechanism is introduced, and a weighted combination of Charbonnier loss and PSNR loss is used for feature extraction and shadow reconstruction. Image fusion is performed by distinguishing shadow and non-shadow regions through shadow mask.

Benefits of technology

It improves the effect of image shadow removal, reduces artifacts, preserves image details, and increases peak signal-to-noise ratio, resulting in higher quality shadow-removed images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118134823B_ABST
    Figure CN118134823B_ABST
Patent Text Reader

Abstract

This invention discloses an image shadow removal method, comprising: distinguishing a first shadow region and a first non-shadow region from an original shadow image using a shadow mask, and performing a linear mapping on the distinguished original shadow image to obtain a low-dimensional mapped image; performing feature extraction and shadow reconstruction on the low-dimensional mapped image using an improved Transformer codec to obtain a reconstructed image; performing a linear mapping on the reconstructed image to obtain a mapping image of the original dimension; distinguishing a second shadow region and a second non-shadow region from the original dimension mapped image using a shadow mask; and fusing the first non-shadow region and the second shadow region to obtain a shadow-removed image; wherein, the improved Transformer codec introduces a coordinate attention mechanism on the basis of a traditional Transformer codec. This invention obtains a higher quality shadow-removed image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an image shadow removal method and apparatus. Background Technology

[0002] When capturing optical images, partial or complete occlusion of light can cause shadows, which can degrade image quality and hinder subsequent visual tasks such as object detection, semantic segmentation, and scene understanding. Therefore, it is necessary to remove image shadows to improve image quality.

[0003] Existing image shadow removal methods are mainly divided into traditional physically-based image shadow removal methods and deep learning-based image shadow removal methods. Traditional physically-based image shadow removal methods typically utilize prior information such as gradients and illumination at shadow locations to generate lighting parameters through physical modeling to remove shadows. Deep learning-based image shadow removal methods utilize deep learning networks to achieve end-to-end image shadow removal. For example, the DeshadowNet network model proposed by Qu et al. expands the network's receptive field by fusing multi-level features and uses contextual semantics and image appearance information to predict shadows with local details to achieve shadow removal; Fu et al. proposed an automatic exposure fusion method, which uses a shadow-aware fusion network to adaptively fuse multiple overexposed images to obtain a shadow-free image.

[0004] However, traditional physical model-based shadow removal methods rely heavily on manually designed extractors during feature extraction, requiring manual tuning of numerous complex parameters. Furthermore, each method is designed for its own specific training images, resulting in poor generalization ability and robustness. Existing deep learning-based image shadow removal methods, on the other hand, do not adequately consider the consistency between shadowed and unshadowed regions after shadow removal, sometimes leading to artifacts, ghosting, and overall image inconsistencies. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, the present invention provides an image shadow removal method and apparatus. The technical problem to be solved by the present invention is achieved through the following technical solution:

[0006] In a first aspect, embodiments of the present invention provide an image shadow removal method, comprising:

[0007] The first shadow region image and the first non-shadow region image are distinguished from the original shadow image using a shadow mask, and the entire original shadow image after distinction is linearly mapped and positionally encoded to obtain a low-dimensional mapped image.

[0008] The reconstructed image is obtained by extracting features and reconstructing shadows from the low-dimensional mapped image using a pre-trained improved Transformer codec.

[0009] The reconstructed image is linearly mapped to obtain a mapped image of the original dimension. The shadow mask is used to distinguish the second shadow region image and the second non-shadow region image from the mapped image of the original dimension. The first non-shadow region image and the second shadow region image are fused to obtain a de-shadowed image.

[0010] The improved Transformer codec introduces a coordinate attention mechanism on the basis of the traditional Transformer codec; and the total loss of the improved Transformer codec is designed by a weighted combination of Charbonnier loss and PSNR loss during training.

[0011] In one embodiment of the present invention, the improved Transformer codec includes an improved Transformer encoder and an improved Transformer decoder; wherein,

[0012] The improved Transformer encoder consists of several sequentially connected encoding structures; the i-th encoding structure includes: a normalization layer, a coordinate attention module, an adder, a normalization layer, a multilayer perceptron, a GELU activation layer, an adder, and a downsampling layer, which are connected in sequence. The input of the first adder in the i-th encoding structure is also connected to the input of the first normalization layer in the i-th encoding structure, and the input of the second adder in the i-th encoding structure is also connected to the output of the first adder in the i-th encoding structure.

[0013] The improved Transformer decoder consists of several sequentially connected decoding structures; the i-th decoding structure includes: a splicing layer, a normalization layer, a coordinate attention module, an adder, a normalization layer, a multilayer perceptron, a GELU activation layer, an adder, and an upsampling layer, all connected in sequence. The splicing layer in the i-th decoding structure is also connected to the output of the i-th encoding structure. The input of the first adder in the i-th decoding structure is also connected to the input of the first normalization layer in the i-th decoding structure. The input of the second adder in the i-th decoding structure is also connected to the output of the first adder in the i-th decoding structure.

[0014] In one embodiment of the present invention, the coordinate attention module in each encoding structure and each decoding structure has the same structure; wherein,

[0015] The coordinate attention module includes: two average pooling layers; a concatenation layer that concatenates the outputs of the two average pooling layers; a two-dimensional convolutional layer, a batch normalization layer, and a nonlinear activation layer connected sequentially to the concatenation layer; two two-dimensional convolutional layers connected to the outputs of the nonlinear activation layers, each of which is connected to a sigmoid activation layer; and a multiplier connected to the outputs of the two sigmoid activation layers, the input of which is also connected to the inputs of the two average pooling layers.

[0016] In one embodiment of the present invention, the total loss formula of the improved Transformer codec is expressed as:

[0017] Loss=α×L Charbonnier +β×L PSNR ;

[0018] Where Loss represents the total loss, α and β represent the weight hyperparameters, and L Charbonnier Indicates Charbonnier loss. This represents the output image of the improved Transformer codec, where I represents the true shadowless image, and h, w, and c represent the height, width, and number of channels of the true shadowless image, respectively. Let L represent the difference between the (i,j)th pixel of the k-th channel in the output image and the (i,j)th pixel of the k-th channel in the true shadowless image, where ε represents the error parameter. PSNR Indicates PSNR loss, MAX I The maximum pixel value in the true shadowless image is represented by , and MSE represents the mean square error between the output image and the true shadowless image.

[0019] In one embodiment of the present invention, L Charbonnier The weight value is set to be greater than L. PSNR The weight value is α > β.

[0020] In one embodiment of the present invention, before distinguishing the first shadow region image and the first non-shadow region image from the original shadow image using a shadow mask, the method further includes:

[0021] The original shadow image and the shadow mask are respectively divided into several shadow image blocks and mask image blocks of fixed size. The mask image blocks are used to distinguish the first shadow region image block and the first non-shadow region image block from the corresponding shadow image block. All the first shadow region image blocks form the first shadow region image, and all the first non-shadow region image blocks form the first non-shadow region image.

[0022] Secondly, embodiments of the present invention provide an image shadow removal apparatus, the apparatus comprising:

[0023] The image mapping module is used to distinguish the first shadow region image and the first non-shadow region image from the original shadow image using a shadow mask, and to perform linear mapping on the entire original shadow image after distinction to obtain a low-dimensional mapped image.

[0024] The image reconstruction module is used to extract features and reconstruct shadows from the low-dimensional mapped image using a pre-trained improved Transformer codec to obtain a reconstructed image. The improved Transformer codec introduces a coordinate attention mechanism on the basis of the traditional Transformer codec. The total loss of the improved Transformer codec is designed by a weighted combination of Charbonnier loss and PSNR loss during training.

[0025] The image mapping and fusion module is used to perform linear mapping on the reconstructed image to obtain a mapped image of the original dimension, use the shadow mask to distinguish a second shadow region image and a second non-shadow region image from the mapped image of the original dimension, and fuse the first non-shadow region image and the second shadow region image to obtain a de-shadowed image.

[0026] In one embodiment of the present invention, the device further includes:

[0027] The image segmentation module is used to divide the original shadow image and the shadow mask into several shadow image blocks and mask image blocks of fixed size, respectively. The mask image blocks are used to distinguish the first shadow region image blocks and the first non-shadow region image blocks from the corresponding shadow image blocks. All the first shadow region image blocks form the first shadow region image, and all the first non-shadow region image blocks form the first non-shadow region image.

[0028] The beneficial effects of this invention are:

[0029] The proposed image shadow removal method includes: using a shadow mask to distinguish a first shadow region image and a first non-shadow region image from an original shadow image, and performing linear mapping and position encoding on the entire original shadow image after distinction to obtain a low-dimensional mapped image; using a pre-trained improved Transformer codec to extract features from the low-dimensional mapped image and reconstruct shadows to obtain a reconstructed image; performing linear mapping on the reconstructed image to obtain a mapping image of the original dimension, using a shadow mask to distinguish a second shadow region image and a second non-shadow region image from the mapping image of the original dimension, and fusing the first non-shadow region image and the second shadow region image to obtain a shadow-removed image; wherein, the improved Transformer codec introduces a coordinate attention mechanism on the basis of the traditional Transformer codec; and during the training process, a weighted combination of Charbonnier loss and PSNR loss is used to design the total loss of the improved Transformer codec. As can be seen, the improved Transformer codec provided by this invention does not require manual parameter tuning. The improved Transformer codec utilizes the powerful context learning capability of the ViT architecture and greatly improves the network model's sensitivity to shadow regions by incorporating a coordinate attention mechanism. It applies the learned information from non-shadow regions, such as illumination and color, to shadow regions to guide shadow restoration. At the same time, it uses a weighted combination of Charbonnier loss and PSNR loss as the total loss during network training, ensuring that the peak signal-to-noise ratio is further improved while better preserving image details, resulting in higher quality shadow-free images and improving the overall image quality.

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0031] Figure 1 This is a schematic flowchart of an image shadow removal method provided in an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the improved Transformer codec network model provided in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of the coordinate attention module provided in an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the entire image shadow removal process provided in an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the structure of an image shadow removal device provided in an embodiment of the present invention;

[0036] Figure 6 This is a schematic diagram of another image shadow removal device provided in an embodiment of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0038] Firstly, please see Figure 1 This invention provides an image shadow removal method, which specifically includes the following steps:

[0039] S10. Using a shadow mask, the first shadow region image and the first non-shadow region image are distinguished from the original shadow image, and the entire original shadow image after distinction is linearly mapped and positionally encoded to obtain a low-dimensional mapped image.

[0040] This invention utilizes a shadow mask to distinguish a first shadow region image and a first non-shadow region image from an original shadow image, and assigns labels to the distinguished first shadow region image and first non-shadow region image so that the images can be used in subsequent network model processing of the two different regions. The entire original shadow image after distinction is linearly mapped to obtain a low-dimensional mapped image, that is, the low-dimensional features of the entire original shadow image are obtained. The linear mapping process is expressed as follows:

[0041]

[0042] Where I1′ represents the low-dimensional mapped image, and Linear(·) represents the linear mapping function. This represents the original shadow image with labels. After mapping, the low-dimensional mapped image I1′ is positionally encoded to obtain a low-dimensional mapped image carrying positional encoding information, denoted as I2′.

[0043] Here, to improve image processing efficiency, before using a shadow mask to distinguish the first shadow region image and the first non-shadow region image from the original shadow image, the following steps are also included:

[0044] The original shadow image and the shadow mask are divided into several fixed-size shadow image blocks and mask image blocks, respectively. The mask image blocks are used to distinguish the first shadow region image block and the first non-shadow region image block from the corresponding shadow image block. All first shadow region image blocks form the first shadow region image, and all first non-shadow region image blocks form the first non-shadow region image. Subsequent linear mapping and positional encoding can also be processed based on the divided image blocks.

[0045] S20. The pre-trained improved Transformer codec is used to extract features from the low-dimensional mapped image and reconstruct shadows to obtain the reconstructed image. The improved Transformer codec introduces a coordinate attention mechanism on the basis of the traditional Transformer codec. The total loss of the improved Transformer codec is designed by a weighted combination of Charbonnier loss and PSNR (Peak Signal-to-Noise Ratio) loss during training.

[0046] The inventors discovered that the Transformer codec in the Vision Transformer (ViT) architecture possesses strong context learning capabilities. Therefore, information learned from non-shadow areas, such as illumination and color, can be applied to shadow areas for shadow restoration. The coordinate attention mechanism decomposes channel attention into two one-dimensional feature codes, aggregating features along two spatial directions (X and Y). This captures long-distance dependencies along one spatial direction while preserving precise positional information along the other. The resulting attention map not only captures inter-channel information but also includes direction-related positional information. Therefore, applying the attention map to the input feature map helps the Transformer codec better locate and identify shadow areas, improving shadow removal performance. Thus, this invention employs a Transformer codec and introduces a coordinate attention mechanism to obtain an improved Transformer codec. This improved codec achieves spatial attention selection, thereby enabling shadow removal. Specifically:

[0047] like Figure 2 As shown, the improved Transformer codec proposed in this embodiment of the invention includes an improved Transformer encoder and an improved Transformer decoder; wherein,

[0048] The improved Transformer encoder consists of several sequentially connected coding structures; the i-th coding structure includes: a normalization layer, a coordinate attention module, an adder, a normalization layer, a multilayer perceptron, a GELU (Gaussian Error Linear Unit) activation layer, an adder, and a downsampling layer, which are connected in sequence. The input of the first adder in the i-th coding structure is also connected to the input of the first normalization layer in the i-th coding structure, and the input of the second adder in the i-th coding structure is also connected to the output of the first adder in the i-th coding structure.

[0049] The improved Transformer decoder consists of several sequentially connected decoding structures. The i-th decoding structure includes, in sequence: a splicing layer, a normalization layer, a coordinate attention module, an adder, a normalization layer, a multilayer perceptron, a GELU activation layer, an adder, and an upsampling layer. The splicing layer in the i-th decoding structure is also connected to the output of the i-th encoding structure. The input of the first adder in the i-th decoding structure is also connected to the input of the first normalization layer in the i-th decoding structure. The input of the second adder in the i-th decoding structure is also connected to the output of the first adder in the i-th decoding structure.

[0050] Depend on Figure 2 As can be seen, by adding the coordinate attention module to the traditional Transformer codec structure, the basic unit of the Transformer codec based on the coordinate attention mechanism in this embodiment of the invention can be obtained. This unit utilizes the coordinate attention mechanism to compress spatial information and captures long-range correlations through a feedforward multilayer perceptron. The L basic units stacked together constitute the improved Transformer codec. The detailed implementation process of the i-th encoder structure in the improved Transformer codec is as follows:

[0051] The position-encoded image I2′ enters the encoder structure, where, under the action of the normalization layer and the coordinate attention module, a preliminary weighted feature map of the image is obtained.

[0052]

[0053] Here, LN is the first normalization layer operation in the encoder structure, and CA is the coordinate attention module.

[0054] Combining a coordinate attention module can more accurately identify the relationship between shadowed and unshadowed regions. Then, a multilayer perceptron is used to sense the long-range correlation between regions to obtain the dependencies between vectors, and weights are generated after activation to obtain the image's sampling features I. L ′ is represented as:

[0055]

[0056] Where LN is the second normalization layer operation in the encoder structure, MLP is a multilayer perceptron, and GELU is the GELU activation function.

[0057] Finally, the output of this encoder structure is obtained after the downsampling layer and denoted as .

[0058] The final output image of the improved Transformer encoder is obtained by repeatedly passing L identical encoder structures, denoted as I. L .

[0059] The output I of the i-th encoder structure corresponding to the i-th decoder structure L i and the output of the previous decoder structure Feature concatenation is performed, and the result is used as the input I′ of the i-th decoder structure. O :

[0060]

[0061] The i-th decoder structure takes input I′ as input. O Perform the same operation as the i-th encoder structure, and after upsampling, obtain the output of this encoder structure, denoted as . The reconstructed image, obtained by repeatedly passing the same decoder structure L times, is the output image of the entire improved Transformer codec. In this process, the improved Transformer decoder finds the lighting, color, and other information that best matches the shadow area in the non-shadow area according to the weights, and then restores the color and lighting in the shadow area to achieve shadow restoration and obtain the output image of the entire improved Transformer codec.

[0062] Furthermore, the coordinate attention modules in each encoding structure and each decoding structure have the same structure; among them,

[0063] like Figure 3As shown, the coordinate attention module includes: two average pooling layers. For example, if the coordinate attention module is located in a certain encoding structure, the inputs of the two average pooling layers are connected to the output of the first normalization layer of that encoding structure; a concatenation layer that concatenates the outputs of the two average pooling layers; a two-dimensional convolutional layer, a batch normalization layer, and a non-linear activation layer connected sequentially to the concatenation layer; two two-dimensional convolutional layers connected to the outputs of the non-linear activation layers, each of which is connected to a sigmoid activation layer; and a multiplier connected to the outputs of the two sigmoid activation layers, the input of which is also connected to the inputs of the two average pooling layers. The detailed implementation process of the coordinate attention module is as follows:

[0064] For the input feature map of the coordinate attention module, i.e. Figure 2 The feature map after the normalization layer is denoted as LN(I2′). This feature map LN(I2′) is average pooled in both the length and width directions (XY directions) to obtain two one-dimensional vectors:

[0065]

[0066]

[0067] Among them, A x and A y Average pooling operations are performed on the feature map LN(I2′) along the X and Y directions, respectively.

[0068] Then concatenate these two one-dimensional vectors:

[0069]

[0070] Then, 1×1 two-dimensional convolution, batch normalization, and the non-linear activation function Sigmoid are used to perform feature transformation on the concatenated result:

[0071]

[0072] Subsequently, independent features in the X and Y directions are obtained and transformed using 1×1 2D convolution and the sigmoid function to obtain attention vectors in the X and Y directions. and

[0073]

[0074]

[0075]

[0076] To ensure its dimensions match the input, the final attention vectors in the X and Y directions are... and Combined into a weight matrix By fusing with the input feature map LN(I2′), a weighted feature map containing both channel and spatial information is obtained:

[0077]

[0078]

[0079] Furthermore, based on Figure 4 The framework shown illustrates that, to further improve the peak signal-to-noise ratio (PSNR) of the output image, this embodiment of the invention uses a weighted combination of Charbonnier loss and PSNR loss as the total loss of the pre-trained improved Transformer codec during training. Research by the inventors has shown that Charbonnier loss has better robustness and smoothness compared to traditional L1 and L2 losses, performs better in handling outliers, reduces artifacts to a greater extent, and better preserves image details. PSNR, as an important indicator of image quality, is often correlated with human perception of image quality, thus directly reflecting the image restoration effect. This embodiment of the invention uses a weighted combination of Charbonnier loss and PSNR loss, determining their weights by adjusting the hyperparameters, thereby enabling the trained improved Transformer codec to improve the PSNR while maintaining effective shadow removal. Specifically:

[0080] The improved Transformer codec's total loss formula proposed in this embodiment is expressed as follows:

[0081] Loss=α×L Charbonnier +β×L PSNR ;

[0082] Where Loss represents the total loss, α and β represent the weight hyperparameters, and L Charbonnier Indicates Charbonnier loss. This represents the output image of the improved Transformer codec, where I represents the true shadowless image, and h, w, and c represent the height, width, and number of channels of the true shadowless image, respectively. This represents the difference between the (i,j)th pixel of the k-th channel in the output image and the (i,j)th pixel of the k-th channel in the true shadowless image. ε represents the error parameter, which can take values ​​of 0.001. PSNR Indicates PSNR loss, MAXI The maximum pixel value in the true shadowless image is represented by , and MSE represents the mean square error between the output image and the true shadowless image. The weights of the two losses can be adjusted by setting different values ​​for α and β.

[0083] During model training, the inventors discovered that the weights of the Charbonnier loss need to be set higher because a higher weight Charbonnier loss makes the network more robust to discrete points and performs better in image detail processing, thus ensuring high-quality restored images. This also indirectly improves the peak signal-to-noise ratio (PSNR) of the restored images. However, if the PSNR loss weight is too high, there is a possibility of a high PSNR but poor actual visual quality of the restored image. Therefore, by combining a high-weight Charbonnier loss and a low-weight PSNR loss, the network can improve the PSNR of the restored image while ensuring effective shadow removal, thus achieving corresponding improvements in both qualitative and quantitative aspects of the image shadow removal task. Therefore, more preferably, in this embodiment of the invention, L... Charbonnier The weight value is set to be greater than L. PSNR The weight value is α > β.

[0084] S30. Perform linear mapping on the reconstructed image to obtain the original dimension mapping image. Use a shadow mask to distinguish the second shadow region image and the second non-shadow region image from the original dimension mapping image. Fuse the first non-shadow region image and the second shadow region image to obtain the de-shadowed image.

[0085] In this embodiment of the invention, the positional encoding information introduced in S10 is used to perform a linear mapping on the reconstructed image to obtain a mapped image of the original dimension, that is, to restore the same data dimension as the original shadow image. The linear mapping process is expressed as follows:

[0086]

[0087] Then, a shadow mask is used to map the image from the original dimension. The image distinguishes between the second shadow region image and the second non-shadow region image. The first non-shadow region image and the second shadow region image are then fused to obtain a de-shadowed image, which retains both the non-shadow region of the original shadow image and the original dimensional mapping image. The shadowed area is identified, and the final unshaded image is output as I. f .

[0088] In summary, the image shadow removal method proposed in this embodiment includes: using a shadow mask to distinguish a first shadow region image and a first non-shadow region image from the original shadow image, and performing linear mapping and position encoding on the entire original shadow image after distinction to obtain a low-dimensional mapped image; using a pre-trained improved Transformer codec to extract features from the low-dimensional mapped image and reconstruct shadows to obtain a reconstructed image; performing linear mapping on the reconstructed image to obtain a mapping image of the original dimension, using a shadow mask to distinguish a second shadow region image and a second non-shadow region image from the mapping image of the original dimension, and fusing the first non-shadow region image and the second shadow region image to obtain a shadow-removed image; wherein, the improved Transformer codec introduces a coordinate attention mechanism on the basis of the traditional Transformer codec; and during the training process, a weighted combination of Charbonnier loss and PSNR loss is used to design the total loss of the improved Transformer codec. As can be seen, the improved Transformer codec provided in this embodiment of the invention does not require manual parameter tuning. The improved Transformer codec utilizes the powerful context learning capability of the ViT architecture and greatly improves the network model's sensitivity to shadow regions by incorporating a coordinate attention mechanism. It applies the learned information from non-shadow regions, such as illumination and color, to shadow regions to guide shadow restoration. At the same time, it uses a weighted combination of Charbonnier loss and PSNR loss as the total loss during network training, ensuring that the peak signal-to-noise ratio is further improved while better preserving image details, resulting in higher quality shadow-free images and improving the overall image quality.

[0089] Secondly, please see Figure 5 This invention provides an image shadow removal apparatus, comprising:

[0090] The image mapping module is used to distinguish the first shadow region image and the first non-shadow region image from the original shadow image using a shadow mask, and to perform linear mapping on the entire original shadow image after distinction to obtain a low-dimensional mapped image.

[0091] The image reconstruction module is used to extract features from low-dimensional mapped images and reconstruct shadows using a pre-trained improved Transformer codec to obtain reconstructed images. The improved Transformer codec introduces a coordinate attention mechanism on the basis of the traditional Transformer codec. The total loss of the improved Transformer codec is designed by a weighted combination of Charbonnier loss and PSNR loss during training.

[0092] The image mapping and fusion module is used to perform linear mapping on the reconstructed image to obtain the original dimension of the mapped image, use a shadow mask to distinguish the second shadow region image and the second non-shadow region image from the original dimension of the mapped image, and fuse the first non-shadow region image and the second shadow region image to obtain the de-shadowed image.

[0093] Furthermore, the improved Transformer codec in the image reconstruction module of this embodiment includes an improved Transformer encoder and an improved Transformer decoder; wherein,

[0094] The improved Transformer encoder consists of several sequentially connected coding structures; the i-th coding structure includes: a normalization layer, a coordinate attention module, an adder, a normalization layer, a multilayer perceptron, a GELU activation layer, an adder, and a downsampling layer, which are connected in sequence. The input of the first adder in the i-th coding structure is also connected to the input of the first normalization layer in the i-th coding structure, and the input of the second adder in the i-th coding structure is also connected to the output of the first adder in the i-th coding structure.

[0095] The improved Transformer decoder consists of several sequentially connected decoding structures. The i-th decoding structure includes, in sequence: a splicing layer, a normalization layer, a coordinate attention module, an adder, a normalization layer, a multilayer perceptron, a GELU activation layer, an adder, and an upsampling layer. The splicing layer in the i-th decoding structure is also connected to the output of the i-th encoding structure. The input of the first adder in the i-th decoding structure is also connected to the input of the first normalization layer in the i-th decoding structure. The input of the second adder in the i-th decoding structure is also connected to the output of the first adder in the i-th decoding structure.

[0096] Furthermore, in the image reconstruction module of this embodiment, the coordinate attention module in each encoding structure and each decoding structure has the same structure; wherein,

[0097] The coordinate attention module includes: two average pooling layers; a concatenation layer that concatenates the outputs of the two average pooling layers; a two-dimensional convolutional layer, a batch normalization layer, and a non-linear activation layer connected sequentially to the concatenation layer; two two-dimensional convolutional layers connected to the outputs of the non-linear activation layers, each of which is connected to a sigmoid activation layer; and a multiplier connected to the outputs of the two sigmoid activation layers, the input of which is also connected to the inputs of the two average pooling layers.

[0098] Furthermore, the total loss formula of the improved Transformer codec in the image reconstruction module of this embodiment is expressed as:

[0099] Loss=α×L Charbonnier +β×L PSNR ;

[0100] Where Loss represents the total loss, α and β represent the weight hyperparameters, and L Charbonnier Indicates Charbonnier loss. This represents the output image of the improved Transformer codec, where I represents the true shadowless image, and h, w, and c represent the height, width, and number of channels of the true shadowless image, respectively. Let L represent the difference between the (i,j)th pixel of the k-th channel in the output image and the (i,j)th pixel of the k-th channel in the true shadowless image, where ε represents the error parameter. PSNR Indicates PSNR loss, MAX I The maximum pixel value in the true shadowless image is represented by , and MSE represents the mean square error between the output image and the true shadowless image.

[0101] Furthermore, in the image reconstruction module of this embodiment of the invention, L Charbonnier The weight value is set to be greater than L. PSNR The weight value is α > β.

[0102] Further, please see Figure 6 The image shadow removal apparatus provided in this embodiment of the invention further includes:

[0103] The image segmentation module is used to divide the original shadow image and the shadow mask into several shadow image blocks and mask image blocks of fixed size, respectively. The mask image blocks are used to distinguish the first shadow region image block and the first non-shadow region image block from the corresponding shadow image block. All the first shadow region image blocks form the first shadow region image, and all the first non-shadow region image blocks form the first non-shadow region image.

[0104] As the apparatus embodiment of the second aspect is basically similar to the method embodiment of the first aspect, the description is relatively simple, and relevant details can be found in the description of the method embodiment of the first aspect.

[0105] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0106] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the specification and accompanying drawings, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0107] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for removing shadows from an image, characterized in that, The method includes: The first shadow region image and the first non-shadow region image are distinguished from the original shadow image using a shadow mask, and the entire original shadow image after distinction is linearly mapped and positionally encoded to obtain a low-dimensional mapped image. The reconstructed image is obtained by extracting features and reconstructing shadows from the low-dimensional mapped image using a pre-trained improved Transformer codec. The reconstructed image is linearly mapped to obtain a mapped image of the original dimension. The shadow mask is used to distinguish the second shadow region image and the second non-shadow region image from the mapped image of the original dimension. The first non-shadow region image and the second shadow region image are fused to obtain a de-shadowed image. The improved Transformer codec introduces a coordinate attention mechanism on the basis of the traditional Transformer codec; and the total loss of the improved Transformer codec is designed by a weighted combination of Charbonnier loss and PSNR loss during training.

2. The image shadow removal method according to claim 1, characterized in that, The improved Transformer codec includes an improved Transformer encoder and an improved Transformer decoder; wherein, The improved Transformer encoder consists of several sequentially connected encoding structures; the i-th encoding structure includes: a normalization layer, a coordinate attention module, an adder, a normalization layer, a multilayer perceptron, a GELU activation layer, an adder, and a downsampling layer, which are connected in sequence. The input of the first adder in the i-th encoding structure is also connected to the input of the first normalization layer in the i-th encoding structure, and the input of the second adder in the i-th encoding structure is also connected to the output of the first adder in the i-th encoding structure. The improved Transformer decoder consists of several sequentially connected decoding structures; the i-th decoding structure includes: a splicing layer, a normalization layer, a coordinate attention module, an adder, a normalization layer, a multilayer perceptron, a GELU activation layer, an adder, and an upsampling layer, all connected in sequence. The splicing layer in the i-th decoding structure is also connected to the output of the i-th encoding structure. The input of the first adder in the i-th decoding structure is also connected to the input of the first normalization layer in the i-th decoding structure. The input of the second adder in the i-th decoding structure is also connected to the output of the first adder in the i-th decoding structure.

3. The image shadow removal method according to claim 2, characterized in that, The coordinate attention modules in each encoding structure and each decoding structure have the same structure; among them, The coordinate attention module includes: two average pooling layers; a concatenation layer that concatenates the outputs of the two average pooling layers; a two-dimensional convolutional layer, a batch normalization layer, and a nonlinear activation layer connected sequentially to the concatenation layer; two two-dimensional convolutional layers connected to the outputs of the nonlinear activation layers, each of which is connected to a sigmoid activation layer; and a multiplier connected to the outputs of the two sigmoid activation layers, the input of which is also connected to the inputs of the two average pooling layers.

4. The image shadow removal method according to claim 1, characterized in that, The total loss formula for the improved Transformer codec is expressed as: Loss=α×L Charbonnier +β×L PSNR ; Where Loss represents the total loss, α and β represent the weight hyperparameters, and L Charbonnier Indicates Charbonnier loss. This represents the output image of the improved Transformer codec, where I represents the true shadowless image, and h, w, and c represent the height, width, and number of channels of the true shadowless image, respectively. Let L represent the difference between the (i,j)th pixel of the k-th channel in the output image and the (i,j)th pixel of the k-th channel in the true shadowless image, where ε represents the error parameter. PSNR Indicates PSNR loss, MAX I The maximum pixel value in the true shadowless image is represented by , and MSE represents the mean square error between the output image and the true shadowless image.

5. The image shadow removal method according to claim 4, characterized in that, L Charbonnier The weight value is set to be greater than L. PSNR The weight value is α > β.

6. The image shadow removal method according to claim 1, characterized in that, Before using a shadow mask to distinguish the first shadow region image and the first non-shadow region image from the original shadow image, the method further includes: The original shadow image and the shadow mask are respectively divided into several shadow image blocks and mask image blocks of fixed size. The mask image blocks are used to distinguish the first shadow region image block and the first non-shadow region image block from the corresponding shadow image block. All the first shadow region image blocks form the first shadow region image, and all the first non-shadow region image blocks form the first non-shadow region image.

7. An image shadow removal device, characterized in that, The device includes: The image mapping module is used to distinguish the first shadow region image and the first non-shadow region image from the original shadow image using a shadow mask, and to perform linear mapping on the entire original shadow image after distinction to obtain a low-dimensional mapped image. The image reconstruction module is used to extract features and reconstruct shadows from the low-dimensional mapped image using a pre-trained improved Transformer codec to obtain a reconstructed image. The improved Transformer codec introduces a coordinate attention mechanism on the basis of the traditional Transformer codec. The total loss of the improved Transformer codec is designed by a weighted combination of Charbonnier loss and PSNR loss during training. The image mapping and fusion module is used to perform linear mapping on the reconstructed image to obtain a mapped image of the original dimension, use the shadow mask to distinguish a second shadow region image and a second non-shadow region image from the mapped image of the original dimension, and fuse the first non-shadow region image and the second shadow region image to obtain a de-shadowed image.

8. The image shadow removal apparatus according to claim 7, characterized in that, The device further includes: The image segmentation module is used to divide the original shadow image and the shadow mask into several shadow image blocks and mask image blocks of fixed size, respectively. The mask image blocks are used to distinguish the first shadow region image blocks and the first non-shadow region image blocks from the corresponding shadow image blocks. All the first shadow region image blocks form the first shadow region image, and all the first non-shadow region image blocks form the first non-shadow region image.

Citation Information

Patent Citations

  • Image shadow removal system and method based on potential diffusion model

    CN116485682A

  • Method for removing illumination influence of single image based on deep learning

    CN117474812A