Single image shadow removal method based on global context perception and computer device

Abstract

The invention relates to the field of shadow removal, in particular to a single-image shadow removal method based on global context perception and a computer device. According to the scheme, an original data set is preprocessed, then a shadow model based on the Retinex theory is introduced, and the model can describe illumination and color differences between a shadow area and a non-shadow area in an image; an encoder and a decoder which are constructed by using a multi-scale channel attention mechanism are used for efficiently capturing hierarchical global information; constructing a shadow removal network based on context awareness based on an encoder and a decoder, and adding a shadow interaction module and a shadow interaction attention mechanism in a bottleneck stage of the shadow removal network based on a shadow model; training a shadow removal network by using a loss function; testing the trained shadow removal network by using the public data set to evaluate the performance of the shadow removal network; and finally, performing image shadow removal according to the evaluated shadow removal network. The method is suitable for single image shadow removal.

Description

Technical Field

[0001] The present invention relates to the field of shadow removal, and in particular to a single image shadow removal method based on global context perception and a computer device. Background Art

[0002] In the field of computer vision and image processing, image shadow removal is a technique that aims to identify and remove shadows from digital images. This technique is essential for improving the visual quality of images and further image analysis processing such as object detection, tracking, and scene understanding. Shadows can interfere with image feature extraction and interpretation, affecting the overall brightness, contrast, and color balance of the image, which can negatively affect the effectiveness of automated vision systems. Therefore, effective shadow removal techniques are essential for many applications, including but not limited to security surveillance, autonomous driving, industrial automation, medical imaging, image editing, and more.

[0003] Shadows are dark areas formed on other surfaces due to light being blocked by objects. Shadows can generally be divided into self-shadows and cast shadows. Self-shadows are shadows formed on the surface of an object itself that blocks light, while cast shadows are shadows formed on other objects or surfaces by objects blocking light. The existence of shadows can provide clues for understanding three-dimensional scenes, but in two-dimensional images, shadows often cause misunderstandings and affect the accuracy of image analysis. Therefore, image shadow removal faces multiple challenges, mainly including: the distinction between shadow areas and non-shadow areas, the diversity of shadow attributes, and illumination and color restoration. First, correctly identifying which areas in the image are shadows is the first step in shadow removal. This task is complicated by lighting conditions, shadow intensity, and the ambiguity of shadow boundaries. Second, different environments and different types of light sources cause large variations in the size, shape, color, and ambiguity of shadows. After removing the shadows, the illumination and color of the original shadow area must be properly adjusted to be consistent with the surrounding environment, which requires complex illumination models and color correction algorithms.

[0004] Initial shadow removal techniques mainly rely on simple image processing techniques, such as thresholding and illumination compensation. These methods often assume that shadow areas are lower in brightness than non-shadow areas and that color changes are not significant, but these assumptions often do not hold true in complex scenes. Later, researchers began to explore shadow removal methods based on physical models, such as methods based on Retinex theory, which attempt to restore the true illumination and color of shadow areas by simulating the human eye's perception of illumination and color. This type of method can take into account more natural illumination changes when processing shadows, but has high computational complexity, low efficiency, and is sensitive to parameter selection. It cannot accurately distinguish between shadows and non-shadow areas in the image, nor can it maintain consistency in color and brightness under different lighting conditions. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a single image shadow removal method and computer device based on global context perception, which achieves high efficiency and accuracy in the image shadow removal process. At the same time, the global context perception mechanism is introduced, which can more accurately distinguish between shadow and non-shadow areas in the image and maintain the consistency of color and brightness under different lighting conditions.

[0006] The present invention adopts the following technical solutions to achieve the above-mentioned purpose. In a first aspect, the present invention provides a single image shadow removal method based on global context perception, comprising:

[0007] S1, preprocess the original data set;

[0008] S2, construct encoder and decoder using multi-scale channel attention mechanism;

[0009] S3. Building a Transformer-based shadow removal network based on the encoder and decoder. The shadow removal network introduces a shadow model based on Retinex theory, and introduces a shadow interaction module and a shadow interaction attention mechanism at the bottleneck stage of the shadow removal network.

[0010] S4, train the shadow removal network using the L1 loss function;

[0011] S5. Perform image shadow removal based on the trained shadow removal network.

[0012] Furthermore, step S1 specifically includes:

[0013] The input images are obtained from the original dataset, and all input images are normalized to a uniform size and color space, where the image size is 256×256 and the color space is converted from BGR format to RGB format. Then, data diversity is increased through data augmentation techniques, including random rotation, flipping, and cropping of images.

[0014] Furthermore, the shadow model based on Retinex theory decomposes the image into two components: illumination and reflectivity. The illumination component L represents the part of the image that changes due to changes in lighting, and the reflectivity component R represents the color and characteristics of the object itself. The shadow removal task is to adjust the illumination and color of the image to remove the shadow noise from the image I. s Recover the bottom shadow-free image I sf , and then define the shadow-free image as I sf =L sf ⊙R, for shadow removal, the formula of the Retinex-based shadow model is as follows:

[0015] I s =I m⊙L s ⊙R+(1-I m )⊙L ns ⊙R;

[0016] I m Represents the mask image, the shadow area is 1, the non-shadow area is 0, L s , L ns and L sf Represent the illumination of the shadow area, non-shadow area and shadow-free image respectively.

[0017] Furthermore, in step S3, the encoder is composed of three BasicFormer layer sequences and corresponding downsampling layers, and the decoder is composed of three BasicFormer layer sequences and corresponding upsampling layers, and the BasicFormer layer includes a custom Transformer, and the Transformer includes a CATransformer module;

[0018] First, given a shadow input I s ∈R 3×H×W , the corresponding shadow mask I m ∈R H×W , then an input projection layer converts the size and number of channels of the input data into the embedding dimension required by the shadow removal network at the initial stage of the network. The projection layer applies a linear projection LinearProj(·) to get the underlying feature embedding of the input, denoted as X 0 ∈R C ×H×W , where C is the embedding dimension, H and W are the height and width of the image, respectively, and then X 0 Input into the encoder to superimpose multi-scale global features. Each CATansformer module includes a channel attention module, which is composed of channel attention layers;

[0019] The channel attention layer first performs global average pooling on the spatial feature information of each channel of the input, that is, compressing it into a single scalar value. The formula is as follows:

[0020]

[0021] Where S c is the global average of the global average pooling operation on the cth channel, is the eigenvalue of the cth channel at position (i, j);

[0022] Each CATansformer module also includes two convolutional layers, which are used to learn the weight of each channel from the features obtained by global average pooling. The first convolutional layer uses a 1×1 convolution kernel and then uses the ReLU activation function. This activation layer is used to introduce nonlinearity. The second convolutional layer uses a 1×1 convolution kernel to restore the channel dimension by expanding the channel and output the attention weight that each channel should have. Finally, the Sigmoid activation function is used to compress the output of the convolutional layer to the range of [0,1] to indicate the importance of each channel. The formula is as follows:

[0023] U c =σ(W 2 ReLU(W 1 ·S c +b 1 )+b 2 );

[0024] Where W 1 and b 1 are the weights and biases of the first convolutional layer, W 2 and b 2 is the weight and bias of the second convolutional layer, and the attention weight U of each channel is obtained. c , U c It is used to adjust the activation strength of the corresponding channel in the original input feature map. The formula is as follows:

[0025]

[0026] in is the value of channel c at position (i, j) of the adjusted output feature map;

[0027] The channel attention module first applies the same convolution operation f twice to the feature map X of a given input. conv The output is X', that is, X' = f conv (f conv (X)), where f conv (X) = PReLU(W·X+b), where W and b are the weight and bias of the convolutional layer respectively;

[0028] Then, the channel attention layer is used to assign attention weights to each channel in X'. Finally, the original feature map input X is added back to the result Y adjusted by the channel attention layer through the residual connection, and the output is Z. The formula is as follows:

[0029] Where PReLU(·) is an activation function;

[0030] The CATansformer module first performs layer normalization on the input feature map X, then compresses the spatial information in sequence through the channel attention module, and then performs a residual connection with the input X. The output feature map is The formula is as follows:

[0031] LN(·) represents layer normalization;

[0032] Then, the obtained feature map Perform layer normalization operations, capture long-range correlations through feed-forward MLP, and finally sum up with the input Perform residual connection, the formula is as follows:

[0033] GELU(·) represents the Gaussian error linear unit activation layer, MLP(·) represents the multi-layer perceptron, and the multi-layer perceptron uses the locally enhanced feed-forward network LeFF;

[0034] The depth of the network is formed by repeating the CATansformer module twice at multiple levels. The formula is as follows:

[0035] X out =Block 2 (Block 1 (X in )), where Block 1 (·) and Block 2 (·) represents two identical CATransformer processing operations, with the underlying feature embedding X 0 After being processed by three BasicFormer layers in the encoder, the hierarchical features {X 1 ,X 2 ,X 3},in

[0036] Each BasicFormer layer in the encoder is processed by a downsampling operation. The downsampling operation halves the height and width of the feature map through a convolution layer with a convolution kernel size of 4×4 and a stride of 2. The input of each CATransformer in the decoder is connected through a skip connection to connect the corresponding feature X L The upsampling operation increases the height and width of the feature map to twice the input size by transposing the convolution layer, that is, the convolution kernel size is 2×2 and the step size is 2. The final output of the decoder is Then, the features finally obtained by the decoder are mapped back to the spatial size and number of channels of the original data through an output projection layer, which applies a linear projection LinearProj(·) to obtain the output I output, where I output ∈R 3×H×W , the shadow removal network finally adds the corrected information back to the original shadow input image I through the residual connection s , the output is a clear image after the shadow is removed The formula is:

[0037] Furthermore, the bottleneck stage of the shadow removal network consists of a sequence of BasicFormer layers, in which two layers of SIMTransformer are used. SIMTransformer combines the shadow interaction attention mechanism to optimize and adjust the shadow features in the image processed by the shadow removal network;

[0038] First, the feature map of the downsampling layer output after the third layer encoder After LN(·) layer normalization, SIMTransformer uses the channel attention module to reweight the channels of the feature map, and then splits the reweighted channel attention feature map into a non-overlapping window sequence X. The size of each window is P×P. is the total number of windows, and then the shadow interaction attention mechanism is performed on the flat features in each window, and then the residual connection is performed with the window sequence X, and the output feature map is The formula is as follows:

[0039] Where ∑ represents the patch correlation map, which is used to describe the correlation between features in each window, CAB represents the channel attention module, and SIA represents the shadow interaction attention mechanism;

[0040] For a position vector in the feature map Y output by the downsampling layer after the third encoder This position vector corresponds to two small blocks in the input shadow image, where i and j are the position indexes of the small window in the feature map, and the value range of i and j is 0 to N-1. The mask map corresponding to position i and position j in the feature map is M i and M j , then the correlation graph between windows ∑ ij The calculation formula is as follows:

[0041] in Represents the exclusive OR operation;

[0042] Next, the attention map is reweighted by using the correlation map ∑ between small blocks, as shown below:

[0043] Where σ∈(0,1) is used to adjust the weights of shadow-shadow pairs and non-shadow-non-shadow pairs, d is a scaling parameter; Q, K, V represent the query vector, key vector, and value vector of the input feature map X, respectively;

[0044] Then the output feature map is transformed through SIMTransformer Perform layer normalization operations, capture long-range correlations through feed-forward MLP, and finally sum up with the input Perform residual connection, the formula is as follows:

[0045] Where LN(·) represents layer normalization, GELIU(·) represents Gaussian error linear unit activation layer, MLP(·) represents multi-layer perceptron, and the multi-layer perceptron uses a locally enhanced feedforward network LeFF.

[0046] Furthermore, the shadow removal network is trained using the L1 loss function as follows:

[0047] in is the output image, I gt is a true shadow-free image.

[0048] Furthermore, in step S5, before performing image shadow removal according to the trained shadow removal network, the method further includes: testing the trained shadow removal network using two public data sets, ISTD and SRD, to evaluate the performance of the shadow removal network;

[0049] The performance of the shadow removal network is evaluated using performance evaluation indicators, including peak signal-to-noise ratio, structural similarity index, and root mean square error;

[0050] The peak signal-to-noise ratio is calculated as follows:

[0051]

[0052] The root mean square error is calculated as follows:

[0053]

[0054] Where PSNR stands for peak signal-to-noise ratio, RMSE stands for root mean square error, which is the square root of the average of the squared values ​​of the difference between the actual value and the predicted value, and MAX I is the maximum pixel value of the real image I. For normalized images ranging from 0 to 1, MAX I =1; I(i,j) is the pixel value of the real image at position (i,j), K(i,j) is the pixel value of the processed image at position (i,j), m and n are the width and height of the image respectively;

[0055] The structural similarity index is calculated as follows:

[0056]

[0057] In the formula, SSIM stands for structural similarity index, which is an indicator to measure the similarity of image visual effects. It is used to measure the similarity of structural information, brightness information and contrast information of two images. x and μ y is the average of the x and y images, and is the variance of the x and y images, σ xy is the covariance of the x and y images, c 1 and c 2 are two small constants used to maintain numerical stability.

[0058] In a second aspect, the present invention provides a computer device, including a memory, wherein the memory stores program instructions, and when the program instructions are executed, the above-mentioned single image shadow removal method based on global context perception is executed.

[0059] The beneficial effects of the present invention are:

[0060] The present invention introduces a shadow model based on Retinex theory, which can describe the illumination and color differences between shadow areas and non-shadow areas in an image. Then, an encoder and a decoder constructed using a multi-scale channel attention mechanism are used to efficiently capture hierarchical global information. A context-aware shadow removal network is constructed based on the encoder and the decoder. Based on the shadow model, a shadow interaction module and a shadow interaction attention mechanism are added to the bottleneck stage of the shadow removal network. The context correlation between shadow and non-shadow areas is effectively modeled.

[0061] The present invention has been tested on two widely popular public datasets, ISTD and SRD, and has demonstrated the significant advantages of the model in removing shadows. This method has significant practical value and broad application prospects for applications in the fields of image processing and computational vision, especially in achieving high-quality image restoration and improving visual perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of a single image shadow removal method based on global context perception provided by an embodiment of the present invention;

[0063] Figure 2 is a schematic diagram of the structure of a single image shadow removal network model provided by an embodiment of the present invention;

[0064] Figure 3 is a schematic diagram of a channel attention structure provided by an embodiment of the present invention;

[0065] Figure 4 It is a schematic diagram of the structure of the shadow interaction attention mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0067] like Figure 1 As shown, the present invention provides a single image shadow removal method based on global context perception, which specifically includes:

[0068] Step 1: First, get the input images from the original dataset and normalize all the input images to a uniform size and color space. The image size is 256×256, and the color space is converted from BGR format to RGB format. Then, in order to improve the generalization ability of the model, data diversity is increased through data augmentation technology, including random rotation, flipping, and cropping of images.

[0069] Step 2, introduce a shadow model based on Retinex theory, which is a theoretical framework used to explain and deal with the impact of shadows in images. This theory is mainly based on the way the human eye perceives color and light, and believes that the color of an object is the result of the combined effect of its reflected light and ambient light. In the application of image processing and shadow removal, Retinex theory can help the model distinguish between shadow areas and non-shadow areas in the image, and perform independent lighting and color correction on these areas. Specifically, the shadow model decomposes the image into two main components: illumination and reflectance. The illumination component L represents the part of the image that changes due to changes in lighting (including the formation of shadows), while the reflectance component R represents the color and characteristics of the object itself, which is generally considered to remain unchanged under the influence of shadows. The shadow removal task is to adjust the illumination and color of the image to obtain a shadow noise from the image I containing shadow noise. s Recover the bottom shadow-free image I sf Next, the shadow-free image is defined as I sf =L sf ⊙ R. Therefore, for shadow removal, the formula of the Retinex-based shadow model is as follows:

[0070] I s =I m ⊙L s ⊙R+(1-I m )⊙L ns ⊙R

[0071] Among them, Im Represents the mask image, the shadow area is 1 and the non-shadow area is 0. s , L ns and L sf Represent the illumination of the shadow area, non-shadow area and shadow-free image respectively. The illumination of the non-shadow area L ns May not be exactly equal to L sf , because shadows may actually affect the illumination of the entire environment.

[0072] Step 3, in order to efficiently capture hierarchical global information in the shadow removal network, the present invention uses a multi-scale channel attention mechanism to construct an encoder and a decoder. The encoder is composed of three BasicFormer layer sequences and corresponding downsampling layers, and the decoder is also composed of three BasicFormer layer sequences and corresponding upsampling layers. These BasicFormer layers in the encoder and decoder are customized Transformer modules, in which the CATransformer module is combined with a channel attention mechanism, which helps the model emphasize important channels when processing image features, thereby improving the ability to capture details.

[0073] Specifically, first give a shadow input I s ∈R 3×H×W , the corresponding shadow mask I m ∈R H×W , then an input projection layer converts the size and number of channels of the input data into the embedding dimension required by the model at the initial stage of the network. This layer applies a linear projection LinearProj(·) to get the underlying feature embedding of the input, denoted as X 0 ∈R C×H×W , where C is the embedding dimension, H and W are the height and width of the image respectively. Then X 0 Input into the encoder to superimpose multi-scale global features. Each CATansformer module includes a channel attention module (CAB), where CAB is composed of a channel attention layer (CALayer).

[0074] The channel attention layer first performs global average pooling on the spatial feature information of each channel of the input, that is, compressing it into a single scalar value, which represents the average activation level of the entire channel. The formula is as follows:

[0075]

[0076] Where S c is the global average of the global average pooling operation on the cth channel, is the eigenvalue of the cth channel at position (i, j).

[0077] Next, two convolutional layers are included to learn the weight of each channel from the features obtained by global average pooling. The first convolutional layer uses a 1×1 convolution kernel, which is a compression operation that aims to reduce the number of channels. The ReLU activation function is then used. This activation layer is used to introduce nonlinearity, which helps the network learn complex features. The second convolutional layer uses a 1×1 convolution kernel to restore the channel dimension by "expanding" the channel, while outputting the attention weight that each channel should have. Finally, the Sigmoid activation function σ(·) is used. This activation layer is used to compress the output of the convolutional layer to the range of [0,1] to indicate the importance of each channel. The formula is as follows:

[0078] U c =σ(W 2 ReLU(W 1 ·S c +B 1 )+b 2 )

[0079] Where W 1 and b 1 are the weights and biases of the first convolutional layer, W 2 and b 2 is the weight and bias of the second convolutional layer. The attention weight U of each channel is obtained c It will be used to adjust the activation strength of the corresponding channel in the original input feature map. The formula is as follows:

[0080]

[0081] in is the value of channel c at position (i, j) of the adjusted output feature map.

[0082] The channel attention module CAB first applies two identical convolution operations f to the feature map X of a given input. conv The output is X', that is, X' = f conv (f conv (X)), where f conv (X) = PReLU (W · X + b), where W and b are the weight and bias of the convolutional layer respectively. Then, the attention weight is assigned to each channel in X' through CALayer. Finally, the original feature map input X is added back to the result Y adjusted by the channel attention layer through the residual connection, and the output is Z. The formula is as follows:

[0083]

[0084] PReLU(·) is an activation function that introduces learnable parameters compared to the standard ReLU(·) activation function, making it more flexible when activating negative values.

[0085] like Figure 3 As shown in Figure 1, the CATransformer module first performs layer normalization on the input feature map X, then compresses the spatial information in sequence through the channel attention module (CAB), and then performs a residual connection with the input X. The output feature map is The formula is as follows:

[0086]

[0087] Where LN(·) represents layer normalization. Then, the obtained feature map Perform layer normalization operations, capture long-range correlations through feed-forward MLP, and finally sum up with the input Perform residual connection, the formula is as follows:

[0088]

[0089] Where GELU(·) represents the Gaussian Error Linear Unit activation layer, MLP(·) represents the multi-layer perceptron, and the multi-layer perceptron uses the local enhancement feedforward network LeFF. Compared with the standard MLP, LeFF can better understand local patterns in data such as images, such as details such as edges, corners, and textures through local enhancement layers. Moreover, LeFF uses techniques such as deep separable convolution to reduce the number of parameters and computational requirements, making the model lighter and more suitable for running in resource-constrained environments.

[0090] The BasicFormer layers in the encoder and decoder use CATransformer modules, which are repeated twice in multiple layers to form the depth of the network. The formula is as follows:

[0091] X out =Block 2 (Block 1 (X in ))

[0092] Block 1 (·) and Block 2 (·) indicates two identical CATransformer module processing operations. The underlying feature embedding of the input X 0 After being processed by three BasicFormer layers in the encoder, the hierarchical features {X 1 ,X 2 ,X 3}, where X L ∈ L∈{1,2,3}. Each BasicFormer layer in the encoder is processed by downsampling. The downsampling module reduces the height and width of the feature map by half through a convolution layer with a convolution kernel size of 4×4 and a stride of 2. The input of each CATransformer module in the decoder is connected through a skip connection, which connects the corresponding feature X L The upsampling module increases the height and width of the feature map to twice the input size by transposing the convolution layer, i.e., the convolution kernel size is 2×2 and the step size is 2. The final output of the decoder is Then, the features finally obtained by the decoder are mapped back to the spatial size and number of channels of the original data through an output projection layer. This layer applies a linear projection LinearProj(·) to obtain the output I output , where I output ∈R 3×H×W The network finally adds the corrected information back to the original shadow input image I through the residual connection s , the output is a clear image after the shadow is removed The formula is as follows:

[0093]

[0094] Step 4, such as Figure 2 As shown in the figure, a lightweight shadow removal network is built based on the Transformer encoder and decoder, and a shadow interaction module and shadow interaction attention mechanism are introduced at the bottleneck stage of the shadow removal network. This step is the key link for the entire shadow removal network to process shadow features and effectively remove shadows.

[0095] The bottleneck stage of the shadow removal network consists of a sequence of BasicFormer layers, where two layers of SIMTransformer modules are used in the BasicFormer layer structure. This module is combined with the Shadow Interaction Attention Mechanism (SIA) to further optimize and fine-tune the shadow features in the network processing image. Figure 4 As shown, first, the feature map of the downsampling layer output after the third layer encoder is After LN(·) layer normalization, the SIMTransformer module uses the channel attention module CAB to reweight the channels of the feature map. In order to reduce the computational cost, the reweighted channel attention feature map is then split into a non-overlapping window sequence X, The size of each window is P×P. is the total number of windows. Here, the receptive field corresponding to a window is very large when mapped to the spatial domain. Then, shadow interaction attention (SIA) is performed on the flat features in each window to capture the global context information. Then, a residual connection is performed with the window sequence X, and the output feature map is The formula is as follows:

[0096]

[0097] Where ∑ represents the patch correlation map, which is used to describe the correlation between features within each window, which helps the network better understand the content and structure of local areas of the image when dealing with scenes with complex textures and shadows. In Shadow Interaction Attention (SIA), this correlation map ∑ is used to weight the attention weights to ensure that the model can fully consider the internal characteristics and details of these areas when dealing with shadow areas.

[0098] For a position vector in the feature map Y output by the downsampling layer after the third encoder This position vector can correspond to two small blocks in the input shadow image. Where i, j are the position indexes of the small window in the feature map, and the value range of i, j is 0 to N-1. The mask map corresponding to position i and position j in the feature map is M i and M j , then the correlation graph between windows ∑ ij The calculation formula is as follows:

[0099]

[0100] in Represents an XOR operation. Next, the correlation map ∑ between small blocks is used to reweight the attention map to emphasize the similarity between the non-shadow area and the shadow area. The formula is as follows:

[0101]

[0102] Where σ∈(0,1) is used to adjust the weights of shadow-shadow pairs and non-shadow-non-shadow pairs, and d is a scaling parameter. Q, K, V represent the query vector, key vector, and value vector of the input feature map X, respectively.

[0103] The SIMTransformer module then outputs the feature map Perform layer normalization operations, capture long-range correlations through feed-forward MLP, and finally sum up with the input Perform residual connection, the formula is as follows:

[0104]

[0105] Where LN(·) represents layer normalization, GELU(·) represents Gaussian error linear unit activation layer, MLP(·) represents multi-layer perceptron, and multi-layer perceptron uses a locally enhanced feedforward network LeFF.

[0106] Step 5: First, use the L1 loss function to constrain pixel-level consistency to train the shadow removal network model. The formula is as follows:

[0107]

[0108] in is the output image, I gt is a true shadow-free image.

[0109] Next, the AdamW optimizer is used and the learning rate is adjusted in combination with the cosine annealing strategy to optimize the network training process.

[0110] Step 6, using two public data sets, ISTD and SRD, to test the trained shadow removal network model to evaluate the performance of the model. The performance evaluation indicators used in the present invention include PSNR (peak signal-to-noise ratio), SSIM (structural similarity index), and RMSE (root mean square error).

[0111] PSNR is a common metric for measuring image quality. It is calculated based on the ratio between the maximum possible pixel value and the mean square error (MSE) of the image. The higher the PSNR value, the better the image quality. The formula is as follows:

[0112]

[0113]

[0114] Among them, RMSE is the root mean square error, which is the square root of the average of the squared values ​​of the differences between the actual value and the predicted value. MAX I is the maximum pixel value of the real image I. For normalized images ranging from 0 to 1, MAX I = 1. I(i,j) is the pixel value of the real image at position (i,j), K(i,j) is the pixel value of the processed image at position (i,j), and m and n are the width and height of the image respectively.

[0115] SSIM is an indicator to measure the similarity of image visual effects. It is used to measure the similarity of structural information, brightness information and contrast information of two images. The calculation formula is:

[0116]

[0117] where μ x and μ yis the average of the x and y images, and is the variance of the x and y images, σ xy is the covariance of the x and y images, c 1 and c 2 are two small constants used to maintain numerical stability.

[0118] Next, the present invention describes the effect of the invention through specific simulation experiments:

[0119] The specific hardware equipment for this simulation experiment is as follows: the computer running memory is 32GB, the processor model is 12vCPUIntel(R)Xeon(R)Silver 4214R CPU@2.40GHz, the graphics card is RTX3080Ti, and the video memory is 12GB;

[0120] The specific software environment for this simulation experiment is: Ubuntu18.04 operating system, Pytorch1.7.0, CUDA11.0, Python3.7;

[0121] The datasets used in this simulation experiment: ISTD (Image Shadow Triple Dataset) is a dataset for shadow detection and removal tasks, containing 1330 training and 540 test triplets, each of which includes a shadowed image, a shadow mask, and a shadow-free image. This dataset supports the study of how to distinguish and remove shadows in images under various lighting conditions; SRD (Shadow Removal Dataset) is also a resource for shadow detection and removal research, containing 2680 training and 408 test pairs. The original dataset includes images with and without shadows, but does not contain images with shadow masks. The present invention uses the SRD dataset provided by DHAN to predict mask images for training and testing.

[0122] The model training uses the Adam method for parameter optimization, with a momentum of (0.9, 0.999) and a weight decay of 0.02. In addition, the initial learning rate is 0.0002, and then gradually reduced to 0.000001 through the cosine annealing algorithm. In the experiment, the weight σ used to adjust the shadow-shadow pairs and non-shadow-non-shadow pairs is set to 0.2. The shadow removal network proposed in the present invention adopts a four-scale encoder-decoder architecture (L=3), and the first feature embedding dimension of the network is set to C=32.

[0123] The simulation experiment uses the root mean square error (RMSE) in the LAB color space as a quantitative evaluation index of the shadow removal result, and compares it with the real shadow-free image. The lower the value, the higher the accuracy of the prediction or reconstruction. In addition, the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) are also used to measure the performance of various methods in the RGB color space. For the PSNR and SSIM indicators, the higher the value, the better the image quality, the less noise, the more similar the two images are, and the better the result. The present invention will use the above evaluation indicators to conduct comparative experiments on the following cutting-edge models.

[0124] Table 1 Quantitative results of shadow removal methods on the ISTD dataset

[0125]

[0126]

[0127] Table 2 Quantitative results of shadow removal methods on SRD dataset

[0128]

[0129] The experimental results are shown in Tables 1 and 2. The method proposed in the present invention is compared with popular shadow removal algorithms in recent years, including a traditional method Guo et al., and several methods based on deep learning. Obviously, the method of the present invention performs better than other methods in image shadow removal on the ISTD and SRD datasets. Although some previous methods can reduce or eliminate shadows to a certain extent, they will destroy the illumination and texture details of the non-shadow area in the image after removing the shadow of the image. This destruction is usually manifested as color distortion, blurred details or reduced contrast in the non-shadow area. The reason is that the processing algorithm may over-adjust the brightness and contrast of the entire image without accurately distinguishing the shadow area from the non-shadow area. Therefore, the present invention uses a multi-scale channel attention mechanism and a shadow interaction attention mechanism to effectively model the global contextual correlation between the shadow area and the non-shadow area, and finally can accurately identify and remove the shadow area in the image while maintaining the illumination consistency of the image.

[0130] In summary, although existing image shadow removal technologies provide effective means for processing shadows, they need to be optimized and improved without damaging other parts of the image. The method provided by the present invention has significant practical value and broad application prospects in achieving high-quality image restoration and visual perception improvement.

[0131] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A single image shadow removal method based on global context perception, characterized in that: include: S1, preprocess the original data set; S2, construct encoder and decoder using multi-scale channel attention mechanism; S3. Building a Transformer-based shadow removal network based on the encoder and decoder. The shadow removal network introduces a shadow model based on Retinex theory, and introduces a shadow interaction module and a shadow interaction attention mechanism at the bottleneck stage of the shadow removal network. S4, train the shadow removal network using the L1 loss function; S5. Perform image shadow removal based on the trained shadow removal network.

2. The single image shadow removal method based on global context perception according to claim 1, characterized in that: Step S1 specifically includes: The input images are obtained from the original dataset, and all input images are normalized to a uniform size and color space, where the image size is 256×256 and the color space is converted from BGR format to RGB format. Then, data diversity is increased through data augmentation techniques, including random rotation, flipping, and cropping of images.

3. The single image shadow removal method based on global context perception according to claim 1, characterized in that: The shadow model based on Retinex theory decomposes the image into two components: illumination and reflectivity. The illumination component L represents the part of the image that changes due to changes in lighting, and the reflectivity component R represents the color and characteristics of the object itself. The shadow removal task is to adjust the illumination and color of the image to remove the shadow noise from the image I. s Recover the bottom shadow-free image I sf , and then define the shadow-free image as I sf =L sf ⊙R, for shadow removal, the formula of the Retinex-based shadow model is as follows: I s =I m ⊙L s ⊙R+(1-I m )⊙L ns ⊙R; I m Represents the mask image, the shadow area is 1, the non-shadow area is 0, L s , L ns and L sf Represent the illumination of the shadow area, non-shadow area and shadow-free image respectively.

4. The single image shadow removal method based on global context perception according to claim 1, characterized in that: The encoder is composed of three BasicFormer layer sequences and corresponding downsampling layers, and the decoder is composed of three BasicFormer layer sequences and corresponding upsampling layers, wherein the BasicFormer layer contains a custom Transformer, and the Transformer contains a CATransformer module; First, given a shadow input I s ∈R 3×H×W , the corresponding shadow mask I m ∈R H×W , then an input projection layer converts the size and number of channels of the input data into the embedding dimension required by the shadow removal network at the initial stage of the network. The projection layer applies a linear projection LinearProj(·) to obtain the underlying feature embedding of the input, denoted as X0∈R C×H×W , where C is the embedding dimension, H and W are the height and width of the image respectively, and then X0 is input into the encoder to superimpose multi-scale global features. Each CATansformer module includes a channel attention module, which is composed of a channel attention layer; The channel attention layer first performs global average pooling on the spatial feature information of each channel of the input, that is, compressing it into a single scalar value. The formula is as follows: Where S c is the global average of the global average pooling operation on the cth channel, is the eigenvalue of the cth channel at position (i, j); Each CATansformer module also includes two convolutional layers, which are used to learn the weight of each channel from the features obtained by global average pooling. The first convolutional layer uses a 1×1 convolution kernel and then uses the ReLU activation function. This activation layer is used to introduce nonlinearity. The second convolutional layer uses a 1×1 convolution kernel to restore the channel dimension by expanding the channel and output the attention weight that each channel should have. Finally, the Sigmoid activation function is used to compress the output of the convolutional layer to the range of [0,1] to indicate the importance of each channel. The formula is as follows: The c =σ(W2·ReLU(W1·S c +b1)+b2); Where W1 and b1 are the weight and bias of the first convolutional layer, W2 and b2 are the weight and bias of the second convolutional layer, and the attention weight U of each channel is obtained. c , U c It is used to adjust the activation strength of the corresponding channel in the original input feature map. The formula is as follows: in is the value of channel c at position (i, j) of the adjusted output feature map; The channel attention module first applies the same convolution operation f twice to the feature map X of a given input. conv The output is X', that is, X' = f conv (f conv (X)), where f conv (X) = PReLU(W·X+b), where W and b are the weight and bias of the convolutional layer respectively; Then, the channel attention layer is used to assign attention weights to each channel in X'. Finally, the original feature map input X is added back to the result Y adjusted by the channel attention layer through the residual connection, and the output is Z. The formula is as follows: Where PReLU(·) is an activation function; The CATansformer module first performs layer normalization on the input feature map X, then compresses the spatial information in sequence through the channel attention module, and then performs a residual connection with the input X. The output feature map is The formula is as follows: LN(·) represents layer normalization; Then, the obtained feature map Perform layer normalization operations, capture long-range correlations through feed-forward MLP, and finally sum up with the input Perform residual connection, the formula is as follows: GELU(·) represents the Gaussian error linear unit activation layer, MLP(·) represents the multi-layer perceptron, and the multi-layer perceptron uses the locally enhanced feedforward network LeFF; The depth of the network is formed by repeating the CATansformer module twice at multiple levels. The formula is as follows: X out =Block2(Block1(X in )), where Block1(·) and Block2(·) represent two identical CATransformer module processing operations. The input low-level feature embedding X0 is processed by three BasicFormer layers in the encoder to obtain hierarchical features {X1, X2, X3}, where L∈{1,2,3}; Each BasicFormer layer in the encoder is processed by a downsampling operation. The downsampling operation halves the height and width of the feature map through a convolution layer with a kernel size of 4×4 and a stride of 2. The input of each CATransformer module in the decoder is connected through a skip connection to connect the corresponding feature X L The upsampling operation increases the height and width of the feature map to twice the input size by transposing the convolution layer, that is, the convolution kernel size is 2×2 and the step size is 2. The final output of the decoder is Then, the features finally obtained by the decoder are mapped back to the spatial size and number of channels of the original data through an output projection layer, which applies a linear projection LinearProj(·) to obtain the output I output , where I output ∈R 3×H×W , the shadow removal network finally adds the corrected information back to the original shadow input image I through the residual connection s , the output is a clear image after the shadow is removed The formula is:

5. The single image shadow removal method based on global context perception according to claim 1, characterized in that: The bottleneck stage of the shadow removal network consists of a sequence of BasicFormer layers. Two layers of SIMTransformer are used in the BasicFormer layer structure. SIMTransformer combines the shadow interaction attention mechanism to optimize and adjust the shadow features in the image processed by the shadow removal network. First, the feature map of the downsampling layer output after the third layer encoder After LN(·) layer normalization, SIMTransformer uses the channel attention module to reweight the channels of the feature map, and then splits the reweighted channel attention feature map into a non-overlapping window sequence X. The size of each window is P×P. is the total number of windows, and then the shadow interaction attention mechanism is performed on the flat features in each window, and then the residual connection is performed with the window sequence X, and the output feature map is The formula is as follows: Where ∑ represents the patch correlation map, which is used to describe the correlation between features in each window, CAB represents the channel attention module, and SIA represents the shadow interaction attention mechanism; For a position vector in the feature map Y output by the downsampling layer after the third encoder This position vector corresponds to two small blocks in the input shadow image, where i and j are the position indexes of the small window in the feature map, and the value range of i and j is 0 to N-1. The mask map corresponding to position i and position j in the feature map is M i and M j , then the correlation graph between windows ∑ ij The calculation formula is as follows: Where ⊕ represents the XOR operation; Next, the attention map is reweighted by using the correlation map ∑ between small blocks, as shown below: Where σ∈(0,1) is used to adjust the weights of shadow-shadow pairs and non-shadow-non-shadow pairs, d is a scaling parameter; Q, K, V represent the query vector, key vector, and value vector of the input feature map X, respectively; Then the output feature map is transformed through SIMTransformer Perform layer normalization operations, capture long-range correlations through feed-forward MLP, and finally sum up with the input Perform residual connection, the formula is as follows: Where LN(·) represents layer normalization, GELU(·) represents Gaussian error linear unit activation layer, MLP(·) represents multi-layer perceptron, and multi-layer perceptron uses a locally enhanced feedforward network LeFF.

6. The single image shadow removal method based on global context perception according to claim 1, characterized in that: The shadow removal network is trained using the L1 loss function as follows: in is the output image, I gt is a true shadow-free image.

7. The single image shadow removal method based on global context perception according to claim 1, characterized in that: In step S5, before performing image shadow removal according to the trained shadow removal network, the method further includes: testing the trained shadow removal network using two public data sets, ISTD and SRD, to evaluate the performance of the shadow removal network; The performance of the shadow removal network is evaluated using performance evaluation indicators, including peak signal-to-noise ratio, structural similarity index, and root mean square error; The peak signal-to-noise ratio is calculated as follows: The root mean square error is calculated as follows: Where PSNR stands for peak signal-to-noise ratio, RMSE stands for root mean square error, which is the square root of the average of the squared values ​​of the difference between the actual value and the predicted value, and MAX I is the maximum pixel value of the real image I. For normalized images ranging from 0 to 1, MAX I =1; I(i,j) is the pixel value of the real image at position (i,j), K(i,j) is the pixel value of the processed image at position (i,j), m and n are the width and height of the image respectively; The structural similarity index is calculated as follows: In the formula, SSIM stands for structural similarity index, which is an indicator to measure the similarity of image visual effects. It is used to measure the similarity of structural information, brightness information and contrast information of two images. x and μ y is the average of the x and y images, and is the variance of the x and y images, σ xy is the covariance of the x and y images, and c1 and c2 are two small constants used to maintain numerical stability.

8. A computer device, comprising a memory, wherein the memory stores program instructions, characterized in that: When the program instructions are executed, the single image shadow removal method based on global context perception as described in any one of claims 1 to 7 is executed.

Images