Deep learning single image de-shadowing method and system based on multi-scale retinal decomposition

Abstract

The application discloses a kind of deep learning single image shadow removal method and system based on multiscale retina decomposition, the method includes: obtaining shadow image data set and pre-processing;The shadow image in the shadow image data set after pre-processing is input into the convolution layer in image shadow removal depth neural network, and the shallow feature of shadow image is extracted;Shallow feature is input into the encoder-decoder in image shadow removal depth neural network, and under the guidance of each scale shadow mask, shadow feature is repaired by mask-guided shadow removal module, and then the shadow-removed image is obtained after convolution layer;Image shadow removal depth neural network is trained using the shadow image data set after pre-processing;The image to be removed shadow is input into trained image shadow removal depth neural network, and the shadow-removed image is obtained.The application can effectively remove shadow from single image by using image shadow removal depth neural network, while retaining the feature information of non-shadow area.

Description

Technical Field

[0001] This invention belongs to the technical field of image processing, and relates to a deep learning-based single-image shadow removal method, system, computer device, and computer-readable storage medium based on multi-scale retinal decomposition. Background Technology

[0002] Shadows in an image are defined as areas of reduced light intensity caused by objects blocking a light source, resulting in decreased brightness and potential color changes. Shadows in images not only detract from their aesthetic quality but also degrade the performance of computer vision algorithms in related applications. Specifically, they can blur object features, trigger false positives in detection tasks, and cause inconsistent lighting. Image shadow removal aims to correct the lighting within these shadowed areas while preserving the original colors and details of the entire image. This technique, as an important preprocessing step, is widely used in various fields, including but not limited to object detection, medical imaging, augmented reality, and satellite imaging. Therefore, research on image shadow removal has significant application value.

[0003] Effectively removing shadows from an image is challenging, primarily due to the complexity of lighting conditions and scene composition. Key issues include the blurring of shadows from other dark areas, the need to handle soft and hard shadows, and preserving original colors and textures. Different lighting conditions, multiple light sources, mutual reflections, and the non-uniformity of shadows further complicate the problem. Finally, another significant challenge is maintaining consistent global illumination when adjusting the lighting in shadow areas, as any oversight can lead to artifacts after removal, such as anomalies along shadow boundaries.

[0004] Traditional image shadow removal methods primarily rely on handcrafted discriminative priors to detect shadows and restore illumination in shadowed areas. These methods mainly focus on utilizing different properties of shadowed and unshadowed areas, such as edges, color, intensity, geometry, and texture. However, the overly simplistic assumptions behind these priors limit their applicability to complex real-world scenarios.

[0005] In recent years, deep learning has made substantial contributions to advancements in image shadow removal. Convolutional neural network-based methods, typically trained end-to-end, focus on directly mapping shadowed images to their shadow-free counterparts. Many techniques utilize generative adversarial networks to detect and subsequently recover shadowed regions. Furthermore, deep learning has made significant progress in detecting shadows in images, which can be used as a preprocessing module for subsequent shadow removal.

[0006] While deep learning has significantly enhanced the effectiveness and robustness of shadow removal techniques, current methods fail to fully leverage the underlying lighting physics of shadows and the entire image. This limitation impacts their performance in complex lighting environments. Although end-to-end models offer convenient solutions, they often fail to generalize well to diverse real-world lighting conditions. Most existing physically-based models, while incorporating some physical properties of shadows, tend to rely on overly simplistic assumptions, such as constant illumination in shadow regions, and resort to basic linear transformations for lighting adjustments. Consequently, these models often produce unwanted artifacts along shadow boundaries and generate unrealistic perceptions in images captured under complex lighting conditions. Summary of the Invention

[0007] For shadow images captured under complex lighting conditions, this invention provides a deep learning-based single-image shadow removal method, system, computer device, and computer-readable storage medium based on multi-scale retinal decomposition. It can handle spatial variations in shadow illumination across multiple scales, ensure consistency of boundaries between shadow and non-shadow areas, and avoid common defects such as over-brightening and erroneous shadow removal.

[0008] The first objective of this invention is to provide a deep learning-based single-image shadow removal method based on multi-scale retinal decomposition.

[0009] The second objective of this invention is to provide a deep learning single-image shadow removal system based on multi-scale retinal decomposition.

[0010] A third objective of this invention is to provide a computer device.

[0011] A fourth objective of this invention is to provide a computer-readable storage medium.

[0012] The first objective of this invention can be achieved by adopting the following technical solution:

[0013] A shadow image dataset is acquired and preprocessed; the samples in the shadow image dataset include shadow images, corresponding shadowless images, and shadow masks.

[0014] The shadow images in the preprocessed shadow image dataset are used to remove the convolutional layers in the deep neural network and extract the shallow features of the shadow images.

[0015] The shallow features are input into a symmetrical U-shaped encoder-decoder in a deep neural network for image shadow removal, which includes a combination of multi-scale mask-guided shadow removal modules. Under the guidance of shadow masks at each scale, the shadow features are repaired by the mask-guided shadow removal modules. The repaired features are then passed through a convolutional layer to obtain the shadow-removed image corresponding to the shadow image.

[0016] A deep neural network for image shadow removal was trained using a preprocessed shadow image dataset; pixel-level retinal decomposition loss was used to supervise the mask-guided shadow removal module.

[0017] The image to be shaded is input into a trained deep neural network for image shading removal to obtain the corresponding shaded image.

[0018] Furthermore, the codec employs a multi-level symmetric encoder-decoder architecture, where the i-th level contains N... i A mask-guided shadow removal module, and as i increases, N... i The corresponding increase is also necessary;

[0019] The restoration of shadow features via a mask-guided deshading module, guided by shadow masks at various scales, includes:

[0020] During the encoding stage, downsampling is used, in which the spatial dimension of the feature map is gradually halved with each downsampling operation, while the number of channels is doubled accordingly, in order to encapsulate and abstract important image features at different scales; during the decoding stage, upsampling is used, in which the spatial dimension of the feature map is gradually halved with each upsampling operation, while the number of channels is halved accordingly.

[0021] During the decoding stage, the features of the previous mask-guided deshading module in each mask-guided deshading module are concatenated with the corresponding features of the mask-guided deshading module in the encoder, and then convolution is performed. The feature map generated after convolution is used as the input feature of the current mask-guided deshading module.

[0022] The shadow mask is downsampled to the corresponding scale by max pooling, and used as the input feature of the mask-guided shadow removal module at the corresponding scale.

[0023] Furthermore, the mask-guided shadow removal module includes a shadow-aware channel attention module and a retinal decomposition-based shadow illumination module. The shadow-aware channel attention module is used to reweight feature channels and process channels rich in shadow features to refine shadow region features. The retinal decomposition-based shadow illumination module is used to refine shadow region features and improve the image quality of shadow region recovery by utilizing non-shadow illumination information.

[0024] Furthermore, the processing of input features by the shadow-aware channel attention module includes:

[0025] The input features are layer normalized and then mapped to the intermediate features V through two convolutions, where the first convolution is a standard convolution and the second convolution is a depthwise splitting convolution.

[0026] A shadow mask M of the same scale as the intermediate feature V is introduced, and the similarity between V and M is measured. Then, the shadow-aware channel attention vector is calculated by using global average pooling and Sigmoid activation in sequence.

[0027] The shadow perception channel attention vector is expanded to reweight each feature channel through element-wise multiplication, guiding the network to focus on channels rich in shadow features;

[0028] Finally, the channel information is fused through convolutional layers, and the module output is obtained through residual connections.

[0029] Furthermore, the processing of input features by the retinal decomposition-based shadow illumination module includes:

[0030] The input features are first normalized by layers, and then decomposed into illumination features and reflectance features by continuous parallel convolutional layers.

[0031] Guided by the shadow mask, the lighting features are divided into shadow area features and non-shadow area features;

[0032] Guided by the shadow mask, the brightness of the shadow area is adjusted using the brightness features of the non-shadow area; after adjusting the brightness of the shadow area, the modified shadow area features are merged with the non-shadow area features to synthesize the improved lighting features;

[0033] The improved lighting and reflectivity characteristics are combined, and the module output is obtained through residual connection.

[0034] Furthermore, the pixel-level retinal decomposition loss supervision of the mask-guided shadow removal module is equivalent to the pixel-level retinal decomposition loss supervision of the shadow illumination module based on retinal decomposition. The corresponding retinal decomposition loss function is as follows:

[0035] L retinex =L R +L LR

[0036] in:

[0037]

[0038]

[0039] Among them, L retinex L is the retinal decomposition loss function. R L is the loss function for constraining the reflectivity branch. LR The loss function is defined as the result of constraining the reconstruction of reflectivity and illumination features, where l is the number of layers in the network, and N is the number of layers in the network. i Let ω be the number of mask-guided deshading modules for the i-th layer. iFor balance coefficient, The ratio is 2 i-1 The downsampling operators, φ and ψ, are functions that map the modified illumination features and reflectance features to the image domain, respectively; and R represents the modified illumination and reflectance features of the j-th element in the mask-guided deshading module of the i-th layer, respectively. X This is a reflectance map obtained by applying the classic retinal decomposition method to a shadowless image X.

[0040] Furthermore, the loss function of the deep neural network for image shadow removal includes a local discriminative loss, specifically:

[0041]

[0042] in:

[0043]

[0044]

[0045] Among them, X and These are the unshaded image corresponding to the shaded image and the shaded image output by the network, respectively; s is the local window size when calculating the variance of the residual image E; W(i,j) is the variance of the residual image E with a local window size of s centered at pixel (i,j); and α is the balance coefficient.

[0046] Furthermore, the preprocessing of the shadow image dataset includes:

[0047] Randomly select samples from the shadow image dataset, and transform the images in the samples to expand the samples in the shadow image dataset; the transformation includes, but is not limited to, at least one of rotation, enlargement, or reduction;

[0048] Then, the images in the sample are cropped into image blocks of a fixed size;

[0049] The shadowed and unshadowed images in the cropped samples are normalized.

[0050] The second objective of this invention can be achieved by adopting the following technical solution:

[0051] A deep learning-based single-image shadow removal system based on multi-scale retinal decomposition, the system comprising:

[0052] The acquisition module is used to acquire a shadow image dataset and perform preprocessing; the samples in the shadow image dataset include shadow images, corresponding shadowless images, and shadow masks.

[0053] The extraction module is used to extract shallow features of shadow images from the preprocessed shadow image dataset by removing the convolutional layers of the deep neural network in the shadow image input image.

[0054] The repair module is used to input the shallow features into the image shadow removal deep neural network. The symmetrical U-shaped encoder and decoder, which includes a combination of multi-scale mask-guided shadow removal modules, repairs the shadow features through the mask-guided shadow removal modules under the guidance of shadow masks at each scale. After the repaired features are passed through a convolutional layer, the shadow-removed image corresponding to the shadow image is obtained.

[0055] The training module is used to train the deep neural network for image shadow removal using a preprocessed shadow image dataset; among which, the mask-guided shadow removal module is supervised by pixel-level retinal decomposition loss.

[0056] The removal module is used to input the image with the shadow to be removed into a trained deep neural network for image shadow removal, and obtain the corresponding shadow-removed image.

[0057] The third objective of this invention can be achieved by adopting the following technical solution:

[0058] A computer device includes a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the above-described deep learning single-image shadow removal method based on multi-scale retinal decomposition.

[0059] The fourth objective of this invention can be achieved by adopting the following technical solution:

[0060] A computer-readable storage medium storing a program that, when executed by a processor, implements the above-described deep learning single-image shadow removal method based on multi-scale retinal decomposition.

[0061] The present invention has the following advantages over the prior art:

[0062] This invention provides a deep learning-based single-image shadow removal method and system based on multi-scale retinal decomposition. The method involves acquiring a shadow image dataset and preprocessing it. The dataset includes shadow images, corresponding shadow-free images, and shadow masks. The shadow images from the preprocessed dataset are input into convolutional layers of a deep neural network for shadow removal to extract shallow features. These shallow features are then input into a symmetrical U-shaped encoder / decoder within the deep neural network, which combines multi-scale mask-guided shadow removal modules. Guided by shadow masks at various scales, the mask-guided shadow removal modules repair shadow features. The repaired features are then passed through convolutional layers to obtain the shadow-free image. The deep neural network is trained using the preprocessed shadow image dataset, with pixel-level retinal decomposition loss supervision applied to the mask-guided shadow removal modules. Finally, the image to be shadow-removed is input into the trained deep neural network to obtain the corresponding shadow-free image. The constructed deep neural network for image shadow removal can handle spatial variations in shadow illumination across multiple scales. It effectively removes shadows from single images while preserving the feature information of non-shadowed areas. Pixel-level retinal decomposition loss supervision is applied to the mask-guided shadow removal module to ensure correct retinal decomposition of image features. This method is applicable to shadow removal tasks for various types of shadowed images. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0064] Figure 1 This is a flowchart of a deep learning single-image shadow removal method based on multi-scale retinal decomposition according to Embodiment 1 of the present invention;

[0065] Figure 2 This is a schematic diagram of the image shadow removal deep neural network of Embodiment 1 of the present invention, wherein (b) is a schematic diagram of the shadow illumination module based on retinal decomposition, and (c) is a schematic diagram of the shadow perception channel attention module;

[0066] Figure 3 This is a partial sample of the shadow image dataset in Embodiment 1 of the present invention;

[0067] Figure 4 This is an image of adjusting lighting in a shadow area according to Embodiment 1 of the present invention;

[0068] Figure 5 This is a rendering of a shadow removal example from Embodiment 1 of the present invention;

[0069] Figure 6 This is a block diagram of the deep learning single-image shadow removal system based on multi-scale retinal decomposition according to Embodiment 2 of the present invention.

[0070] Figure 7 This is a structural block diagram of the computer device according to Embodiment 3 of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be understood that the specific embodiments described are merely used to explain this application and are not intended to limit this application.

[0072] Example 1:

[0073] like Figure 1 As shown, the deep learning-based single-image shadow removal method based on multi-scale retinal decomposition provided in this embodiment specifically includes the following steps:

[0074] S101. Obtain the shadow image dataset.

[0075] We collected publicly available real-world shadow datasets, such as the Image Shadow Triplets Dataset (ISTD), the Adjusted Image Shadow Triplets Dataset (ISTD+), and the Shadow Removal Dataset (SRD). These datasets contain thousands of triplet images, including shadowed images, corresponding clean images without shadows, and binary shadow masks. Some samples are available for reference. Figure 2 .

[0076] S102. Preprocess the shadow image dataset.

[0077] The images are preprocessed, such as randomly selecting a group of images from the training dataset, transforming the image group, such as rotating, scaling, slicing, normalizing, etc., and then randomly sorting the real shadow images to form a training set.

[0078] After collecting the shadow image data, directly inputting the training samples into the network for training will cause two main problems: one is that the image size is too large, requiring excessive computational resources to process a single image during training; the other is that the pixel values ​​of the images range from [0, 255], and directly using inputs within this range is not conducive to network convergence. Therefore, the following three preprocessing steps are required before inputting the images into the network:

[0079] (1) Randomly select a group of images from the training dataset and transform the image group, such as by rotating, enlarging or shrinking. This process is equivalent to expanding the training dataset. In reality, due to various reasons, the dataset that can be collected is often very small, and the lack of a training set is very detrimental to network training. Therefore, such a preprocessing step is needed to increase the dataset so that the trained network can reduce the risk of overfitting.

[0080] (2) Randomly cut image pairs from the image group to obtain image blocks of size 256*256. This is because the original image size is large. Under the same computing resources, the image after cutting can increase the number of batches for each training, which is beneficial for networks optimized using the batch gradient descent algorithm.

[0081] (3) The image pairs are regularized because the range of image values ​​is 0-255. The network learns the mapping from [0,255] to [0,255] directly, which will converge more slowly than the mapping from [0,1] to [0,1]. Therefore, normalization can be used.

[0082] S103. Construct a deep neural network for image shadow removal.

[0083] like Figure 3 As shown, the network structure includes two convolutional layers and a symmetrical U-shaped encoder-decoder that combines multi-scale mask-guided shadow removal modules. First, the preprocessed shadow image is passed through a convolutional layer to extract shallow features. Then, it is input into the encoder-decoder containing the multi-scale mask-guided shadow removal modules. Guided by shadow masks at various scales, shadow features are repaired within the feature domain. Finally, the image is passed through a convolutional layer to output the network's shadow removal result.

[0084] Specifically, both convolutional layers are 3×3 convolutional layers.

[0085] Specifically, the codec employs a 4-stage symmetric encoder-decoder neural network. The i-th stage of this codec structure contains N... iThe system employs a mask-guided shadow removal module, with N1 to N4 set to 2, 4, 6, and 8 respectively, and a channel cardinality C of 32. During the encoding phase, it starts with a high-resolution feature map of size C×H×W, containing H×W pixels and C channels. For multi-scale analysis, a pixel unshuffle downsampling operation is used, where the spatial dimension of the feature map is gradually halved with each downsampling operation, while the number of channels is correspondingly doubled. This hierarchical downsampling encapsulates and abstracts important image features at different scales. Conversely, the decoder iteratively restores the lower-resolution input features to a higher-resolution representation using a pixel shuffle upsampling operation. In each mask-guided shadow removal module of the decoder, the features from the previous module are concatenated with the corresponding features from the encoder, and then a 1×1 convolution is performed. The resulting feature map is used as the input feature. Furthermore, max pooling is used to downsample the shadow mask to the corresponding scale, which is then used as the input to the mask-guided shadow removal module at that scale.

[0086] Specifically, the mask-guided deshading module includes a shadow-aware channel attention module (SACA) and a retinal decomposition-based shadow illumination module (RDSL), wherein:

[0087] (1) Shadow-aware channel attention module (SACA).

[0088] It is specifically designed to work collaboratively within the concept of retinal decomposition in the feature channels, to reweight feature channels, focusing on channels rich in shadow features to refine shadow region features.

[0089] like Figure 3 As shown in (c), the specific processing procedure is as follows:

[0090] (1-1) First, the input feature tensor is normalized by layer, and then it is mapped to the intermediate feature V through two convolutional layers. The first convolutional kernel size is 1, the second convolutional kernel size is 3, the padding is 1, and the stride is 1.

[0091] (1-2) Introduce a shadow mask M of the same size, use element-wise multiplication to measure the similarity between V and M, followed by global average pooling (GAP) and sigmoid activation to compute the shadow-aware channel attention vector;

[0092] (1-3) The channel attention vector is expanded to reweight each feature channel through element-wise multiplication, guiding the network to focus on channels rich in shadow features;

[0093] (1-4) Finally, the channel information is further fused by 1×1 convolution, and the module output is obtained by residual connection.

[0094] (2) Shadow illumination module based on retinal decomposition (RDSL).

[0095] It utilizes the concept of retinal decomposition and, based on refinement features, further improves the image quality of shadow area restoration using non-shadow illumination information.

[0096] like Figure 3 As shown in (b), the input feature tensor is first normalized through layers, and then decomposed into illumination features L using consecutive parallel convolutional layers. F and reflectivity characteristics R F Guided by the shadow mask M, the lighting features are divided into shadow and non-shadow regions; the lighting features of the non-shadow regions are then used to adjust the brightness of the shadow regions. After brightness adjustment in the shadow regions, the improved shadow region features are... Features of non-shaded areas Merging and synthesizing improved lighting features Finally, the improved lighting features and reflectivity characteristics R F These features are then combined and enhanced through residual connections. The specific processing procedure is as follows:

[0097] (2-1) First, the input feature tensor is normalized through layers to obtain F. Then, using the idea of ​​retinal decomposition, it is divided into two parallel branches. One is the illumination branch L. F This represents the light illuminating the scene. Another branch of reflectivity, R... F It captures the inherent properties of an object.

[0098] (2-2)L F and R F F is obtained by decomposing it into independent convolutional layers, which are passed through 1×1 convolutional layers and 3×3 depth separation convolutional layers in sequence. The padding of the 3×3 depth separation convolutional layer is 1, and the stride is 1.

[0099] (2-3) In the lighting branch, introducing a shadow mask M can effectively mask the lighting feature L. F Features of the shaded area Non-shaded area features

[0100]

[0101]

[0102] The process of obtaining the shadow area by processing the shadow image can be found in [reference needed]. Figure 4 .

[0103] (2-4) Derive two types of coefficients from the non-shaded region, namely the global coefficient W. F and local coefficient BF To handle shadow areas, modify the lighting as follows:

[0104]

[0105] The improved shadow area features;

[0106] (2-5) Global coefficient W F Features of non-shadowed regions are obtained through average pooling of the shadow mask. The global illumination information is integrated, and then two 1×1 convolutions, a ReLU activation function, and a spreading operation are performed to obtain the local coefficients B. F The shadow degradation problem is addressed in a differentiated manner by using horizontal / vertical map layers (H / V-MAP alternating between modules), followed by two 1×1 convolutions, Tanh activation, and a spreading operation. The map layers and H / V-MAPs are defined as follows:

[0107]

[0108]

[0109]

[0110] (2-6) After adjusting the lighting in the shadow area, combine the improved shadow area features. Features of non-shaded areas To synthesize modified lighting features

[0111] (2-7) Finally, the modified lighting features and reflectivity characteristics R F The image feature domain is restored through dot product operation, and then the channel information is fused through a 1×1 convolution. Finally, the enhanced features are reconstructed through residual connections.

[0112]

[0113] S104. Train the deep neural network for image shadow removal using the preprocessed shadow image dataset.

[0114] The total loss function used in the network model includes the pixel fidelity loss function, the local discriminant loss function, and the retina decomposition loss function, with the specific formula as follows:

[0115] L = L char +λ1L ld +λ2L retinex

[0116] Wherein, λ1 and λ2 are hyperparameters, both of which are set to 1 in this embodiment.

[0117] (1) Pixel fidelity loss function.

[0118] The formula is:

[0119]

[0120] Among them, X and These represent the image without shadows and the shadow-removed image output by the network, respectively. ∈ is a perturbation coefficient that effectively prevents model overfitting; in this embodiment, ∈ is set to 10. -3 L char The network's final output of the repaired image can be supervised, thereby better optimizing the network parameters.

[0121] (2) Local discriminant loss function.

[0122] The formula is:

[0123]

[0124] Where W represents the local variance of the residual map E between the two images, and σ is the global variance used for scaling. W is defined as:

[0125]

[0126] in In this embodiment, s and α are set to 7 and 5, respectively. The local discrimination weight W is used to penalize inconsistencies between the restored image and the real image, especially at shadow boundaries. The global coefficient σ captures the overall error and helps preserve color and texture. In the context of shadow removal, L... ld The problem of restoring shadow boundaries can be solved by utilizing local statistics.

[0127] (3) Retinal decomposition loss function.

[0128] The formula is:

[0129] L retinex =L R +L LR

[0130] in:

[0131]

[0132]

[0133] Where l is the number of layers in the network, and N i ω is the number of mask-guided deshading modules in the i-th layer. i It is the balance coefficient, ↓r It is a downsampling operator with a scale of r, where φ and ψ are functions that map illumination features and reflectance features to the image domain, respectively. and R represents the illumination and reflectance characteristics of the j-th element in the mask-guided deshading module of layer i. X This represents the reflectance map obtained by applying the classic retinal decomposition method to a shadowless image X. L retinex The loss function can effectively guide the model to embed retinal decomposition in the feature domain.

[0134] This embodiment introduces local discriminative loss and retinal decomposition loss into the loss function to guide the model to focus on shadow boundaries and ensure the consistency of boundaries between shadow and non-shadow regions, thus avoiding common defects such as over-brightening and incorrect shadow removal.

[0135] S105. Input the image to be shaded into the trained image shading removal deep neural network, and output the corresponding shaded image.

[0136] After cropping and normalizing the image to be shaded, the processed image is input into a trained deep neural network for image shading removal to obtain the corresponding shading-removed image.

[0137] This embodiment utilizes a trained deep neural network for image shadow removal to process a portion of the shadowed images in the dataset. The resulting shadow removal results can be found in [link to documentation]. Figure 5 .

[0138] The method provided in this implementation not only outperforms traditional methods in both qualitative and quantitative metrics, but also demonstrates advantages in handling various shadow shapes, sizes, and intensities when compared with clean, shadow-free images in the benchmark dataset.

[0139] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium.

[0140] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the order of execution of the described steps may be changed. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0141] Example 2:

[0142] like Figure 6As shown, this embodiment provides a deep learning single-image shadow removal system based on multi-scale retinal decomposition. The system includes an acquisition module 601, an extraction module 602, a repair module 603, a training module 604, and a removal module 605, wherein:

[0143] The acquisition module 601 is used to acquire a shadow image dataset and perform preprocessing; the samples in the shadow image dataset include shadow images, corresponding shadowless images, and shadow masks.

[0144] The extraction module 602 is used to extract shallow features of the shadow images from the preprocessed shadow image dataset by removing the convolutional layer of the deep neural network from the shadow images in the input image.

[0145] The repair module 603 is used to input the shallow feature image shadow removal deep neural network into the symmetrical U-shaped encoder and decoder containing multi-scale mask-guided shadow removal modules. Under the guidance of shadow masks at each scale, the shadow features are repaired by the mask-guided shadow removal modules. After the repaired features are passed through a convolutional layer, the shadow-removed image corresponding to the shadow image is obtained.

[0146] Training module 604 is used to train a deep neural network for image shadow removal using a preprocessed shadow image dataset; wherein, the mask-guided shadow removal module is supervised by pixel-level retinal decomposition loss.

[0147] The removal module 605 is used to input the image to be shaded into a trained image shading removal deep neural network to obtain the corresponding shading-removed image.

[0148] The specific implementation of each module in this embodiment can be found in Embodiment 1 above, and will not be repeated here. It should be noted that the system provided in this embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0149] Example 3:

[0150] This embodiment provides a computer device, which can be a computer, such as... Figure 7As shown, the processor 702, memory, input device 703, display 704, and network interface 705 are connected via system bus 701. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 706 and internal memory 707. The non-volatile storage medium 706 stores the operating system, computer programs, and database. The internal memory 707 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the processor 702 executes the computer programs stored in the memory, it implements the deep learning single-image shadow removal method based on multi-scale retinal decomposition in Embodiment 1 described above, as follows:

[0151] A shadow image dataset is acquired and preprocessed. The dataset includes shadow images, corresponding shadow-free images, and shadow masks. The shadow images from the preprocessed dataset are input into the convolutional layers of a deep neural network for shadow removal to extract shallow features. These shallow features are then input into a symmetrical U-shaped encoder / decoder within the deep neural network, which combines multi-scale mask-guided shadow removal modules. Guided by shadow masks at various scales, the mask-guided shadow removal modules repair shadow features. The repaired features are then passed through convolutional layers to obtain the shadow-free image. The deep neural network for shadow removal is trained using the preprocessed shadow image dataset, with pixel-level retinal decomposition loss supervision applied to the mask-guided shadow removal modules. The image to be shadow-removed is then input into the trained deep neural network to obtain the corresponding shadow-free image.

[0152] Example 4:

[0153] This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the deep learning single-image shadow removal method based on multi-scale retinal decomposition in Embodiment 1 above, as follows:

[0154] A shadow image dataset is acquired and preprocessed. The dataset includes shadow images, corresponding shadow-free images, and shadow masks. The shadow images from the preprocessed dataset are input into the convolutional layers of a deep neural network for shadow removal to extract shallow features. These shallow features are then input into a symmetrical U-shaped encoder / decoder within the deep neural network, which combines multi-scale mask-guided shadow removal modules. Guided by shadow masks at various scales, the mask-guided shadow removal modules repair shadow features. The repaired features are then passed through convolutional layers to obtain the shadow-free image. The deep neural network for shadow removal is trained using the preprocessed shadow image dataset, with pixel-level retinal decomposition loss supervision applied to the mask-guided shadow removal modules. The image to be shadow-removed is then input into the trained deep neural network for shadow removal to obtain the corresponding shadow-free image.

[0155] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0156] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A deep learning-based single-image shadow removal method based on multi-scale retinal decomposition, characterized in that, The method includes: A shadow image dataset is acquired and preprocessed; the samples in the shadow image dataset include shadow images, corresponding shadowless images, and shadow masks. The shadow images in the preprocessed shadow image dataset are used to remove the convolutional layers in the deep neural network and extract the shallow features of the shadow images. The shallow features are input into a symmetrical U-shaped encoder-decoder in a deep neural network for image shadow removal, which includes multi-scale mask-guided shadow removal modules. Guided by shadow masks at each scale, the mask-guided shadow removal modules repair shadow features. The repaired features are then passed through convolutional layers to obtain the shadow-removed image corresponding to the original shadow image. The encoder-decoder employs a multi-level symmetrical encoder-decoder structure. Level includes A mask-guided shadow removal module, and as i increases... The corresponding increase is also necessary; A deep neural network for image shadow removal was trained using a preprocessed shadow image dataset; pixel-level retinal decomposition loss was used to supervise the mask-guided shadow removal module. The image to be shaded is input into a trained deep neural network for image shading removal to obtain the corresponding shaded image; The mask-guided deshading module includes a shadow perception channel attention module and a shadow illumination module based on retinal decomposition, wherein: The shadow-aware channel attention module processes input features including: The input features are normalized by layers, and then mapped to intermediate features through two convolutions. The first convolution is a standard convolution, and the second convolution is a depthwise splitting convolution; Introduction and intermediate features Shadow mask of the same size and measure and The similarity between them is then used to calculate the shadow-aware channel attention vector by sequentially applying global average pooling and Sigmoid activation. The shadow perception channel attention vector is expanded to reweight each feature channel through element-wise multiplication, guiding the network to focus on channels rich in shadow features; Finally, the channel information is fused through convolutional layers, and the module output is obtained through residual connections. The retinal decomposition-based shadow illumination module processes input features including: The input features are first normalized by layers, and then decomposed into illumination features and reflectance features by continuous parallel convolutional layers. Guided by the shadow mask, the lighting features are divided into shadow area features and non-shadow area features; Guided by the shadow mask, the brightness of the shadow area is adjusted using the brightness features of the non-shadow area; after adjusting the brightness of the shadow area, the modified shadow area features are merged with the non-shadow area features to synthesize the improved lighting features; The improved lighting and reflectivity characteristics are combined, and the module output is obtained through residual connection.

2. The deep learning single-image shadow removal method according to claim 1, characterized in that, The restoration of shadow features via a mask-guided deshading module, guided by shadow masks at various scales, includes: During the encoding stage, downsampling is used, in which the spatial dimension of the feature map is gradually halved with each downsampling operation, while the number of channels is doubled accordingly, in order to encapsulate and abstract important image features at different scales; during the decoding stage, upsampling is used, in which the spatial dimension of the feature map is gradually halved with each upsampling operation, while the number of channels is halved accordingly. During the decoding stage, the features of the previous mask-guided deshading module in each mask-guided deshading module are concatenated with the corresponding features of the mask-guided deshading module in the encoder, and then convolution is performed. The feature map generated after convolution is used as the input feature of the current mask-guided deshading module. The shadow mask is downsampled to the corresponding scale by max pooling, and used as the input feature of the mask-guided shadow removal module at the corresponding scale.

3. The deep learning single-image shadow removal method according to claim 1, characterized in that, The pixel-level retinal decomposition loss supervision of the mask-guided shadow removal module is equivalent to the pixel-level retinal decomposition loss supervision of the shadow illumination module based on retinal decomposition. The corresponding retinal decomposition loss function is as follows: ； in: ； ； In the formula, The loss function is the retinal decomposition function. The loss function for constraining the reflectivity branch, The loss function is used to constrain the reconstruction results of reflectivity features and illumination features. The number of layers in the network. For the first Number of mask-guided deshading modules in the layer. For balance coefficient, The ratio is downsampling operator, and These are functions that map the modified illumination and reflectance features to the image domain, respectively. and The first The first layer mask-guided deshading module The modified lighting characteristics and reflectivity characteristics of each, For images without shadows The reflectance map was obtained by applying the classic retinal decomposition method.

4. The deep learning single-image shadow removal method according to any one of claims 1 to 3, characterized in that, The loss function of the deep neural network for image shadow removal includes a local discrimination loss, specifically: ； in, For global variance and local variance used for scaling Defined as : ； ， ； In the formula, and These are the shadowless image corresponding to the shadowed image and the shadow-removed image output by the network, respectively. The size of the local window used to calculate the variance of the residual plot E; For the residual map E, in pixels The variance centered at a local window of size s; This is the balance coefficient.

5. The deep learning single-image shadow removal method according to any one of claims 1 to 3, characterized in that, The preprocessing of the shadow image dataset includes: Randomly select samples from the shadow image dataset, and transform the images in the samples to expand the samples in the shadow image dataset; the transformation includes, but is not limited to, at least one of rotation, enlargement, or reduction; Then, the images in the sample are cropped into image blocks of a fixed size; The shadowed and unshadowed images in the cropped samples are normalized.

6. A deep learning-based single-image shadow removal system based on multi-scale retinal decomposition, characterized in that, The system includes: The acquisition module is used to acquire a shadow image dataset and perform preprocessing; the samples in the shadow image dataset include shadow images, corresponding shadowless images, and shadow masks. The extraction module is used to extract shallow features of shadow images from the preprocessed shadow image dataset by removing convolutional layers in the deep neural network of the shadow image input image. The repair module is used to input the shallow features into the image shadow removal deep neural network. The symmetrical U-shaped encoder and decoder, which includes a combination of multi-scale mask-guided shadow removal modules, repairs the shadow features through the mask-guided shadow removal modules under the guidance of shadow masks at each scale. After the repaired features are passed through a convolutional layer, the shadow-removed image corresponding to the shadow image is obtained. The training module is used to train the deep neural network for image shadow removal using a preprocessed shadow image dataset; among which, the mask-guided shadow removal module is supervised by pixel-level retinal decomposition loss. The removal module is used to input the image with shadows to be removed into a trained deep neural network for image shadow removal, and obtain the corresponding shadow-removed image. The mask-guided deshading module includes a shadow perception channel attention module and a shadow illumination module based on retinal decomposition, wherein: The shadow-aware channel attention module processes input features including: The input features are normalized by layers, and then mapped to intermediate features through two convolutions. The first convolution is a standard convolution, and the second convolution is a depthwise splitting convolution; Introduction and intermediate features Shadow mask of the same size and measure and The similarity between them is then used to calculate the shadow-aware channel attention vector by sequentially applying global average pooling and Sigmoid activation. The shadow perception channel attention vector is expanded to reweight each feature channel through element-wise multiplication, guiding the network to focus on channels rich in shadow features; Finally, the channel information is fused through convolutional layers, and the module output is obtained through residual connections. The retinal decomposition-based shadow illumination module processes input features including: The input features are first normalized by layers, and then decomposed into illumination features and reflectance features by continuous parallel convolutional layers. Guided by the shadow mask, the lighting features are divided into shadow area features and non-shadow area features; Guided by the shadow mask, the brightness of the shadow area is adjusted using the brightness features of the non-shadow area; after adjusting the brightness of the shadow area, the modified shadow area features are merged with the non-shadow area features to synthesize the improved lighting features; The improved lighting and reflectivity characteristics are combined, and the module output is obtained through residual connection.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the deep learning single-image shadow removal method according to any one of claims 1 to 5.

Images