An image shadow removal method, device, computer equipment and storage medium

Abstract

The application discloses an image shadow removal method and device, computer equipment and a storage medium, which adopts a UNet network structure as an image shadow removal model, uses the image segmentation function of the UNet network to realize shadow detection, uses the self-encoding and decoding structure of the UNet network to realize image restoration, and combines a unique loss function designed by the application to simultaneously use multi-level and single-level, multi-channel and single-channel predicted images and label images for loss calculation in the training process, so that the features of the images with shadows are brought close to the features of the images without shadows and then restored, so that the image shadow removal can be completed at one time, and the image texture after the shadow removal is clear and has good detail performance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image shadow removal method, apparatus, computer device, and storage medium. Background Technology

[0002] Image shadow removal has important applications in computer vision and computer graphics. Currently, convolutional neural network technology is developing rapidly, so using end-to-end processing through convolutional neural networks to remove image shadows has become the mainstream solution.

[0003] However, most current image shadow removal schemes based on convolutional neural networks use VGG network structures. Due to the limited data feature extraction capabilities of VGG networks and the use of a small number of deep and shallow layer feature fusions, they are not very effective in image shadow removal. Summary of the Invention

[0004] Objective of the Invention: To overcome the shortcomings of existing technologies, this invention proposes an image shadow removal method, apparatus, computer device, and storage medium. This invention employs a UNet convolutional image segmentation network based on pixel-level semantic segmentation, utilizing extensive multi-layer deep and shallow feature fusion and an improved loss function. This results in a network model with powerful feature extraction and image restoration capabilities, achieving superior performance in image shadow removal.

[0005] Technical Solution: To achieve the above objectives, this invention proposes an image shadow removal scheme based on UNet. UNet is an image segmentation neural network with a self-encoding structure. The self-encoding / decoding structure implies image reconstruction capability, while image segmentation implies shadow detection capability. By approximating the detected shadowed features to features without shadows before reconstruction, image shadow removal can be completed in one step.

[0006] Based on the above principles, the present invention proposes the following solution:

[0007] In a first aspect, the present invention proposes an image shadow removal method, comprising the following steps:

[0008] (1) Construct training samples: Obtain multiple pairs of sample images. Each pair of sample images includes an image with only background and no shadow and an image with shadow on the same background.

[0009] (2) Construct a shadow image removal model based on UNet;

[0010] (3) Use the shadow image in each pair of samples as the input image and the shadow image as the label image to train the model; during the training process, use the loss function to calculate the loss of the predicted image and the corresponding label image, and then update the weight of each layer of the UNet network with gradient, so that the predicted image output by the upsampling part of each layer of the convolutional neural network in the UNet network and the single predicted image fused by the predicted image respectively approximate the label image, and at the same time, make the color space channel image of each predicted image and the overall predicted image fused by all color space channel images approximate the label image;

[0011] (4) After the shadow image removal model is trained, the shadow removal model is used to remove shadows from the input shadowed image.

[0012] Regarding the image shadow removal method, several alternative methods are provided below, but these are not intended as additional limitations on the overall solution described above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution described above, or multiple alternative methods can be combined with each other.

[0013] Optionally, in the image shadow removal method, the loss function used is:

[0014]

[0015] Where L represents the overall loss, N represents the number of layers in the convolutional neural network of the UNet network, and C represents the total number of RGB color space channels in the predicted image. Let represent the loss function used to calculate the loss of the i-th color space channel of the predicted image output by the n-th convolutional neural network. F represents the loss function used to calculate the loss of the predicted image output by the nth layer of the convolutional neural network. fuse This represents the loss function used to calculate the loss over the overall predicted image. and w fuse These are the weighting coefficients.

[0016] Optionally, the loss function is determined by... and F fuse It consists of three sub-loss functions, each of which is composed of a corresponding binary cross-entropy loss function and a mean squared error loss function, namely:

[0017]

[0018]

[0019]

[0020] Among them, Ψ boe Let Φ represent the binary cross-entropy loss function. mse Let w represent the mean squared error loss function. boe and w mse All are weighting coefficients; H and W represent the height and width of the image, respectively, (i,j) represents the pixel coordinates in the image, and P... G(i,j) and P S(i,j) These represent the color value of the pixel at coordinates (i,j) in the label image and the prediction image, respectively.

[0021] Optionally, the shadow image removal model is implemented using a U2Net network.

[0022] Secondly, the present invention proposes an image shadow removal device, comprising: an image input module, an image processing module, and an image output module; wherein,

[0023] The image input module is used to acquire input images and perform image preprocessing, and then input the preprocessed image into the image processing module;

[0024] The image processing module is used to remove shadows from the input image based on the image shadow removal method according to any one of claims 1 to 4;

[0025] The image output module is used to process the image output by the image processing module to meet the preset output image format requirements.

[0026] The image shadow removal device is further provided with several alternative methods below, but these are not intended as additional limitations on the overall solution above, but merely as further additions or preferences. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0027] Optionally, the device further includes an image display module for displaying an image after the shadow has been removed.

[0028] Thirdly, the present invention proposes a storage medium storing a computer program, which, when executed, implements the image shadow removal method.

[0029] Fourthly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the image shadow removal method.

[0030] Beneficial effects:

[0031] Compared to existing techniques for removing image shadows based on convolutional neural networks, this invention employs the UNet network structure as the image shadow removal model. It utilizes the image segmentation function of the UNet network to achieve shadow detection and the self-encoded decoding structure of the UNet network to achieve image restoration. Combined with the unique loss function designed in this invention, during the training process, multi-level and single-level, multi-channel and single-channel predicted images and label images are used simultaneously for loss calculation. The features of the detected shadowed image are moved towards the features of the shadowless image before restoration. In this way, image shadow removal can be completed in one step, and the image after shadow removal has clear texture and good detail. Attached Figure Description

[0032] Figure 1 The flowchart is as described in Example 1;

[0033] Figure 2 This is a diagram of the U2Net network structure involved in Example 1;

[0034] Figure 3 This is a structural diagram of the device involved in Example 2. Detailed Implementation

[0035] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0036] Example 1:

[0037] This embodiment proposes an image shadow removal method, the specific process of which is as follows: Figure 1 As shown, it mainly includes:

[0038] Step 1: Sample collection;

[0039] Step 2: Preprocessing and augmentation of sample data;

[0040] Step 3: Construct a convolutional neural network model based on image segmentation;

[0041] Step 4: Design the loss function;

[0042] Step 5: Divide the dataset into training and test sets;

[0043] Step 6: Load data into the convolutional neural network model for training and testing;

[0044] Step 7: Reasoning and Application.

[0045] Each step will be explained in detail below.

[0046] I. Sample Collection.

[0047] The sample collection was divided into online public dataset collection and offline manual on-site photography collection.

[0048] Most publicly available online shadow removal datasets contain an image with shadows (A), a shadow mask (B), and an image without shadows (C), such as ISTD and DESOBA.

[0049] Offline, real-world photography is conducted using a fixed tripod to stabilize the camera, ensuring that both shadowed and shadowless photos use the same parameters (e.g., position, angle, size, focal length, exposure, hue, etc.) and cover the same area. Areas not intended to be shadowed must not contain shadows; either the area in question is discarded, or all shadowed areas are processed to obtain the corresponding shadowless image C. Since shadow formation is affected by weather, a high-powered flashlight with adjustable lumens can be used in low-light conditions. Based on the different shadowed targets, photos containing shadows can be categorized into the following four types:

[0050] 1. A photograph containing shadows of movable objects without shadow sources (such as a photograph of a person's silhouette without a human figure);

[0051] 2. Photographs containing shadows of movable objects with shadow sources (such as photographs containing people and their shadows);

[0052] 3. Photographs containing shadows of immovable objects without shadow sources (e.g., photographs of building shadows without buildings themselves);

[0053] 4. Photographs containing shadows of immovable objects (such as photographs containing buildings and their shadows).

[0054] The acquisition method for Type 1 (shadowed photos of movable objects without shadow sources) is relatively simple. After setting the camera parameters, take a photo A of the shadowed area (usually on the ground) in real-world conditions. Since the source object of the shadow is movable, move the source object away and take another photo of the same area using the same camera parameters. This will yield a shadowless photo C. The publicly available shadow dataset ISTD belongs to this type.

[0055] The acquisition method for Type 2 (shadowed photographs with movable shadow source objects) is slightly more complex than that for Type 1. The difference lies in the fact that the photographed area includes not only the real scene and the shadowed area but also the shadow source object. First, photograph the area to obtain Photo A. Then, remove the shadow source object and photograph the same area again using the same camera parameters to obtain Shadowless Photo C. Since Shadowless Photo C does not contain the shadow source object, the shadow area can be manually labeled or automatically detected / segmented using existing artificial neural networks to obtain the corresponding shadow mask image B. Then, bitwise operations in the image matrix are used to synthesize the shadow source object from the shadowed image containing the shadow source object into the Shadowless Image C (the synthesis process will be described in detail in the data preprocessing section). Most samples in the publicly available shadow dataset DESOBA belong to this type.

[0056] For Type 3 (shadowed photographs of immovable objects without shadow sources), the acquisition method primarily employs simulated real-world scenes and lighting techniques. Simulating a real-world scene involves scaling down the actual scene to be photographed into a model (e.g., a detachable building model), making immovable shadow source objects movable. If the simulated real-world scene model is indoors or in a low-light environment, additional simulated lighting is required, achieved by using a high-powered flashlight with adjustable lumens to simulate sunlight. Subsequent sample acquisition / photographing steps can utilize the Type 1 method.

[0057] For type four (shadowed photos of immovable objects with shadow sources), the method of type three can be used to first simulate the real scene and lighting, and then the method of type two can be used for sample collection / photography.

[0058] II. Preprocessing and Augmentation of Sample Data

[0059] 1. Sample Inspection and Matching. Samples collected offline through real-world photography are matched against the descriptions in the photos. Photos with shadows are stored in directory X, and photos without shadows are stored in directory Z, using the same filename to maintain the match. Photos that do not match are discarded. Samples collected online also require inspection for each group. The criteria for inspection and matching are: photo A (with shadows) and photo C (without shadows) must be in the exact same scene area, with identical shooting position, angle, size, and focal length. Minor deviations in exposure and tone are acceptable (corrected in subsequent processes).

[0060] 2. Sample Annotation. For samples collected through offline real-world photography, the shaded areas of the samples will be annotated for subsequent correction and expansion operations. Annotation can be done manually using annotation software or automatically detected / segmented using existing artificial neural networks. The purpose is to generate a shadow mask image B (i.e., a mask image describing the area where the shadow is located, with the same size as the shadow-containing image A; the pixel value of the corresponding area in B for the shadow area in A is 255, and the pixel value of the non-shaded area is 0), which will then be placed in directory Y. For samples collected online, such as ISTD and DESOBA, since the image shadow mask image B is already included, they can be directly placed in directory Y.

[0061] 3. Sample Correction. Since the collected image samples, whether gathered online or captured manually in real-world settings, have a time difference between the two images (image A with shadow and image C without shadow), errors in lighting and tone are likely to occur. As long as these errors are not significant, correction can be performed. The principle is to use bitwise operations on the image matrix to cover the corresponding part of the image C without shadow with the shadowed portion of the image A, thus synthesizing a new image A' with shadow. This can be represented using set operations as follows:

[0062]

[0063] Where A′ is the synthesized image matrix with shadow, A is the image matrix with shadow, B is the shadow mask image matrix, C is the image matrix without shadow, i and j are matrix subscripts, and m and n are the image width and height or the number of rows and columns of the matrix.

[0064] 4. Data Augmentation by Scaling and Rotation. To improve the robustness of convolutional neural networks, samples can be scaled to different sizes and rotated to different angles to augment them. This operation is also called data augmentation. Typically, images are scaled to multiples of 8 while maintaining the same width and height, and the rotation results are saved every 5 degrees.

[0065] III. Constructing a Convolutional Neural Network Model Based on Image Segmentation

[0066] UNet is an autoencoder-based image segmentation neural network. The autoencoder-decoder structure means it has image reconstruction capabilities, while image segmentation means it has shadow detection capabilities. By approximating the features of the detected shadowed image to the features of the shadowless image before reconstruction, image shadow removal can be completed in one step.

[0067] The UNet series includes: UNet, UNet++, U2Net, UNet+++, AttUnet, TransUnet, SwinUnet, etc.

[0068] U2Net was chosen as an example here. It is a convolutional neural network for image segmentation in salient object detection, which can achieve pixel-level semantic segmentation. By collecting and preprocessing the dataset, and making minor modifications to the network and loss function, the dataset was loaded into the network for training and testing, achieving the desired results with clear textures and excellent detail.

[0069] like Figure 2 As shown, U2Net uses a total of 6 symmetrical convolutional neural network layers. Each layer contains two sub-convolutional neural networks: one for downsampling and one for upsampling. The 6th layer is a transition layer for high-dimensional feature extraction and upsampling (with only one sub-convolutional neural network). The sub-convolutional neural networks in each layer are similar to the outer layer structure, also having a U-shaped structure. This is where the name U2Net comes from, meaning U squared.

[0070] The U2Net model uses multi-level deep and shallow feature channel fusion once, symmetric deep and shallow feature channel fusion a total of 29 times, batch normalization (BatchNorm) a total of 112 times, convolutional layer (Conv2d) a total of 130 times, non-linear activation function (LeakyReLU) a total of 112 times, residual network 11 times, max pooling (MaxPool) a total of 33 times, downsampling 32 times, and upsampling 32 times.

[0071] Downsampling extracts deeper semantic features, while upsampling restores shallower contour features for image reconstruction. Channel fusion of the deep and shallow features obtained from downsampling and upsampling at each layer enhances the expressive power of the features, resulting in more complete contours and detailed textures. Each downsampling process includes a nested convolutional neural network and a max-pooling layer; the upsampling process is the reverse of downsampling, also containing a nested convolutional neural network and an upsampling function based on linear interpolation.

[0072] The number of layers in each sub-convolutional neural network varies depending on the feature state. When the features are relatively shallow, the sub-network has more layers. As features are extracted and the network becomes deeper, the number of layers gradually decreases. Each sub-network is a U-shaped residual network, which enhances the fluidity of features and solves the gradient vanishing problem. Similarly, deep and shallow channel fusion is used in symmetrical sub-layers to enhance the expressive power of features. Each sub-layer uses convolution, batch normalization, non-linear activation, max pooling operations, and corresponding non-linear interpolation functions for upsampling.

[0073] IV. Designing the Loss Function

[0074] This embodiment uses two common loss functions: cross-entropy and mean squared error. It's worth noting that both multi-layer and single-layer, multi-channel and single-channel predicted and labeled images are used for loss calculation. During training, the loss function is first used to calculate the loss, and then the weights of each layer are updated using gradients. This ensures that the predicted images output by the upsampled parts of each of the six convolutional neural network layers approximate the real labeled images. These six predicted images are then fused into a single predicted image, which is then approximated again by the real labeled image. Simultaneously, the RGB color space channels of the predicted image and the fused image are approximated by the labeled image.

[0075] The overall loss L is defined as:

[0076]

[0077] Where N=6, it means that loss calculation is performed on 6 layers of prediction images, and C=3, it means that loss calculation is performed on each channel of the RGB color space of the prediction image. and F fuse There are three sub-loss functions, where, Let represent the loss function used to calculate the loss of the i-th color space channel of the predicted image output by the n-th convolutional neural network. F represents the loss function used to calculate the loss for each predicted image layer (the image synthesized from RGB three channels). fuse This represents the loss function used to calculate the loss of the overall prediction map after fusing the channels of the 6-layer prediction map. and w fuse This represents the weight of the corresponding loss function.

[0078] Each sub-loss function F is composed of two loss functions, defined as follows:

[0079]

[0080]

[0081]

[0082] Among them, Ψ boe Let Φ represent the binary cross-entropy loss function. mse Let w represent the mean squared error loss function. boe and w mse All are weighting coefficients; H and W represent the height and width of the image, respectively, (i,j) represents the pixel coordinates in the image, and P... G(i,j) and P S(i,j) These represent the color values ​​of the pixel at coordinates (i,j) in the label image and the prediction image, respectively.

[0083] By calculating the loss and gradient between the predicted image and the real label image, the neural network learns its current learning state and future learning direction, and continuously optimizes the weights of each layer in this direction. Through continuous iterative training, the predicted shadowless image becomes more and more similar to the labeled shadowless image, ultimately achieving the effect of removing the shadow from the image.

[0084] V. The dataset is divided into training and test sets.

[0085] In the collected dataset, based on the filenames in directory X, 20% of the samples are randomly selected from directories X, Y, and Z as the test set, and the remaining 80% are used as the training set. Directory X contains the input samples / images with shadows, directory Y contains the shadow mask (not used for now), and directory Z contains the sample label images (images without shadows).

[0086] The training set samples are shuffled and then divided into batches. The number of samples in each batch is set according to the graphics card memory capacity. Within each batch, image samples are uniformly compressed into square images with a height H equal to a width W using linear interpolation and black border padding (for easy matrix operations). The size of these square images depends on actual needs and graphics memory capacity, typically a multiple of 8, such as 512x512. If the image to be shadow-removed is a high-resolution image and its size exceeds the graphics card's maximum storage capacity, the maximum square size that the graphics memory can hold can be used as the sample loading size, with each batch containing only one sample. Super-resolution techniques can be used for restoration during the final inference stage.

[0087] To avoid computational or overflow issues caused by excessively large values, the sample data needs to be normalized to map the data to the range of 0-1. Here, the Min-Max normalization method is used:

[0088]

[0089] Where x′ represents the normalized sample value, and x is the original sample value.

[0090] VI. Load data into the convolutional neural network model for training and testing.

[0091] The training set samples are loaded into the U2Net convolutional neural network based on image segmentation in batches for training. The optimizer is Adam, the learning rate is 0.0005 and decays to 0.000001. The loss function indicates the direction of model learning. Through continuous learning and weight updates, the image output by the model gradually becomes similar to the image without shadows.

[0092] Testing and weight saving can be performed every N iterations of training as needed. The testing process is similar to the training process, and the loading method of the test set is exactly the same as that of the training set. However, it is not necessary to calculate the loss and update the weights. It is only necessary to denormalize the network output image and save it to a file.

[0093] VII. Reasoning and Application

[0094] When the loss decreases to a plateau and the test output image set has almost no shadows, training can be stopped first, and the trained weight data can be loaded into the neural network for inference.

[0095] The reasoning process is similar to the testing process, and the process is as follows:

[0096] 1. Initialize the neural network;

[0097] 2. Load the trained weights;

[0098] 3. The shadow image is batched, scaled into squares, and normalized;

[0099] 4. Load the shadow images into the neural network in batches;

[0100] 5. The neural network makes predictions and outputs;

[0101] 6. Denormalize and scale the output image to its original size to obtain a high-quality, shadow-free image.

[0102] It is important to note here that:

[0103] 1. Only images with shadows (i.e., images that need to have their shadows removed) are loaded into the neural network; labeled images are not required.

[0104] 2. To scale the output image to its original size, the previously filled black borders need to be removed first;

[0105] 3. If the input image is loaded into the neural network after upsampling and scaling, then only downsampling and scaling of the output image is needed to restore its size. Otherwise, super-resolution technology is needed to upsample and scale the output image to restore its size.

[0106] Example 2:

[0107] This embodiment exemplarily proposes an image shadow removal device, the structure of which is as follows: Figure 3 As shown, it includes: an image input module, an image processing module, an image output module, and an image display module; wherein,

[0108] The image input module is used to acquire input images and perform image preprocessing, and then input the preprocessed image into the image processing module;

[0109] The image processing module is used to remove shadows from the input image based on the image shadow removal method described above;

[0110] The image output module is used to process the image output by the image processing module to meet the preset output image format requirements;

[0111] The image display module is used to display the image after the shadow has been removed.

[0112] Example 3:

[0113] This embodiment proposes a storage medium storing a computer program, which, when executed, implements the image shadow removal method.

[0114] Example 4:

[0115] This embodiment proposes a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it implements the image shadow removal method.

[0116] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for removing image shadows, characterized in that, The method includes the following steps: (1) Construct training samples: Obtain multiple pairs of sample images. Each pair of sample images includes an image with only background and no shadow and an image with shadow on the same background. (2) Construct a shadow image removal model based on UNet; (3) Use the shadow image in each pair of samples as the input image and the shadow image as the label image to train the model; during the training process, use the loss function to calculate the loss of the predicted image and the corresponding label image, and then update the weight of each layer of the UNet network with gradient, so that the predicted image output by the upsampling part of each layer of the convolutional neural network in the UNet network and the single predicted image fused by the predicted image respectively approximate the label image, and at the same time, make the color space channel image of each predicted image and the overall predicted image fused by all color space channel images approximate the label image; The loss function is: Where L represents the overall loss, N represents the number of layers in the convolutional neural network of the UNet network, and C represents the total number of RGB color space channels in the predicted image. Let represent the loss function used to calculate the loss of the i-th color space channel of the predicted image output by the n-th convolutional neural network. F represents the loss function used to calculate the loss of the predicted image output by the nth layer of the convolutional neural network. fuse This represents the loss function used to calculate the loss over the overall predicted image. and w fuse These are weighting coefficients; and F fuse Each is composed of a corresponding binary cross-entropy loss function and a mean squared error loss function, i.e.: Among them, Ψ boe Let Φ represent the binary cross-entropy loss function. mse Let w represent the mean squared error loss function. boe and w mse All are weighting coefficients; H and W represent the height and width of the image, respectively, (i,j) represents the pixel coordinates in the image, and P... G(i,j) and P S(i,j) These represent the color values ​​of the pixel at coordinates (i,j) in the label image and the prediction image, respectively. (4) After the shadow image removal model is trained, the shadow removal model is used to remove shadows from the input shadowed image.

2. The image shadow removal method according to claim 1, characterized in that, The shadow image removal model is implemented using the U2Net network.

3. An image shadowing device, characterized in that, include: The module comprises an image input module, an image processing module, and an image output module; among which, The image input module is used to acquire input images and perform image preprocessing, and then input the preprocessed image into the image processing module; The image processing module is used to remove shadows from the input image based on the image shadow removal method according to any one of claims 1 to 2; The image output module is used to process the image output by the image processing module to meet the preset output image format requirements.

4. The image shadowing device according to claim 3, characterized in that, The device also includes an image display module for displaying an image after the shadow has been removed.

5. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed, implements the image shadow removal method according to any one of claims 1 to 2.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the image shadow removal method according to any one of claims 1 to 2.

Images