An image shadow repairing method based on residual mask guidance

Abstract

The application discloses an image shadow repair method based on residual mask guidance to improve the shadow removal effect of an image. The method mainly comprises the following steps: using the excellent performance of a Transformer network in processing long-range dependence and global context information, learning the correlation between the shadow area and the non-shadow area, and realizing preliminary shadow removal; a mask extraction algorithm is proposed, and a residual mask is calculated by using the preliminary repair result and a real non-shadow image; a residual mask loss function is designed, and a proper iterative repair threshold is set, and the iterative repair of the shadow image is realized through the guidance of the residual mask, so that a more natural best repair effect is obtained. The application can obtain an accurate and realistic shadow repair effect according to the input shadow image and the inaccurate mask prompt, and can be widely used in the shadow removal task, improve the image quality, and improve the performance of the subsequent downstream task.

Description

Technical Field

[0001] This invention relates to the fields of image processing and video processing, specifically to an image shadow restoration method based on residual mask guidance. Background Technology

[0002] Shadows are dark areas formed on a scene or target due to an object blocking direct light from a light source, and are ubiquitous in nature. Appropriate shadows can provide depth information and enhance the realism of a scene; however, inappropriate shadows not only reduce the visual quality and aesthetics of an image, but also affect the color and texture information of the target, obscuring important information and details in the image, thus increasing the difficulty of visual recognition and reducing the stability and accuracy of subsequent tasks. Because the type, intensity, and hue of the light source that forms shadows vary, as do the material, size, and shape of the obscured target, the resulting shadows are also diverse. Therefore, how to efficiently and accurately remove shadows from images has become a significant challenge in the field of image processing.

[0003] Traditional image shadow removal algorithms are typically based on mathematical methods and conventional image processing techniques: after modeling and analyzing the background of the image, statistical methods are used to identify shadow and non-shadow regions. Shadow regions are then filled using interpolation or texture synthesis, and finally, appropriate post-processing is applied to smooth the image. While this method is simple in principle, it performs poorly in complex scenes, easily leading to image distortion and information loss. Furthermore, traditional shadow removal methods rely on shadow detection; therefore, if the detection results are inaccurate, the restoration results will contain irreparable errors.

[0004] In recent years, deep learning has experienced explosive growth and achieved remarkable processing results, making breakthroughs in image shadow removal. Shadow inpainting methods based on convolutional neural networks and generative adversarial networks have emerged. These algorithms do not require precise modeling and analysis of the image background as traditional algorithms. Instead, they utilize powerful deep models to learn texture, color, and other feature information of non-shadow areas from large-scale data, directly achieving efficient and flexible shadow removal. However, there is a certain correlation between shadow and non-shadow areas; appropriately capturing this relationship can effectively improve the shadow inpainting effect. In 2023, Guo et al. proposed the Transformer-based shadow inpainting algorithm ShadowFormer, attempting to leverage the Transformer's superior performance in handling long-range dependencies and global contextual information to process shadow areas, thereby obtaining more natural shadow-inpainted images. However, similar to traditional algorithms, this algorithm is highly dependent on the input mask cues; furthermore, the shadow edge effect of the inpainted image obtained using this algorithm is obvious, and the shadow-inpainted area is more prominent than the surrounding background, affecting image quality and visual appeal. Summary of the Invention

[0005] To overcome the problem of excessive reliance on mask cues and significant shadow edge effects in shadow inpainting, this invention provides an image shadow inpainting method guided by residual masks. Utilizing the global awareness capability of the Transformer, iterative inpainting of the shadow image is achieved based on the residual mask between the real shadow-free image and the inpainted result, along with a designed residual mask loss function. This results in a more natural and accurate shadow inpainting effect, significantly improving the performance of subsequent tasks and enhancing its engineering application capabilities.

[0006] To achieve the above objectives, the technical solution proposed in this invention is: an image shadow inpainting method based on residual mask guidance. The method mainly includes the following steps: Step (1) Preliminary shadow removal of the input image. Based on the characteristic of the Transformer network that it can perceive global context information without being limited by the length of the input sequence, the input image is first linearly mapped, and the image features are extracted using an L-layer encoder structure. Then, the correlation between shadow and non-shadow regions is learned using the cue information provided by the input mask. Next, the fused feature information of the output is decoded using an L-layer decoder structure and mapped back to the original image feature space, thereby achieving preliminary shadow removal. Step (2) Residual Mask Extraction. During training, real shadowless images are used as a reference. Therefore, the residual mask can be calculated based on the preliminary shadow removal results and the real shadowless images. The main idea is to perform background subtraction on these two images, supplemented by appropriate thresholding algorithms and morphological processing methods.

[0007] Step (3) Calculate the residual mask loss. Calculate the residual mask loss value obtained at this time according to the designed mask loss function, and compare it with the set threshold.

[0008] Step (4) Iterative update of repair effect. When the loss value calculated in step (3) is greater than the set threshold, the repaired image obtained in step (1) is used as the shadowed image in the next iteration process, and the residual mask is used as the guiding mask at this time. Steps (1) to (3) are executed again. When the loss value calculated in step (3) is less than or equal to the set threshold, the repair process is completed. The output image at this time is the final shadow repair result obtained by the network.

[0009] Furthermore, in step (1), the distinction between shaded and unshaded areas is achieved through a mask. The mask is a binary image, with shaded areas represented by 1 and unshaded areas by 0. Therefore, this binary labeled image can be used to label all shaded areas of the original input image.

[0010] Furthermore, in step (2), since the shadowed image is usually darker than the real image without shadow, it is necessary to normalize the two images first, that is, to unify the maximum value of the two images to 1.0 according to the following formula: , Wherein, image is the image to be normalized, and max(·) is the maximum value operation; then, background difference is performed on the two images to ensure that the background difference between them is minimized, which can be directly filtered out by threshold segmentation.

[0011] Compared with the prior art, the beneficial effects and advantages of the present invention are as follows: (1) This invention utilizes the Trasnformer network structure to effectively capture global context information and long-range dependencies in images, enabling better learning of the correlation between shadow and non-shadow regions. Compared with traditional algorithms, it can handle more complex and varied shadow situations. (2) This invention utilizes the background subtraction method to obtain the shadow mask. By performing background subtraction on the real image without shadow and the image with shadow, and supplementing it with appropriate threshold segmentation and morphological processing methods, a relatively accurate shadow mask can be obtained. The algorithm is simple and easy to implement.

[0012] (3) The present invention uses residual masks to achieve iterative restoration of shadow images. Compared with existing deep learning algorithms and other shadow removal algorithms implemented by Transformer, the present invention does not require input of accurate shadow area mask prompts. Only coarse mask information is required to obtain natural and realistic shadow removal effect, complete and repair effective information occluded by shadow, and improve the performance of subsequent advanced image processing tasks such as target recognition and image segmentation. In addition, the restoration effect obtained by the method of the present invention has weak edge effect and smaller difference from the background. Therefore, it has wider applicability and higher practicality and commercial value. Attached Figure Description

[0013] Figure 1 This is an overall flowchart of the image shadow restoration method based on residual mask guidance of the present invention; Figure 2 This is a network structure diagram of the image shadow inpainting method based on residual mask guidance of the present invention; Figure 3A The shadow repair effect obtained by using the method described in this invention; Figure 3B The repair effect on the baseline network ShadowFormer; Figure 3C The original image with shadows; Figure 3D This is a true, shadow-free image. Detailed Implementation

[0014] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0015] Prepare a sufficient dataset of shadow images of different types and sizes, including indoor and outdoor shadow scenes, as well as mask cues for shadow areas; during the model training phase, you also need to prepare shadow-free images corresponding to the shadow data, and then start building and training the network.

[0016] like Figure 1 As shown, the image shadow inpainting method based on residual mask guidance of the present invention mainly includes the following steps: Step (1) Preliminary shadow removal of the input image. The Transformer network consists of an input module, an encoder structure, a decoder structure, and an output module. The input module encodes the input image information into a vector and adds positional encoding information to it. The encoder module consists of L stacked encoder layers. Each sub-layer includes a multi-head self-attention structure, a normalization layer, a feedforward fully connected layer, and a residual connection structure, which is used to extract image features, i.e., the encoding process. The decoder module consists of L stacked decoder layers. Each sub-layer also consists of a multi-head self-attention structure, a normalization layer, a feedforward fully connected layer, and a residual connection, which is used to predict the output result from the features, i.e., the decoding process. The output module consists of a linear layer and a softmax layer, which is used to map the decoder output to a specified dimension to obtain the target output result.

[0017] The overall structure diagram of the algorithm is as follows: Figure 2 As shown, the Transformer network structure ensures superior performance compared to other networks in handling long-range dependencies and global context information. Therefore, the Transformer network can be used to achieve preliminary shadow removal of images. First, the input image is linearly mapped to a feature space, and image features are extracted using an L-layer traditional encoder structure. The encoder structure is the same as the original Transformer encoder module, consisting of a multi-head self-attention structure, a normalization layer, a feedforward fully connected layer, and a residual connection structure; the value of L can be designed according to actual requirements, and L=6 is set in the experiment of this invention. Then, the correlation between shadow and non-shadow regions in the original image is learned using the shadow region cues provided by the input mask. The region correlation calculation idea is as follows: the shadow mask is a binary image of the same size as the input shadowed image, the shadowed region is labeled as 1, i.e., the white foreground region; the non-shadowed image is labeled as 0, i.e., the black background region. After feeding the shadowed image and the corresponding shadow mask into the encoder, two feature maps are obtained, denoted as X. L and M L , representing the features extracted by the L-layer coding layer from the original shadowed image and the features extracted from the shadow mask, respectively. Multiplying the two feature maps by a matrix yields a fused feature map that integrates shadow location information, i.e.: , X L+1 This is the output fused feature map. This represents element-wise multiplication of matrices. Since X L+1The method considers both shadow location information and original image information, thus describing the correlation between shadowed and non-shadowed regions. This fused feature is then input into an L-layer decoder structure to decode the output fused feature information. The decoder structure is the same as the original Transformer decoder module, consisting of a multi-head self-attention structure, a normalization layer, a feedforward fully connected layer, and a residual connection structure, essentially the inverse process of the decoder. Finally, the decoder output is mapped back to the original image feature space through a linear mapping, thus achieving preliminary shadow inpainting.

[0018] Step (2) Residual Mask Extraction. The residual mask is extracted using the real shadowless image and the preliminary shadow removal result obtained in step (1). Since the real shadowless image is usually brighter than the image with shadows, the two images are first normalized according to the following formula:

[0019] Here, `image` is the image to be normalized, and `max(·)` is the maximum value operation; this makes the gray values ​​of the background parts of the two images similar, reducing the probability that the extracted mask includes the background part. Next, background subtraction is performed on the two images, calculated using the following formula: dif = gt - image; if gt > image, else 0, Where 'dif' represents the obtained difference image, 'gt' represents the true shadowless image, and 'image' represents the preliminary shadow restoration result. The above formula means that when the grayscale value of the true shadowless image is greater than the grayscale value of the preliminary shadow restoration result, the grayscale value of the difference image is the difference between the two; otherwise, it is 0. This difference processing avoids negative values ​​and is more effective than directly using absolute values. This approach is reasonable because if the grayscale value of the true shadowless image is less than the grayscale value of the preliminary shadow restoration result, it indicates that this is likely a background area or an area that has already been restored, since the grayscale value of an unrestored shadow area is necessarily less than that of a shadowless area. If the absolute value of the difference between the true shadowless image and the input shadowed image is directly used, unnecessary background terms may be introduced into the threshold segmentation result, affecting the final processing effect. Therefore, the segmentation method shown in the formula is adopted. Next, threshold segmentation is performed on the difference image, and the Otsu threshold segmentation method can be used for automatic threshold calculation. Otsu's thresholding method is currently the best method for thresholding the entire image, applicable to most scenarios. It obtains the optimal segmentation threshold by maximizing the inter-class variance between the foreground and background image regions. Variance is a measure of the uniformity of gray-level distribution; a larger inter-class variance indicates a greater difference between the two regions. Therefore, based on the gray-level information of the input image, a threshold is selected to divide the image into foreground and background regions, and the inter-class variance between them is calculated separately. The formula for calculating variance is: , Where m1 and m2 refer to the average grayscale values ​​of the foreground and background, respectively. G σ refers to the global image mean, p1 and p2 represent the probabilities of a pixel belonging to the foreground and background classes, respectively. 2 This is the final calculated variance result. By iterating through all pixel values ​​in the range of 0-255, and performing threshold segmentation based on the pixel value corresponding to the maximum inter-class variance, the optimal segmentation result based on Otsu's threshold segmentation method can be obtained.

[0020] Finally, appropriate morphological processing methods are applied to the obtained segmentation results, using a combination of erosion, dilation, opening, and closing operations to remove isolated points and holes without affecting the size of the target region. However, the processing template should not be too large, otherwise it will lead to obvious blockiness in the processing results, making it impossible to obtain a smooth and natural mask. This yields the residual mask between the ground truth shadowless image and the preliminary shadow inpainting result.

[0021] Step (3) Calculation of residual mask loss. Shadow mask images typically have shadowed areas labeled as 1 and unshadowed areas labeled as 0. The residual mask is obtained based on the real unshadowed image and the restoration result. Therefore, if the restoration result is similar to the real unshadowed image, the resulting residual mask should be approximately an all-zero image. Based on this, the formula for calculating the residual mask loss function is designed as follows: , Where H and W refer to the height and width of the mask (or input image), respectively; Mask[i, j] refers to the grayscale value of the pixel corresponding to the mask. The residual mask loss is expressed as the average pixel value of the residual mask.

[0022] Step (4) Iterative update of the repair effect. The better the repair effect of the output image, the smaller the difference between it and the real shadowless image, and the smaller the residual loss value should be. Therefore, a suitable loss value is selected as the threshold. When the loss value calculated in step (3) is greater than the set threshold, it means that the repair result obtained at this time is much different from the target result. At this time, the repaired image obtained in step (1) is used as the shadowed input image in the next iteration process, and the residual mask is used as the guiding mask at this time. Steps (1) to (3) are repeated. When the loss value calculated in step (3) is less than or equal to the set threshold, it means that the difference between the repair result obtained at this time and the real shadowless image is small and can be ignored. The output image obtained at this time is the shadow repair result finally obtained by the network. At the same time, in order to prevent the network from falling into an infinite loop, the maximum number of iterations for repair needs to be set to ensure that the network can be trained normally.

[0023] The overall process of this invention is as follows: The input consists of a shadowed image and a corresponding shadow mask. Linear projections are performed on both, and the feature maps are fed into an L-layer encoder structure. Next, the feature maps of the shadow mask and the shadowed image are fused. The fused features are decoded using an L-layer decoder and projected into the original feature space to obtain a preliminary shadow restoration result. Then, the residual mask is calculated using the real shadowless image and the restoration result, and the residual mask loss is calculated. When the loss value is greater than a threshold, the restoration result is used as the image to be restored in the next round, and the residual mask is used as the new input mask to perform a new round of iterative restoration. When the loss value is less than the threshold, the iterative process ends, and the final shadow restoration result is obtained.

[0024] This invention sets the image size fed into the network to a uniform 320×320 pixels, with a batch size of 6. It uses three A5000 GPUs for training, for a total of 300 training epochs. The learning strategy employs the AdamW optimization algorithm, with a weight decay coefficient of 0.02, momentum parameters set to (0.9, 0.999), and an initial learning rate of 2×10⁻⁶. -4Then, by cosine annealing, the temperature is gradually reduced to 1×10. -6 To avoid the network getting stuck in an infinite loop due to the loss value never falling below the threshold during iteration, the maximum number of iterations was limited to 3 in the experiment. The network was trained according to the above parameter settings.

[0025] Experimental results demonstrate the effectiveness of the image shadow restoration method based on residual mask guidance proposed in this invention. Figure 3A The repair result proposed in this invention, Figure 3B The results of the repair of the baseline network ShadowFormer. Figure 3C The original image with shadows. Figure 3D The images presented are true shadow-free images. A comparison reveals that the restoration results of this invention are more natural than the original baseline network results, with smaller color differences between the restored shadow areas and the background, and improved contour edge effects. Furthermore, Table 1 shows the training results of the method described in this invention and the baseline method ShadowFormer in the current training environment. Comparing the training results of the two methods on different datasets in Table 1, it can be seen that the model trained using the method proposed in this invention has a smaller loss value and a higher PSNR value, indicating that the model performs better.

[0026] Table 1 Comparison of training results between the method of this invention and ShadowFormer

[0027] It should be emphasized that the above are merely preferred examples of the present invention, but do not impose any limitations on the present invention.

[0028] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. An image shadow inpainting method based on residual mask guidance, characterized in that, The method includes the following steps: Step (1) Preliminary shadow removal of the input image: The Transformer network is used to realize the perception of global context information. First, the input image is linearly mapped and the image features are extracted using the L-layer encoder structure. Then, the correlation between the shadow area and the non-shadow area is learned using the prompt information provided by the input mask. Next, the L-layer decoder structure is used to decode the output fused feature information and map it back to the original image feature space, thereby realizing the preliminary shadow removal. Step (2) Residual mask extraction: Calculate the residual mask at this time based on the preliminary shadow removal results and the real shadowless image. By performing background subtraction on these two images and supplementing them with appropriate threshold segmentation algorithms and morphological processing methods, the residual mask for the current repair result is finally obtained. Step (3) Calculation of residual mask loss: Calculate the residual mask loss value obtained at this time according to the designed mask loss function, and compare it with the set threshold; Step (4) Iterative update of repair effect: When the loss value calculated in step (3) is greater than the set threshold, the repaired image obtained in step (1) is used as the shadowed image in the next iteration process, and the residual mask is used as the guiding mask at this time. Steps (1) to (3) are re-executed. When the loss value calculated in step (3) is less than or equal to the set threshold, the repair process can be completed. The output image at this time is the shadow repair result finally obtained by the network.

2. The image shadow inpainting method based on residual mask guidance according to claim 1, characterized in that, In step (1), the distinction between shadowed and non-shadowed areas is achieved through a mask. The mask is a binary image, with shadowed areas represented by 1 and non-shadowed areas represented by 0. This binary labeled image is used to label all shadowed areas of the original input image.

3. The image shadow inpainting method based on residual mask guidance according to claim 1, characterized in that, In step (2), the two images are first normalized, that is, the maximum value of the two images is unified to 1.0 according to the following formula: ， Wherein, image is the image to be normalized, and max(·) is the maximum value operation; then, background difference is performed on the two images to ensure that the background difference between them is minimized, and the images are directly filtered out through threshold segmentation.

Images