Image shadow removal method, device, storage medium and equipment

Abstract

This application provides an image shadow removal method, apparatus, storage medium, and device, comprising: obtaining a shadow image and a shadow mask image, inputting the shadow image and shadow mask image into a mask depth fusion module of a Transformer model to obtain a feature vector matrix; and inputting the feature vector matrix into an attention mechanism module of the Transformer model to obtain a de-shadowed image. This method, through shadow mask fusion and multi-stage fusion of the Transformer model, enables the Transformer model to focus on repairing the shadow areas of the image, efficiently and accurately obtaining a de-shadowed image.

Description

Technical Field

[0001] The present invention relates to image processing, and in particular to an image shadow removal method, device, storage medium and equipment. Background Art

[0002] Shadows in images are dark areas caused by objects completely or partially blocking light. There are two types of shadows in images: self-shadowing, where parts of the object are not directly illuminated by light; and cast shadows, where the object is projected toward the light source. Shadows are ubiquitous in nature. Shadows in images not only affect human visual perception but also the performance of computer vision tasks such as image recognition, object detection, and object tracking. Therefore, removing shadows is crucial for image processing.

[0003] The current mainstream approach to shadow removal is to train deep learning methods, such as generative adversarial networks and convolutional neural networks, using large datasets of real-world shadow images to achieve high performance. However, these methods have limitations: First, convolutional neural network-based model frameworks are no longer optimal for some image processing tasks; second, while shadow masks have been proven to be effective in assisting shadow removal by numerous deep learning-based methods, effectively integrating these masks with the model framework remains a challenge. Summary of the Invention

[0004] Based on this, the present invention provides an image shadow removal method, apparatus, storage medium and device, which efficiently integrate the mask and model framework to accurately and efficiently remove the shadow part in the image.

[0005] In a first aspect, the present invention provides a method for removing image shadows, comprising:

[0006] Get shadow image and shadow mask image;

[0007] Inputting the shadow image and the shadow mask image into the mask depth fusion module of the Transformer model to obtain a feature vector matrix;

[0008] The feature vector matrix is ​​input into the attention mechanism module of the Transformer model to obtain a shadow-removed image.

[0009] Furthermore, the shadow image and the shadow mask image are input into the mask depth fusion module of the Transformer model to obtain a feature vector matrix, including:

[0010] performing binarization and normalization processing on the shadow mask image respectively to obtain a first shadow mask and a second shadow mask;

[0011] The shadow image is subjected to a block embedding process to obtain a first eigenvector, and the second shadow mask is subjected to a block embedding process to obtain a second eigenvector;

[0012] The first eigenvector, the first shadow mask, and the second eigenvector are fused to obtain an eigenvector matrix.

[0013] Furthermore, the Transformer model is trained with mask fusion loss.

[0014] In a second aspect, the present invention provides an image shadow removal device, comprising:

[0015] an acquisition unit, configured to acquire a shadow image and a shadow mask image;

[0016] A preprocessing unit, configured to input the shadow image and the shadow mask image into a mask deep fusion module of a Transformer model to obtain a feature vector matrix;

[0017] The shadow removal unit is used to input the feature vector matrix into the attention mechanism module of the Transformer model to obtain a shadow-removed image.

[0018] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the image shadow removal methods in the first aspect.

[0019] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it executes any one of the image shadow removal methods in the first aspect.

[0020] The beneficial effects of adopting the above technical solution are as follows: The image shadow removal method of the present invention is based on the idea of ​​shadow mask fusion and multi-stage fusion of the Transformer model, and respectively sets a mask depth fusion module in the preprocessing stage of the model, and sets a mask fusion loss in the loss training stage. The introduction of the mask depth fusion module in the preprocessing stage enables the Transformer model to focus on repairing the shadow area of ​​the image, avoiding meaningless model parameter solution. The mask fusion loss in the loss training stage can effectively improve the performance of the Transformer model in removing image shadows and remove image shadows more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0022] Figure 1 This is a schematic diagram of a method for removing image shadows in one embodiment of the present application;

[0023] Figure 2 This is a schematic diagram of a framework of an image shadow removal method in one embodiment of the present application;

[0024] Figure 3 This is a detailed schematic diagram of the image shadow removal method in one embodiment of the present application;

[0025] Figure 4 This is a comparison diagram of the effects of the image shadow removal method in one embodiment of the present application and other shadow removal methods;

[0026] Figure 5 This is another comparison diagram of the effects of the image shadow removal method in one embodiment of the present application and other shadow removal methods;

[0027] Figure 6 This is a first comparative diagram of the peak signal-to-noise ratio, structural similarity, and root mean square error of the image shadow removal method in one embodiment of the present application and other shadow removal methods;

[0028] Figure 7 A second comparative diagram of the peak signal-to-noise ratio, structural similarity, and root mean square error of the image shadow removal method in one embodiment of the present application and other shadow removal methods;

[0029] Figure 8 This is a schematic diagram showing a comparison of parameters between an image shadow removal method in one embodiment of the present application and other shadow removal methods;

[0030] Figure 9 Schematic diagram of an image shadow removal device in one embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In order to explain the present invention in more detail, the image shadow removal method, device, storage medium and equipment provided by the present invention are specifically described below in combination with the drawings.

[0032] Image shadow removal is due to the dark areas formed by the projection of objects in the image onto the light source. The shadows in the image are not conducive to the application of some technologies in subsequent image processing, such as image recognition, target detection, and target tracking. Therefore, removing shadows in the image processing stage is particularly important.

[0033] The present application provides an application scenario for a method for removing image shadows. This application scenario includes a terminal device provided in the embodiment, including but not limited to a smartphone and a computer device, wherein the computer device can be at least one of a desktop computer, portable computer, laptop computer, mainframe computer, tablet computer, and the like. A user operates the terminal device to obtain an image after removing shadows. For detailed procedures, please refer to the embodiment of the method for removing image shadows.

[0034] Based on this, the embodiment of the present invention provides an image shadow removal method, which is described by taking the application of the method to a terminal device as an example. Figure 1 Schematic diagram of image shadow removal method shown in FIG. Figure 2 and attached Figure 3 Schematic diagram of the image shadow removal method framework.

[0035] Step S101: Obtain a shadow image and a shadow mask image.

[0036] Specifically, a shadow image is acquired by an image acquisition device, wherein the shadow of the shadow image is formed by an object blocking light or projecting light in the direction of the light source.

[0037] The shadow mask image is used to indicate whether the pixels in the shadow area of ​​the shadow image should be considered or processed. The shadow mask image is usually composed of a pixel grid of the same size and shape as the original image, where each pixel in the shadow mask image corresponds to the pixel at the corresponding position in the original image. Each pixel value of the shadow mask image is usually 0 or 255, where 0 indicates that the pixel at the corresponding position should be ignored or excluded, indicating a shadow area; 255 indicates that the pixel at the corresponding position should be retained or processed, indicating a non-shadow area. By performing element-level pixel-by-pixel operations on the shadow image through the shadow mask image, selective operations on specific areas can be achieved, such as masking, cropping, filtering, fusion, etc.

[0038] Step S102: Input the shadow image and the shadow mask image into the mask depth fusion module of the Transformer model to obtain a feature vector matrix.

[0039] Specifically, the shadow image and the shadow mask image are input into the mask depth fusion module of the Transformer model to obtain the feature vector matrix, including the following steps:

[0040] Step S201: performing binarization and normalization processing on the shadow mask image to obtain a first shadow mask and a second shadow mask.

[0041] The specific expression for binarization of the shadow mask image is as follows:

[0042]

[0043] Among them, M s is the first shadow mask, i is the i-th pixel in the shadow image. The binarization of the shadow mask image reshapes all pixels corresponding to the shadow area to 1 and all pixels corresponding to the non-shadow area to 0.

[0044] The specific expression for normalizing the shadow mask is as follows:

[0045]

[0046] Among them, M p is the second shadow mask, P i is the pixel value of the i-th pixel in the shadow image. Normalization of the shadow mask image rescales the pixel range to [-1, 1]. Information in shadow and non-shadow areas is represented as opposite values. Compared to the traditional [0, 1] normalization operation, pixel information in non-shadow areas is represented as negative values, avoiding irreversible information loss caused by subsequent operations.

[0047] Step S202: The shadow image is subjected to a block embedding process to obtain a first eigenvector, and the second shadow mask is subjected to a block embedding process to obtain a second eigenvector.

[0048] Specifically, the patch embedding process (Patch Embedding) refers to dividing the image into a series of small patches and converting each patch into a vector representation, encoding the local information of the image into a vector form that can be processed by the Transformer model. It is a key step in the subsequent use of the Transformer model. s The first eigenvector obtained after the tile embedding process is recorded as T i , the second shadow mask M p The second eigenvector obtained after the tile embedding process is denoted as T m .

[0049] Step S203: Fusing the first eigenvector, the first shadow mask, and the second eigenvector to obtain an eigenvector matrix.

[0050] Specifically, the first eigenvector T i , the first shadow mask M s and the second eigenvector T m The expression of the eigenvector matrix obtained by fusion is:

[0051] I=((w1*M s +w2*(1-M s ))*T i)*T m ,

[0052] Where I is the eigenvector matrix, w1 is the weight of the first region, w2 is the weight of the second region, M s is the first shadow mask, T i is the first eigenvector, T m is the second eigenvector.

[0053] The above fusion process includes two stages. The first stage is to redistribute the pixel information of the shadow area and non-shadow area in the shadow image through the first area weight and the second area weight to achieve the purpose of preliminary fusion of the shadow mask image; the second stage is to further fuse the pixel information after redistribution of weights with the second eigenvector extracted from the second shadow mask to avoid invalid fusion of non-shadow area information or irreversible information loss.

[0054] Step S103: Input the feature vector matrix into the attention mechanism module of the Transformer model to obtain a shadow-removed image.

[0055] Specifically, the Transformer model is a neural network model based on the self-attention mechanism. By introducing this mechanism, it can effectively capture the dependencies between different positions in the input sequence, thereby improving the model's performance and parallel computing capabilities. When using the Transformer model to process image data, the input image is segmented into a set of fixed-size image blocks. Each image block is then converted into a corresponding feature vector. The feature vector is input into the Transformer encoder for processing. The self-attention mechanism is used to capture the dependencies between image blocks. Through the multi-layer stacked Transformer encoder, high-level feature representations of the image are gradually extracted. Finally, a fully connected layer maps the feature representations to prediction results for different categories to obtain a de-shadowed image.

[0056] The Transformer model includes multi-head attention (MHA), a multi-layer perceptron (MLP), and a regularization layer (Layer Normalization). Multi-head attention is used to divide the model into multiple heads, forming multiple subspaces, allowing the Transformer model to focus on different aspects of information. The regularization layer is used to pre-process the input feature vectors and data to ensure better learning of the data content.

[0057] Furthermore, considering that shadow and non-shadow areas of an image often originate from the same harmonious scene, if the model overly focuses on inpainting shadow areas without simultaneously processing non-shadow areas, noticeable boundary artifacts will appear between them. To address this issue, a mask fusion loss is proposed to supervise the overall inpainting effect, improving the overall image quality. Specifically, the Transformer model is trained using the mask fusion loss.

[0058] The mask fusion loss includes color consistency loss and weighted similarity loss, and the specific expression is:

[0059] L=L clo +L m ,

[0060] Among them, L is the mask fusion loss, L clo is the color consistency loss, L m is the weighted similarity loss.

[0061] The color consistency loss L clo Smooth Loss is used to ensure color consistency between the obtained de-shadowed image and the corresponding true shadow-free image, specifically:

[0062]

[0063] Where x is the difference between the shadow-free image output by the Transformer model and the true shadow-free image.

[0064] The weighted similarity loss L m The idea of ​​weighted masking is introduced in

[15] , which allocates different attention to the shadow area and non-shadow area of ​​the image by weight, further improving the local de-shadowing effect of the image. Specifically:

[0065] L m =(w*M s +(1-w)*M s )*L ssim ;

[0066]

[0067]

[0068]

[0069] Among them, w is the weight assigned to the shaded area, (1-w) is the weight assigned to the non-shaded area, and M s is the first shadow mask, L ssim is the loss based on structural similarity, μx is the average brightness of the output de-shadowed image, μ y is the average brightness of the real image, σ x is the pixel variance of the output de-shadowed image, σ y is the variance of the pixels of the real image, σ xy is the pixel covariance of the output de-shadowed image and the true image, x i is the pixel value of the i-th pixel of the output shadow-removed image, c1 and c2 are preset constants, and N is the total number of pixels of the shadow-removed image.

[0070] The structural similarity loss measures the similarity between two given images by extracting brightness, contrast, and structure from the images, and is used to improve the similarity between the de-shadowed image and the true shadow-free image, thereby ensuring the generation of a high-quality de-shadowed image.

[0071] In addition, weighted masks are not introduced in the color consistency loss because there is often color inconsistency between shadow image pairs in the real world. This is a problem that exists in the global image, not just the shadow area. Therefore, maintaining global color consistency is the primary task, and local color repair is no longer performed by using weighted masks.

[0072] In order to better illustrate that the image shadow removal method proposed by the present invention has a better shadow removal effect, see the attached Figure 4-7 The image shadow removal method of the present invention is compared with the existing shadow removal method, and the three parameters of peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and root mean square error (RMSE) as well as the shadow removal effect diagram are compared. The image shadow removal method of the present invention has better effect; In addition, see the attached specification. Figure 8 The image shadow removal method of the present invention adopts the lowest number of model parameters, reduces the amount of floating-point calculations, and simplifies model deployment and operation pressure.

[0073] It should be understood that although the Figure 1 The steps in the flowchart are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 1 At least part of the steps may include multiple sub-steps or sub-stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0074] The above embodiments of the present invention describe in detail the image shadow removal method. The above methods disclosed in the present invention can be implemented using various devices. Therefore, the present invention also discloses an image shadow removal device corresponding to the above methods. Figure 9 , specific embodiments are given below to explain in detail.

[0075] The acquisition unit 301 is configured to acquire a shadow image and a shadow mask image.

[0076] The pre-processing unit 302 is configured to input the shadow image and the shadow mask image into a mask deep fusion module of a Transformer model to obtain a feature vector matrix.

[0077] The shadow removal unit 303 is used to input the feature vector matrix into the attention mechanism module of the Transformer model to obtain a shadow-removed image.

[0078] The specific limitations of the image shadow removal device can be found in the limitations of the method above and will not be repeated here. Each unit in the above-mentioned device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned units can be embedded in or independent of the processor of the terminal device in hardware form, or can be stored in the memory of the terminal device in software form so that the processor can call and execute the corresponding operations of each of the above-mentioned units.

[0079] In one embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned image shadow removal method when executed by a processor.

[0080] The computer-readable storage medium can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM (Erasable Programmable Read-Only Memory), a hard disk, or a ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written into one or more computer program products, and the program codes can be compressed in an appropriate form.

[0081] In one embodiment, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor performs the above-mentioned image shadow removal method when executing the computer program.

[0082] The computer device includes a memory, a processor, and one or more computer programs, wherein the one or more computer programs can be stored in the memory and configured to be executed by one or more processors, and the one or more application programs are configured to execute the above-mentioned image shadow removal method.

[0083] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire computer device, and performs various functions of the computer device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to handle wireless communication. It is understandable that the above-mentioned modem may not be integrated into the processor and may be implemented separately through a communication chip.

[0084] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the terminal device during use, etc.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for removing image shadows, characterized in that: include: Get shadow image and shadow mask image; Inputting the shadow image and the shadow mask image into the mask depth fusion module of the Transformer model to obtain a feature vector matrix; Input the feature vector matrix into the attention mechanism module of the Transformer model to obtain a shadow-removed image; The shadow image and the shadow mask image are input into the mask depth fusion module of the Transformer model to obtain a feature vector matrix, including: Binarizing and normalizing the shadow mask image to obtain a first shadow mask and a second shadow mask; binarizing the shadow mask image to obtain the first shadow mask, and normalizing the pixels in the shadow mask image to obtain the second shadow mask; The shadow image is subjected to a block embedding process to obtain a first eigenvector, and the second shadow mask is subjected to a block embedding process to obtain a second eigenvector; Fusing the first eigenvector, the first shadow mask, and the second eigenvector to obtain an eigenvector matrix; The first eigenvector, the first shadow mask and the second eigenvector are fused to obtain an eigenvector matrix, which is specifically expressed as: ， in, is the eigenvector matrix, is the first region weight, is the second region weight, is the first shadow mask, is the first eigenvector, is the second eigenvector.

2. The image shadow removal method according to claim 1, wherein: The Transformer model is trained with mask fusion loss.

3. The image shadow removal method according to claim 2, wherein: The mask fusion loss includes color consistency loss and weighted similarity loss, and the specific expression is: ， in, is the mask fusion loss, is the loss of color consistency, is the weighted similarity loss.

4. The image shadow removal method according to claim 3, wherein: The specific expression of the weighted similarity loss is: ； ； ； ； in, The weight assigned to the shaded area, The weight assigned to the non-shaded area, is the first shadow mask, is the loss based on structural similarity, is the average brightness of the output de-shadowed image, is the average brightness of the real image, is the pixel variance of the output de-shadowed image, is the variance of the pixels in the real image, is the pixel covariance of the output de-shadowed image and the true image, The output shadow-removed image The pixel value of each pixel, 、 is a preset constant, is the total number of pixels in the de-shadowed image.

5. An image shadow removal device, characterized in that: include: an acquisition unit, configured to acquire a shadow image and a shadow mask image; A preprocessing unit, configured to input the shadow image and the shadow mask image into a mask deep fusion module of a Transformer model to obtain a feature vector matrix; A shadow removal unit, configured to input the feature vector matrix into an attention mechanism module of a Transformer model to obtain a shadow-removed image; The shadow image and the shadow mask image are input into the mask depth fusion module of the Transformer model to obtain a feature vector matrix, including: Binarizing and normalizing the shadow mask image to obtain a first shadow mask and a second shadow mask; binarizing the shadow mask image to obtain the first shadow mask, and normalizing the pixels in the shadow mask image to obtain the second shadow mask; The shadow image is subjected to a block embedding process to obtain a first eigenvector, and the second shadow mask is subjected to a block embedding process to obtain a second eigenvector; Fusing the first eigenvector, the first shadow mask, and the second eigenvector to obtain an eigenvector matrix; The first eigenvector, the first shadow mask and the second eigenvector are fused to obtain an eigenvector matrix, which is specifically expressed as: ， in, is the eigenvector matrix, is the first region weight, is the second region weight, is the first shadow mask, is the first eigenvector, is the second eigenvector.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image shadow removal method according to any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the image shadow removal method according to any one of claims 1 to 4 is performed.

Images