Document shadow removing method based on light correction

Abstract

The invention discloses a document shadow removal method based on light correction, which belongs to the technical field of artificial intelligence and image processing, is realized through a feature extraction module, a shadow mask generation module, a light correction module and an image restoration module, and comprises the following steps: S1, performing feature extraction on a document image with a shadow by adopting the feature extraction module; s2, generating a shadow soft mask by adopting a shadow mask generation module; s3, correcting the dark shadow of the document by adopting a light correction module; and S4, processing the shadow soft mask obtained in the step S2, the light correction image output in the step S3 and the originally input shadow document image by adopting an image restoration module to generate a shadow-free document image. According to the optical principle of shadow formation, the light correction rule is directly learned through the neural network, the complexity and the calculation amount of the model are effectively reduced, the shadow removal efficiency is improved while the effect is ensured, the requirement for calculation equipment is reduced, and the method is more suitable for small mobile equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and image processing, and in particular to a document shadow removal method based on light correction. Background Art

[0002] With the widespread use of mobile devices, mobile phones have become people's preferred tool for scanning and saving documents. However, due to unfavorable lighting conditions or shooting angles, shadows often appear in documents taken with mobile phones, affecting the clarity and readability of the document, and even causing inaccurate or erroneous information. Removing shadows from document images can not only improve the overall quality and appearance of the document, but also help achieve more efficient data management, ensure that the document content is accurately recorded and reproduced, while optimizing the user experience, meeting the needs of digital office, and making documents easier to print, archive and use for a long time.

[0003] Existing document image shadow removal methods mainly directly learn the image pixel values ​​after removing the shadow. Although effective in theory, the method is relatively complex and requires high computing power. It needs the support of high-performance computers and is difficult to apply to small mobile devices such as mobile phones.

[0004] Based on this, a document image shadow removal method based on light correction is designed which is more suitable for small mobile devices. Summary of the invention

[0005] The purpose of the present invention is to provide a document shadow removal method based on light correction to solve the problems in the background technology.

[0006] To achieve the above object, the present invention provides a document shadow removal method based on light correction, which is implemented by a feature extraction module, a shadow mask generation module, a light correction module and an image restoration module, and specifically includes the following steps:

[0007] S1, using a feature extraction module to extract features from the channel dimension and local spatial information of the original input shadowed document image, and extracting a first feature image;

[0008] S2, using a shadow mask generation module to process the first feature image obtained in S1 to generate a shadow soft mask;

[0009] S3, using a light correction module to correct the dark shadow of the document on the first feature image obtained in S1 and the shadow soft mask obtained in S2, and outputting a light correction image;

[0010] S4, using an image restoration module to process the shadow soft mask obtained in S2, the light correction image output by S3, and the original input document image with shadows, to generate a document image without shadows.

[0011] Preferably, the S1 specifically comprises the following steps:

[0012] S11, performing an average pooling operation on the channel dimension of the input shadowed document image to compress it into a single-channel image;

[0013] S12, using the Softmax function to process the single-channel image to obtain the probability distribution of each pixel of the single-channel image;

[0014] S13, using the probability value at each position in the probability distribution, weighting the pixel value at the corresponding position in the input shadowed document image, and outputting a first feature image; the first feature image is represented as:

[0015]

[0016] Where Y1 is the first feature image, softmax(·) is the Softmax function processing process, avePool(·) is the average pooling operation, Represents element-wise matrix multiplication, where X is the shadowed document image.

[0017] Preferably, the S2 specifically includes the following steps:

[0018] S21, deleting the text in the first feature image obtained in S1 through an expansion operation to obtain a second feature image;

[0019] S22, smoothing the second feature image by using a median filter to obtain a third feature image;

[0020] S23, sorting all pixel values ​​in the third feature image to obtain a sequence, setting the proportion of shadows, selecting the first m pixel values ​​in the sequence as shadows according to the proportion of shadows, and calculating the average value of the first m pixel values;

[0021] S24. Perform normalization based on the result of S23 to obtain a shadow soft mask of the shadowed document image.

[0022] Preferably, in S23, the calculation formula of the average value is:

[0023]

[0024] In the formula, p mean is the average value of the first m pixel values, i is the pixel value in sequence I sort The position in I sort (i) indicates sequence I sort The i-th pixel value in .

[0025] Preferably, the calculation formula for the selected number of pixels m is:

[0026] m = a × (w × h), 0 <a<1;

[0027] Where a is the proportion of the shadow, w is the width of the third feature image, and h is the height of the third feature image.

[0028] Preferably, in S24, the calculation formula of the shadow soft mask is:

[0029]

[0030] In the formula, I m is the shadow soft mask, Y3 is the third feature image, p max Represents the maximum value of the first m pixel values.

[0031] Preferably, the S3 specifically includes the following steps:

[0032] S31. Build a convolutional neural network, which includes four convolutional layers and two shadow attention correction blocks;

[0033] S32, concatenating the first feature image and the shadow soft mask to obtain a tensor with a depth of D, inputting it into the convolution layer, and outputting a light correction image through the convolution layer and the shadow attention correction block;

[0034] S33. Define the light correction loss function as the L1 distance between the true shadow-free image and the light correction image and the weighted mean square error under the weighting of the shadow soft mask.

[0035] Preferably, the convolution layer in S31 adopts a three-dimensional equal-width convolution of P1×Q1×D, and uses a ReLU activation function to implement a nonlinear relationship between the output and input of the convolutional neural network;

[0036] Among them, P1 is the width of the convolution kernel, and Q1 is the height of the convolution kernel.

[0037] Preferably, the shadow attention correction block in S31 includes a two-level residual network, and the structure of the first-level residual network is: layer normalization, convolutional layer, convolutional layer, convolutional layer; the structure of the second-level residual network is: layer normalization, convolutional layer, convolutional layer.

[0038] Preferably, in S33, the calculation formula of L1 distance is:

[0039] L ill1 =||I l -I out ||1;

[0040] Where, L ill1 is the L1 distance between the true shadow-free image and the light-corrected image, Il is a true shadow-free image, I out Corrects the image for light.

[0041] Preferably, in S33, the weighted mean square error is calculated as follows:

[0042]

[0043] Where, L ill2 is the weighted mean square error of the true shadow-free image and the light-corrected image under the weighting of the soft shadow mask, n is the total number of pixels, is the value of the i-th pixel in the true shadow-free image, is the value of the th pixel in the light-corrected image, is the value of the i-th pixel in the shadow soft mask.

[0044] Preferably, in S4, the shadow-free document image is represented as:

[0045] O=X×(1-I m )+I out ×I m ;

[0046] Where O is the shadow-free document image.

[0047] Therefore, the present invention provides a document shadow removal method based on light correction. According to the optical principle of shadow formation, the correction law of light is directly learned through a neural network. Compared with the existing shadow removal method, the method can effectively reduce the complexity and calculation amount of the model, improve the efficiency of shadow removal while ensuring the effect, reduce the requirements for computing equipment, and is more suitable for small mobile devices.

[0048] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a system architecture diagram of an embodiment of the present invention;

[0050] Figure 2 Schematic diagram of feature extraction process of an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the network structure of a light correction module according to an embodiment of the present invention;

[0052] Figure 4 Schematic diagram of the network structure of the shadow correction attention block of an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0055] Example

[0056] like Figure 1 As shown, the document shadow removal method based on light correction provided by the present invention is implemented by a feature extraction module, a shadow mask generation module, a light correction module and an image restoration module, and specifically includes the following steps:

[0057] S1, such as Figure 2 As shown in the figure, the feature extraction module is used to extract the channel dimension and local spatial information of the original input document image with shadows, and the features of the text and shadow areas can be extracted. take out The first feature image is:

[0058] S11, performing an average pooling operation on the channel dimension of the input shadow document image (three-channel color image) with a height of H and a width of W, and compressing the shadow document image into a single-channel two-dimensional image, denoted as avePool(X);

[0059] S12. Use the Softmax function to process the single-channel image to obtain the probability distribution of each pixel of the single-channel image, which is recorded as softmax(avePool(X));

[0060] S13, using the probability value at each position in the probability distribution, weighting the pixel value at the corresponding position in the shadowed document image, and outputting a first feature image;

[0061] The first feature image is represented as:

[0062]

[0063] Where Y1 is the first feature image, softmax(·) is the Softmax function processing process, avePool(·) is the average pooling operation, Represents element-wise matrix multiplication, where X is the shadowed document image.

[0064] S2, using the shadow mask generation module to process the first feature image obtained in S1, to generate a shadow soft mask, which accurately reflects the information of the shadow strength, specifically:

[0065] S21, deleting the text in the first feature image obtained in S1 through the dilation operation in image processing, obtaining a second feature image, effectively avoiding misclassification of black text as shadows, thereby improving the accuracy of shadow detection, specifically:

[0066] 1) Set a window of size N×N and modify the pixel value at the center of the window to the maximum value of all pixels in the entire window;

[0067] 2) Slide the window across the image to traverse the entire image and repeat the above process;

[0068] S22, smoothing the second feature image by using median filtering to obtain a third feature image, specifically:

[0069] 1) Set a window of size M×M and sort all pixel values ​​in the window in ascending order;

[0070] 2) Modify the pixel value at the center of the window to the middle value of all pixel values ​​in the entire window after sorting;

[0071] 3) Slide the window across the entire image, so that it traverses the entire image, and repeat the above process;

[0072] S23, sorting all pixel values ​​in the third feature image to obtain a sequence, setting the proportion of shadows, selecting the first m pixel values ​​in the sequence as shadows according to the proportion of shadows, and calculating the average value of the first m pixel values, specifically:

[0073] Sorting all pixel values ​​in the third feature image obtained by S22, obtaining a sequence of pixel values ​​from small to large, denoted as I sort , the length of the sequence is w×h, w and h represent the width and height of the third feature image respectively. The smaller the pixel value in the sequence, the darker the color reflected on the image, and the greater the probability that it is a shadow; set the proportion of the shadow, select the first m pixel values ​​in the sequence as the shadow according to the proportion of the shadow, and calculate the average and maximum value of the first m pixel values;

[0074] Among them, the calculation formula for the selected pixel number m is:

[0075] m = a × (w × h), 0 <a<1;

[0076] In the formula, a is the proportion of shadow;

[0077] The calculation formula for the average value of the first m pixel values ​​is:

[0078]

[0079] In the formula, p meanis the average value of the first m pixel values, i is the pixel value in sequence I sort The position in I sort (i) indicates sequence I sort The i-th pixel value in .

[0080] S24, normalizing based on the result of S23 to obtain a shadow soft mask of the shadowed document image, the calculation formula of the shadow soft mask is:

[0081]

[0082] In the formula, I m is the shadow soft mask, Y3 is the third feature image, p max Represents the maximum value of the first m pixel values.

[0083] S3, such as Figure 3 As shown, the light correction module is used to correct the dark shadow of the document on the first feature image obtained by S1 and the shadow soft mask obtained by S2, and a light correction image is output, specifically:

[0084] S31. Build a convolutional neural network, which includes four convolutional layers and two shadow attention correction blocks;

[0085] In this embodiment, the convolution layer uses a 3x3x4 three-dimensional equal-width convolution to ensure that the output image and the input image have the same size, and uses a ReLU activation function to implement the nonlinear relationship between the output and input of the convolutional neural network;

[0086] like Figure 4 As shown in the figure, the shadow attention block contains two levels of residual networks. The structure of the first level residual network is: layer normalization, 1x1 convolution, 3x3 equal-width convolution and 1x1 convolution; the structure of the second level residual network is: layer normalization, 1x1 convolution and 1x1 convolution.

[0087] S32, concatenating the first feature image and the shadow soft mask to obtain a tensor with a depth of 4 as a net input, inputting it into the convolution layer, and outputting a light correction image through the convolution layer and the shadow attention correction block;

[0088] S33, define a light correction loss function to measure the difference between the light correction image and the true shadow-free image to ensure the accuracy of shadow removal; the light correction loss function includes the L1 distance between the true shadow-free image and the light correction image and the weighted mean square error under the weighting of the shadow soft mask, specifically:

[0089] 1) The calculation formula of L1 distance is:

[0090] L ill1 =||I l -Iout ||1;

[0091] Where, L ill1 is the L1 distance between the true shadow-free image and the light-corrected image, I l is a true shadow-free image, I out Corrects the image for light.

[0092] 2) The calculation formula of weighted mean square error is:

[0093]

[0094] Where, L ill2 is the weighted mean square error of the true shadow-free image and the light-corrected image under the weighting of the soft shadow mask, n is the total number of pixels, is the value of the i-th pixel in the true shadow-free image, is the value of the th pixel in the light-corrected image, is the value of the i-th pixel in the shadow soft mask.

[0095] S4, using the image restoration module to process the shadow soft mask obtained by S2, the light correction image output by S3, and the original input document image with shadows, to generate a document image without shadows, which is represented by:

[0096] O=X×(1-I m )+I out ×I m ;

[0097] Where O is the shadow-free document image.

[0098] Therefore, the present invention provides a document shadow removal method based on light correction. According to the optical principle of shadow formation, the correction law of light is directly learned through a neural network. Compared with the existing shadow removal method, the method can effectively reduce the complexity and calculation amount of the model, improve the efficiency of shadow removal while ensuring the effect, reduce the requirements for computing equipment, and is more suitable for small mobile devices.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A document shadow removal method based on light correction, implemented by a feature extraction module, a shadow mask generation module, a light correction module and an image restoration module, characterized in that: The specific steps include: S1, using a feature extraction module to extract features from the channel dimension and local spatial information of the original input shadowed document image, and extracting a first feature image; S2, using a shadow mask generation module to process the first feature image obtained in S1 to generate a shadow soft mask; S3, using a light correction module to correct the dark shadow of the document on the first feature image obtained in S1 and the shadow soft mask obtained in S2, and outputting a light correction image; S4, using an image restoration module to process the shadow soft mask obtained in S2, the light correction image output by S3, and the original input document image with shadows, to generate a document image without shadows.

2. The document shadow removal method based on light correction according to claim 1, characterized in that: The S1 specifically includes the following steps: S11, performing an average pooling operation on the channel dimension of the input shadowed document image to compress it into a single-channel image; S12, using the Softmax function to process the single-channel image to obtain the probability distribution of each pixel of the single-channel image; S13, using the probability value at each position in the probability distribution, weighting the pixel value at the corresponding position in the input shadowed document image, and outputting a first feature image; the first feature image is represented as: Where Y1 is the first feature image, softmax(·) is the Softmax function processing process, avePool(·) is the average pooling operation, Represents element-wise matrix multiplication, where X is the shadowed document image.

3. The document shadow removal method based on light correction according to claim 1, characterized in that: The S2 specifically includes the following steps: S21, deleting the text in the first feature image obtained in S1 through an expansion operation to obtain a second feature image; S22, smoothing the second feature image by using a median filter to obtain a third feature image; S23, sorting all pixel values ​​in the third feature image to obtain a sequence, setting the proportion of shadows, selecting the first m pixel values ​​in the sequence as shadows according to the proportion of shadows, and calculating the average value of the first m pixel values; S24. Perform normalization based on the result of S23 to obtain a shadow soft mask of the shadowed document image.

4. The document shadow removal method based on light correction according to claim 3, characterized in that: In S23, the calculation formula of the average value is: Where P mean is the average value of the first m pixel values, i is the pixel value in sequence I sort The position in I sort (i) indicates sequence I sort The i-th pixel value in .

5. The document shadow removal method based on light correction according to claim 4, characterized in that: In S24, the calculation formula of the shadow soft mask is: In the formula, I m is the shadow soft mask, Y3 is the third feature image, p max Represents the maximum value of the first m pixel values.

6. The method for removing document shadows based on light correction according to claim 1, characterized in that: The S3 specifically includes the following steps: S31. Build a convolutional neural network, which includes four convolutional layers and two shadow attention correction blocks; S32, concatenating the first feature image and the shadow soft mask to obtain a tensor with a depth of D, inputting it into the convolution layer, and outputting a light correction image through the convolution layer and the shadow attention correction block; S33. Define the light correction loss function as the L1 distance between the true shadow-free image and the light correction image and the weighted mean square error under the weighting of the shadow soft mask.

7. The method for removing document shadows based on light correction according to claim 6, characterized in that: In S33, the calculation formula of L1 distance is: L ill1 =||I l -I out ||1; Where, L ill1 is the L1 distance between the true shadow-free image and the light-corrected image, I l is a true shadow-free image, I out Corrects the image for light.

8. The method for removing document shadows based on light correction according to claim 7, characterized in that: In S33, the weighted mean square error is calculated as follows: Where, L ill2 is the weighted mean square error of the true shadow-free image and the light-corrected image under the weighting of the soft shadow mask, n is the total number of pixels, is the value of the i-th pixel in the true shadow-free image, is the value of the th pixel in the light-corrected image, is the value of the i-th pixel in the shadow soft mask.

9. The method for removing document shadows based on light correction according to claim 8, characterized in that: In S4, the shadow-free document image is represented as: O=X×(1-I m )+I out ×I m ; Where O is the shadow-free document image.

Images