Document image processing method and document image processing device

Abstract

This application provides a document image processing method and device. The method comprises: first, obtaining an initial document image and determining whether the initial document image contains shadows; then, if shadows exist in the initial document image, determining the normal vectors of each pixel in the initial document image; then, inputting the initial document image and the corresponding normal vectors of each pixel into at least three de-shadowing models to obtain at least three preliminary document images, each de-shadowing model having different constraints and corresponding loss functions; finally, fusing the at least three preliminary document images using a target loss function to obtain a target document image, where the target loss function is the weighted sum of the loss functions corresponding to the at least three de-shadowing models. This method improves the de-shadowing effect of the target document image and enhances its robustness, thereby resolving the problem of poor image de-shadowing effect in the prior art.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for processing document images. Background Art

[0002] The shadow removal technologies currently available on the market are mainly based on specific scenarios, such as road shadow removal, and mainly use traditional image processing solutions. First, the region of interest is extracted, and then morphological filtering and hole filling are performed in the region. Although these solutions are feasible, document shadow removal in photo shooting scenarios is not the same scenario. Document shadow removal has high requirements for the effect. Traditional solutions use the effect of removal on the document or cannot perform global shadow removal or make the subsequent content of the document unrecognizable, and cannot be used. Some available solutions do not describe the technical solutions, and the color distortion is serious after the shadow removal of color images, and the handwriting will appear broken, which cannot be achieved.

[0003] The above information disclosed in the background technology section is only used to enhance the understanding of the background technology of the technology described in this article. Therefore, the background technology may contain certain information that does not form the prior art known in this country to those skilled in the art. Summary of the Invention

[0004] The main purpose of the present application is to provide a document image processing method and a document image processing device to solve the problem of poor image shadow removal effect in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for processing a document image is provided, comprising: obtaining an initial document image and determining whether there is a shadow in the initial document image, the initial document image containing a document; in the case where there is a shadow in the initial document image, determining the normal vector of each pixel of the initial document image; inputting the initial document image and the normal vector of each corresponding pixel into at least three de-shadowing models to obtain at least three preliminary document images, the de-shadowing model being a model for removing shadows from an image obtained by training a convolutional neural network after back-propagation processing of a loss function, the constraints of each de-shadowing model being different and the corresponding loss function being different; using a target loss function to fuse at least three of the preliminary document images to obtain a target document image, the target loss function being obtained by weighted summation of the loss functions corresponding to at least three de-shadowing models.

[0006] Optionally, when there is a shadow in the initial document image, the normal vector of each pixel of the initial document image is determined, including: inputting the initial document image into a pre-trained normal vector prediction model to obtain the normal vector corresponding to the initial document image, the pre-trained normal vector prediction model is a model of the normal vectors of each pixel of the predicted image obtained by training a convolutional neural network.

[0007] Optionally, before inputting the initial document image and the normal vectors of the corresponding pixels into at least three de-shadowing models to obtain at least three preliminary document images, the method further includes: acquiring multiple sample images and the normal vectors corresponding to the multiple sample images; training at least three preliminary de-shadowing models under at least three constraints based on the sample images and the normal vectors corresponding to the sample images; and using at least three loss functions to perform back-propagation processing on the at least three preliminary de-shadowing models respectively to obtain at least three de-shadowing models.

[0008] Optionally, based on the sample image and the normal vector corresponding to the sample image, at least three preliminary shadow removal models are trained under at least three of the constraint conditions, including: under the constraint that the data of the predicted sample image is the product of the data of the sample image and the data of the shadow, training the sample image and the normal vector corresponding to the sample image to obtain a first preliminary shadow removal model; under the constraint that the data of the predicted sample image is the linear superposition of the data of the sample image and the data of the shadow, training the sample image and the normal vector corresponding to the sample image to obtain a second preliminary shadow removal model; directly training the sample image and the normal vector corresponding to the sample image to obtain a third preliminary shadow removal model.

[0009] Optionally, at least three loss functions are used to perform back propagation processing on at least three of the preliminary de-shadowing models respectively to obtain at least three de-shadowing models, including: using a first loss function to perform back propagation processing on the first preliminary de-shadowing model to obtain a first de-shadowing model, and the first loss function is a Relu function; using a second loss function to perform back propagation processing on the second preliminary de-shadowing model to obtain a second de-shadowing model, and the second loss function is a tanh function; using a third loss function to perform back propagation processing on the third preliminary de-shadowing model to obtain a third de-shadowing model, and the third loss function is a sigmoid function.

[0010] Optionally, the target loss function is: loss=aloss1+bloss2+closs3, where a, b, c are all constants greater than zero and a+b+c=1, loss1 is the first loss function, loss2 is the second loss function, and loss3 is the third loss function.

[0011] Optionally, before determining the normal vectors of each pixel of the initial document image, the method further includes: performing edge expansion processing on the initial document image, wherein the edge expansion processing is to expand the four edges of the initial document image to all sides respectively, and each of two opposite edges of a group of the initial document image is expanded by a first predetermined value, and the other group of two opposite edges is expanded by a second predetermined value, and the expanded area is filled with 0 pixels.

[0012] Optionally, obtaining an initial document image includes: obtaining an initial image and determining whether the initial image contains a document; if the initial image does not contain a document, obtaining a new initial image; if the initial image contains a document, calculating the height and width of the initial image, scaling the initial image, and obtaining the initial document image, wherein the height of the initial document image is an integer multiple of 256 and is closest to the height of the initial image, and the width of the initial document image is the product of the height of the initial document image and the aspect ratio of the initial image.

[0013] Optionally, the method further includes: performing binarization processing on the document image when no shadow exists in the initial document image to obtain the target document image.

[0014] According to another aspect of the present application, a device for removing shadows from document images is provided, comprising a first acquisition unit, a determination unit, a first processing unit, and a second processing unit, wherein the first acquisition unit is used to acquire an initial document image and determine whether the initial document image contains a shadow, the initial document image containing a document; the determination unit is used to determine the normal vector of each pixel of the initial document image when a shadow exists in the initial document image; the first processing unit is used to input the initial document image and the corresponding normal vector of each pixel into at least three removal of shadow models to obtain at least three preliminary document images, the removal of shadow models being models for removing shadows from images obtained by training a convolutional neural network after back-propagation processing of a loss function, the constraints of each of the removal of shadow models being different and the corresponding loss functions being different; the second processing unit is used to fuse at least three of the preliminary document images using a target loss function to obtain a target document image, the target loss function being obtained by weighted summation of the loss functions corresponding to at least three removal of shadow models.

[0015] Applying the technical solution of the present application, in the document image processing method, first, an initial document image is obtained and it is determined whether the initial document image contains a shadow, and the initial document image contains a document; then, when the initial document image contains a shadow, the normal vector of each pixel of the initial document image is determined; then, the initial document image and the normal vector of each corresponding pixel are input into at least three de-shadowing models to obtain at least three preliminary document images, and the de-shadowing model is a model for removing shadows from an image obtained by training a convolutional neural network after back-propagation processing of a loss function, and the constraints of each de-shadowing model are different and the corresponding loss functions are different; finally, a target loss function is used to fuse at least three of the preliminary document images to obtain a target document image, and the target loss function is obtained by weighted summation of the loss functions corresponding to at least three de-shadowing models. The method inputs an initial document image and a corresponding normal vector into at least three de-shadowing models to obtain at least three preliminary document images, and the de-shadowing model is a model for removing image shadows obtained by training a convolutional neural network after back-propagation processing using a loss function. The target loss function obtained by weighted summation of the loss functions corresponding to the at least three de-shadowing models is used to fuse the at least three preliminary document images, so that the de-shadowing effect of the target document image is improved and the robustness is better, thereby solving the problem of poor image de-shadowing effect in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0017] Figure 1 A flowchart of a method for processing a document image according to an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram showing a classification network structure according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram showing a network structure of a normal vector prediction model according to an embodiment of the present application is shown;

[0020] Figure 4 A schematic diagram of the structure of MobileNetV2Block according to an embodiment of the present application is shown;

[0021] Figure 5 A schematic diagram showing a network structure of a preliminary shadow removal model according to an embodiment of the present application is shown;

[0022] Figure 6A flowchart of a document image processing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element or intervening elements may be present. Moreover, in the specification and claims, when it is described that an element is "connected to" another element, the element may be "directly connected to" the other element or "connected to" the other element through a third element.

[0027] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0028] Convolutional Neural Networks (CNN): A type of feedforward neural network that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning.

[0029] Image enhancement: refers to the purposeful emphasis on the overall or local characteristics of an image, making the originally unclear image clear or emphasizing certain features of interest, expanding the differences between the features of different objects in the image, suppressing features of no interest, so as to improve image quality, enrich the amount of information, enhance image interpretation and recognition effects, and meet the needs of certain special analysis.

[0030] Image binarization: This process sets the grayscale value of each point on the image to 0 or 255, which makes the entire image appear black and white. This process involves selecting an appropriate threshold value for the 256 brightness levels of the grayscale image to obtain a binary image that still reflects both the overall and local features of the image.

[0031] LeNet-5 is a relatively simple convolutional neural network. The input 2D image first passes through two convolutional layers to the pooling layer, then passes through a fully connected layer, and finally uses softmax classification as the output layer.

[0032] As mentioned in the background art, the prior art has a poor effect in removing shadows from images. To solve the above problem, a typical embodiment of the present application provides a method for processing a document image and a device for processing a document image.

[0033] According to an embodiment of the present application, a method for processing a document image is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in a different order than that shown or described herein.

[0034] Figure 1 FIG is a flowchart of a method for processing a document image according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0035] Step S101, obtaining an initial document image and determining whether there is a shadow in the initial document image, wherein the initial document image contains a document;

[0036] Specifically, the initial document image can be classified by a pre-trained classification model to determine whether the initial document image has a shadow. The classification model can be built using any existing neural network, such as Figure 2 The component network structure in .

[0037] The most straightforward approach to removing shadows from document images is to retake a new image with normal shadows. However, this approach is time-consuming and laborious. The optimal solution lies in a post-processing algorithm designed based on the unique characteristics of document images. One approach is to binarize the image. This method uses the brightness difference between the foreground and background to distinguish between them using an adaptive threshold, achieving the goal of removing shadows. However, this approach is only effective when the shadows are simple; complex background shadows can easily cause residual images.

[0038] In order to be able to directly and quickly process the shadow-free image, in another embodiment of the present application, the above method also includes: binarizing the above document image when the above initial document image does not have a shadow, to obtain the above target document image.

[0039] In practical applications, the binarization algorithms used in binarization can be roughly divided into global thresholding and local thresholding. The global thresholding method uses a single threshold to divide the document image into two categories: characters (foreground) and background. For example, the Otsu algorithm uses the image's grayscale histogram to select an optimal threshold, maximizing the inter-class variance of foreground and background pixels after threshold segmentation. The local thresholding method (also known as the adaptive thresholding method) uses a sliding window convolution with the document image to set different thresholds for different parts of the image. Algorithms such as Niblack, Sauvola, and Wolf use the grayscale mean and variance within a pixel neighborhood to construct a threshold segmentation surface.

[0040] In another embodiment of the present application, obtaining an initial document image includes: obtaining an initial image and determining whether the initial image contains a document; if the initial image does not contain a document, obtaining a new initial image; if the initial image contains a document, calculating the height and width of the initial image, scaling the initial image, and obtaining the initial document image, wherein the height of the initial document image is an integer multiple of 256 and is closest to the height of the initial image, and the width of the initial document image is the product of the height of the initial document image and the aspect ratio of the initial image. Specifically, the initial image can be a picture taken by the user in any scene. If the initial image does not contain a document, a new initial image is obtained again, and the user can be prompted to retake an image with a document. If the initial image contains a document, the initial image is scaled, making subsequent data processing more convenient.

[0041] Specifically, the Lenet-5 classification network model can be used to classify the initial image into images containing and without documents. Furthermore, if the initial image containing the document has a height × width of 1100 × 1500, it can be scaled to the nearest multiple of 256 (i.e., 1024 / 1100), resulting in a height × width of 1024 × 1396, thus maximizing the accuracy of the original image after scaling back.

[0042] Step S102 : when the initial document image has a shadow, determining a normal vector of each pixel of the initial document image.

[0043] Specifically, for example, if the initial document image has wrinkles, the normal vector of each pixel on the image is different. By determining the normal vector of each pixel of the initial document image, the 3D information of the initial document image can be obtained. Adding 3D information in subsequent model training makes the training results more accurate and can improve the effect of audio and video removal.

[0044] In order to avoid the loss of document information due to possible distortion of the edges of the document image, in another embodiment of the present application, before determining the normal vectors of each pixel of the initial document image, the method further comprises: performing edge expansion processing on the initial document image, wherein the edge expansion processing is to expand the four edges of the initial document image in all directions, with two opposite edges of one group of the initial document image each expanded by a first predetermined value, and the other two opposite edges expanded by a second predetermined value, and the expanded area is filled with 0 pixels. The image is adaptively expanded up, down, left, and right, such as expanding the image area by 5% of its height at the top and 5% of its height at the bottom, and similarly expanding the left and right areas.

[0045] In another embodiment of the present application, in the presence of a shadow in the initial document image, determining the normal vector of each pixel of the initial document image includes: inputting the initial document image into a pre-trained normal vector prediction model to obtain the normal vector corresponding to the initial document image, wherein the pre-trained normal vector prediction model is a model of normal vectors of each pixel of the predicted image obtained by training a convolutional neural network. The normal vector prediction model can be used to quickly determine the normal vector of each pixel of the initial document image, and the model compression used in the normal vector prediction model can be implemented using a currently commonly used compression model.

[0046] In practical applications, the above normal vector prediction model can use the MobileNetV2 structure BigMBlock based on the model compression idea, such as Figure 3As shown, the efficiency is about 3 times higher than that of the Unet structure. The black vertical bars in BigMBlock represent feature maps, and the thick arrows represent convolutions. The convolution in the left half is 3×3 stride=2 with downsampling, and the right half is 3×3 convolution + upsampling. The MBlock module in the middle uses 4. The MBlock in the middle represents the MobileNetV2Block module. The specific structure is as follows Figure 4 As shown, the thin arrows indicate process flow and there is no related convolution operation.

[0047] Step S103: Inputting the initial document image and the normal vector of each corresponding pixel into at least three shadow removal models to obtain at least three preliminary document images, wherein the shadow removal models are models for removing shadows from an image obtained by training a convolutional neural network using a back-propagation loss function, and each shadow removal model has different constraints and corresponds to a different loss function.

[0048] In practical applications, at least three prepared document images are cropped according to the original ratio based on the scaling ratio of the edge expansion process to obtain images with the same size as the images before the edge expansion process, and then input into the subsequent normal vector training model.

[0049] In another embodiment of the present application, before inputting the initial document image and the normal vectors of the corresponding pixels into at least three de-shadowing models to obtain at least three preliminary document images, the method further includes: obtaining a plurality of sample images and the normal vectors corresponding to the plurality of sample images; training at least three preliminary de-shadowing models based on the sample images and the normal vectors corresponding to the sample images under at least three of the constraints; and performing backpropagation processing on the at least three preliminary de-shadowing models using at least three of the loss functions to obtain the at least three de-shadowing models. Performing backpropagation processing on the at least three preliminary de-shadowing models using at least three of the loss functions can improve the de-shadowing effect.

[0050] In order to further improve the effect of removing shadows, in another embodiment of the present application, based on the above-mentioned sample image and the above-mentioned normal vector corresponding to the above-mentioned sample image, at least three preliminary removing shadows models are trained under at least three of the above-mentioned constraint conditions, including: under the constraint that the data of the predicted sample image is the product of the data of the above-mentioned sample image and the data of the above-mentioned shadow, the above-mentioned sample image and the above-mentioned normal vector corresponding to the above-mentioned sample image are trained to obtain a first preliminary removing shadows model; under the constraint that the data of the above-mentioned predicted sample image is the linear superposition of the data of the above-mentioned sample image and the data of the above-mentioned shadow, the above-mentioned sample image and the above-mentioned normal vector corresponding to the above-mentioned sample image are trained to obtain a second preliminary removing shadows model; directly train the above-mentioned sample image and the above-mentioned normal vector corresponding to the above-mentioned sample image to obtain a third preliminary removing shadows model.

[0051] In order to greatly improve the details of the image and the continuity of the handwriting, and improve the subsequent recognition rate, in another embodiment of the present application, at least three of the above-mentioned loss functions are used to perform back propagation processing on at least three of the above-mentioned preliminary de-shadowing models, respectively, to obtain at least three of the above-mentioned de-shadowing models, including: using a first loss function to perform back propagation processing on the above-mentioned first preliminary de-shadowing model to obtain a first de-shadowing model, and the above-mentioned first loss function is a Relu function; using a second loss function to perform back propagation processing on the above-mentioned second preliminary de-shadowing model to obtain a second de-shadowing model, and the above-mentioned second loss function is a tanh function; using a third loss function to perform back propagation processing on the above-mentioned third preliminary de-shadowing model to obtain a third de-shadowing model, and the above-mentioned third loss function is a sigmoid function.

[0052] In practical applications, the model compression used in the preparatory shadow removal model can be implemented using the currently commonly used compression model and can be processed using BigMBlock. The specific network structure is as follows Figure 5As shown, A represents the predicted sample image. The input model inputs the original image I: 3×256×256 (number of channels × height × width) and the normal vector N: 3×256×256 (number of channels × height × width). The feature map value can be between 0 and 1. After that, through three different channels, the framework used by BigMBlock has the same structure as the normal vector prediction model. Among them, the first preliminary shadow removal model BigMBlock1 predicts the shadow feature map, which is defined as (I+σ)*S=A. That is, the predicted sample image A can be considered as the product of the original image I and the shadow S. σ is a small constant. BigMBlock1 predicts S1 as the shadow image of the original image. Then, Relu(S)=Relu(A / (I+σ)) can be used as the final loss function loss1 to perform backpropagation processing on the first preliminary shadow removal model. Among them, I+σ is used to avoid infinity due to the denominator being zero, and A / (I+σ) takes values ​​between 0 and infinity, so the activation function is selected as ReLU to ensure the same value range; the second preliminary shadow removal model BigMBlock2 also predicts the shadow feature map and is defined as IA=tanh(S), that is, it is considered that the predicted sample image A can be considered as the linear superposition of the original image I and the shadow S, and S2 predicted by BigMBlock2 is considered to be the shadow image of the original image, where IA takes values ​​between -1 and +1, so the activation function is selected as tanh to meet the value range, and is directly compared as loss2, and the second preliminary shadow removal model is back-propagated; the third preliminary shadow removal model BigMBlock3 directly predicts the shadow removal map and is defined as sigmoid(S)=sigmoid(A), that is, it is considered that the original image I and the normal vector can be directly predicted to the predicted sample image, and A1 predicted by BigMBlock3 is considered to be the shadow-removed image of the original image. The predicted image and A are directly compared to obtain loss3, and the third preliminary shadow removal model is back-propagated.

[0053] Specifically, the above loss1, loss2 and loss3 are all L1loss. During the experiment, it was found that directly measuring the shadow-removed picture has a good effect. However, therefore, three branches are designed to use the predicted shadows. The two branches fine-tune the model from the loss perspective. The final designed loss = 0.5loss3+0.25loss1+0.25loss2, and the model can be trained directly.

[0054] Step S104 , fusing at least three of the prepared document images using a target loss function to obtain a target document image. The target loss function is obtained by weighted summing the loss functions corresponding to at least three shadow removal models.

[0055] In order to avoid noise on the edges of the restored image when the light is very dim, in another embodiment of the present application, the above-mentioned target loss function is: loss = aloss1 + bloss2 + closs3, wherein a, b, c are all constants greater than zero and a+b+c=1, loss1 is the above-mentioned first loss function, loss2 is the above-mentioned second loss function, and loss3 is the above-mentioned third loss function.

[0056] In practical applications, after obtaining the target document image, the method further includes: scaling the target document image so that the height and width of the target document image are consistent with the height and width of the initial image.

[0057] Applying the technical solution of the present application, in the above-mentioned document image processing method, first, an initial document image is obtained and it is determined whether the above-mentioned initial document image contains a shadow, and the above-mentioned initial document image contains a document; then, in the case that the above-mentioned initial document image contains a shadow, the normal vector of each pixel of the above-mentioned initial document image is determined; then, the above-mentioned initial document image and the above-mentioned normal vector of each corresponding pixel are input into at least three de-shadowing models to obtain at least three preliminary document images, and the above-mentioned de-shadowing model is a model for removing shadows of an image obtained by training a convolutional neural network after back-propagation processing of a loss function, and the constraints of each of the above-mentioned de-shadowing models are different and the corresponding loss functions are different; finally, the target loss function is used to fuse the at least three above-mentioned preliminary document images to obtain a target document image, and the above-mentioned target loss function is obtained by weighted summation of the above-mentioned loss functions corresponding to at least three de-shadowing models. The method inputs an initial document image and a corresponding normal vector into at least three de-shadowing models to obtain at least three preliminary document images, and the de-shadowing model is a model for removing image shadows obtained by training a convolutional neural network after back-propagation processing using a loss function. The target loss function obtained by weighted summation of the above-mentioned loss functions corresponding to the at least three de-shadowing models is used to fuse the at least three preliminary document images, so that the de-shadowing effect of the target document image is improved and the robustness is better, thereby solving the problem of poor image de-shadowing effect in the prior art.

[0058] The present application also provides a document image processing device. It should be noted that the document image processing device of the present application can be used to execute the document image processing method provided in the present application. The document image processing device provided in the present application is introduced below.

[0059] Figure 6 FIG is a schematic diagram of a document image processing device according to an embodiment of the present application. Figure 6As shown, the device includes a first acquisition unit 10, a determination unit 20, a first processing unit 30 and a second processing unit 40, wherein:

[0060] A first acquiring unit 10 is configured to acquire an initial document image and determine whether a shadow exists in the initial document image, wherein the initial document image contains a document;

[0061] a determining unit 20, configured to determine a normal vector of each pixel of the initial document image when a shadow exists in the initial document image;

[0062] a first processing unit 30 configured to input the initial document image and the normal vector of each corresponding pixel into at least three shadow removal models to obtain at least three preliminary document images, wherein the shadow removal models are shadow removal models obtained by training a convolutional neural network using a back-propagation loss function, each shadow removal model having different constraints and corresponding loss functions;

[0063] The second processing unit 40 is configured to fuse the at least three prepared document images using a target loss function to obtain a target document image. The target loss function is obtained by weighted summing the loss functions corresponding to at least three shadow removal models.

[0064] The above-mentioned document image processing device obtains an initial document image through the above-mentioned first acquisition unit and determines whether there is a shadow in the above-mentioned initial document image, and the above-mentioned initial document image contains a document; determines the normal vector of each pixel of the above-mentioned initial document image through the above-mentioned determination unit when there is a shadow in the above-mentioned initial document image; inputs the above-mentioned initial document image and the above-mentioned normal vector of each corresponding pixel into at least three de-shadowing models through the above-mentioned first processing unit to obtain at least three preliminary document images, and the above-mentioned de-shadowing model is a model for removing shadows of the image obtained by training based on a convolutional neural network after back-propagation processing of a loss function, and the constraints of each above-mentioned de-shadowing model are different and the corresponding above-mentioned loss functions are different; and fuses at least three above-mentioned preliminary document images using a target loss function through the above-mentioned second processing unit to obtain a target document image, and the above-mentioned target loss function is obtained by weighted summation of the above-mentioned loss functions corresponding to at least three de-shadowing models. The device inputs the initial document image and the corresponding normal vector into at least three de-shadowing models to obtain at least three preliminary document images, and the de-shadowing model is a model for removing image shadows obtained by training a convolutional neural network after back-propagation processing of a loss function. The target loss function obtained by weighted summation of the above-mentioned loss functions corresponding to at least three de-shadowing models is used to fuse the at least three preliminary document images, so that the de-shadowing effect of the target document image is improved and the robustness is better, thereby solving the problem of poor image de-shadowing effect in the prior art.

[0065] Specifically, the initial document image can be classified by a pre-trained classification model to determine whether the initial document image has a shadow. The classification model can be built using any existing neural network, such as Figure 2 Furthermore, if the initial document image has wrinkles, the normal vector of each pixel in the image will be different. By determining the normal vector of each pixel in the initial document image, we can obtain the 3D information of the initial document image. Incorporating this 3D information into subsequent model training makes the training results more accurate and improves the effect of de-voicing.

[0066] In practical applications, at least three preliminary document images are cropped to their original proportions based on the scaling ratios of the edge expansion process, resulting in images of the same size as the pre-edge expansion images, which are then input into a subsequent normal vector training model. After obtaining the target document image, the method further includes scaling the target document image so that its height and width are consistent with those of the initial image.

[0067] The most straightforward approach to removing shadows from document images is to retake a new image with normal shadows. However, this approach is time-consuming and laborious. The optimal solution lies in a post-processing algorithm designed based on the unique characteristics of document images. One approach is to binarize the image. This method uses the brightness difference between the foreground and background to distinguish between them using an adaptive threshold, achieving the goal of removing shadows. However, this approach is only effective when the shadows are simple; complex background shadows can easily cause residual images.

[0068] In another embodiment of the present application, the determination unit includes an input module configured to input the initial document image into a pre-trained normal vector prediction model to obtain the normal vector corresponding to the initial document image. The pre-trained normal vector prediction model is a model of normal vectors for each pixel of the predicted image obtained through training of a convolutional neural network. The normal vector prediction model can be used to quickly determine the normal vectors for each pixel of the initial document image, and the model compression used in the normal vector prediction model can be implemented using a currently commonly used compression model.

[0069] In practical applications, the above normal vector prediction model can use the MobileNetV2 structure BigMBlock based on the model compression idea, such as Figure 3As shown, the efficiency is about 3 times higher than that of the Unet structure. The black vertical bars in BigMBlock represent feature maps, and the thick arrows represent convolutions. The convolution in the left half is 3×3 stride=2 with downsampling, and the right half is 3×3 convolution + upsampling. The MBlock module in the middle uses 4. The MBlock in the middle represents the MobileNetV2Block module. The specific structure is as follows Figure 4 As shown, the thin arrows indicate process flow and there is no related convolution operation.

[0070] In another embodiment of the present application, before the initial document image and the corresponding normal vectors of each pixel are input into at least three de-shadowing models to obtain at least three preliminary document images, the apparatus further includes a second acquisition unit, a training unit, and a third processing unit, wherein the second acquisition unit is used to acquire a plurality of sample images and the normal vectors corresponding to the plurality of sample images; the training unit is used to train the sample images and the normal vectors corresponding to the sample images under at least three of the above-mentioned constraints to obtain at least three preliminary de-shadowing models; and the third processing unit is used to perform back-propagation processing on the at least three preliminary de-shadowing models using at least three of the above-mentioned loss functions to obtain at least three of the above-mentioned de-shadowing models. Performing back-propagation processing on the at least three preliminary de-shadowing models using at least three of the above-mentioned loss functions can improve the de-shadowing effect.

[0071] In order to further improve the effect of de-shadowing, in another embodiment of the present application, the above-mentioned training unit includes a first training module, a second training module and a third training module, wherein the above-mentioned first training module is used to train the above-mentioned sample image and the above-mentioned normal vector corresponding to the above-mentioned sample image under the constraint that the data of the predicted sample image is the product of the data of the above-mentioned sample image and the data of the above-mentioned shadow, so as to obtain a first preliminary de-shadowing model; the above-mentioned second training module is used to train the above-mentioned sample image and the above-mentioned normal vector corresponding to the above-mentioned sample image under the constraint that the data of the above-mentioned predicted sample image is the linear superposition of the data of the above-mentioned sample image and the data of the above-mentioned shadow, so as to obtain a second preliminary de-shadowing model; the above-mentioned third training module is used to directly train the above-mentioned sample image and the above-mentioned normal vector corresponding to the above-mentioned sample image, so as to obtain a third preliminary de-shadowing model.

[0072] In order to greatly improve the details of the image and the continuity of the handwriting, and improve the subsequent recognition rate, in another embodiment of the present application, the above-mentioned third processing unit includes a first processing module, a second processing module and a third processing module, wherein the above-mentioned first processing module is used to adopt a first loss function to perform back propagation processing on the above-mentioned first preliminary de-shadowing model to obtain a first de-shadowing model, and the above-mentioned first loss function is a Relu function; the above-mentioned second processing module is used to adopt a second loss function to perform back propagation processing on the above-mentioned second preliminary de-shadowing model to obtain a second de-shadowing model, and the above-mentioned second loss function is a tanh function; the above-mentioned third processing module is used to adopt a third loss function to perform back propagation processing on the above-mentioned third preliminary de-shadowing model to obtain a third de-shadowing model, and the above-mentioned third loss function is a sigmoid function.

[0073] In practical applications, the model compression used in the preparatory shadow removal model can be implemented using the currently commonly used compression model and can be processed using BigMBlock. The specific network structure is as follows Figure 5As shown, A represents the predicted sample image. The input model inputs the original image I: 3×256×256 (number of channels × height × width) and the normal vector N: 3×256×256 (number of channels × height × width). The feature map value can be between 0 and 1. After that, through three different channels, the framework used by BigMBlock has the same structure as the normal vector prediction model. Among them, the first preliminary shadow removal model BigMBlock1 predicts the shadow feature map, which is defined as (I+σ)*S=A. That is, the predicted sample image A can be considered as the product of the original image I and the shadow S. σ is a small constant. BigMBlock1 predicts S1 as the shadow image of the original image. Then, Relu(S)=Relu(A / (I+σ)) can be used as the final loss function loss1 to perform backpropagation processing on the first preliminary shadow removal model. Among them, I+σ is used to avoid infinity due to the denominator being zero, and A / (I+σ) takes values ​​between 0 and infinity, so the activation function is selected as ReLU to ensure the same value range; the second preliminary shadow removal model BigMBlock2 also predicts the shadow feature map and is defined as IA=tanh(S), that is, it is considered that the predicted sample image A can be considered as the linear superposition of the original image I and the shadow S, and S2 predicted by BigMBlock2 is considered to be the shadow image of the original image, where IA takes values ​​between -1 and +1, so the activation function is selected as tanh to meet the value range, and is directly compared as loss2, and the second preliminary shadow removal model is back-propagated; the third preliminary shadow removal model BigMBlock3 directly predicts the shadow removal map and is defined as sigmoid(S)=sigmoid(A), that is, it is considered that the original image I and the normal vector can be directly predicted to the predicted sample image, and A1 predicted by BigMBlock3 is considered to be the shadow-removed image of the original image. The predicted image and A are directly compared to obtain loss3, and the third preliminary shadow removal model is back-propagated.

[0074] Specifically, the above loss1, loss2 and loss3 are all L1loss. During the experiment, it was found that directly measuring the shadow-removed picture has a good effect. However, therefore, three branches are designed to use the predicted shadows. The two branches fine-tune the model from the loss perspective. The final designed loss = 0.5loss3+0.25loss1+0.25loss2, and the model can be trained directly.

[0075] In order to avoid noise on the edges of the restored image when the light is very dim, in another embodiment of the present application, the above-mentioned target loss function is: loss = aloss1 + bloss2 + closs3, where a, b, and c are all constants greater than zero and a+b+c=1, loss1 is the above-mentioned first loss function, loss2 is the above-mentioned second loss function, and loss3 is the above-mentioned third loss function.

[0076] In order to avoid the loss of document information due to possible distortion of the edges of the document image, in another embodiment of the present application, before determining the normal vectors of each pixel of the initial document image, the apparatus further comprises a fourth processing unit for performing edge expansion processing on the initial document image. The edge expansion processing comprises expanding the four edges of the initial document image in all directions, with two opposite edges of one group of the initial document image each expanded by a first predetermined value, and two opposite edges of another group expanded by a second predetermined value, and the expanded area is filled with 0 pixels. The image is adaptively expanded up, down, left, and right, such as expanding the image area by 5% of its height at the top and 5% of its height at the bottom, and similarly expanding the left and right areas.

[0077] In another embodiment of the present application, the first acquisition unit includes a first acquisition module, a second acquisition module, and a calculation module, wherein the first acquisition module is used to acquire an initial image and determine whether the initial image contains a document; the second acquisition module is used to acquire a new initial image if the initial image does not contain a document; the calculation module is used to calculate the height and width of the initial image if the initial image contains a document, scale the initial image, and obtain the initial document image, wherein the height of the initial document image is an integer multiple of 256 and is closest to the height of the initial image, and the width of the initial document image is the product of the height of the initial document image and the aspect ratio of the initial image. Specifically, the initial image can be a picture taken by the user in any scene. When the initial image does not contain a document, a new initial image is acquired again, and the user can be prompted to retake an image with a document. When the initial image contains a document, the initial image is scaled, making subsequent data processing more convenient.

[0078] Specifically, the Lenet-5 classification network model can be used to classify the initial image into images containing and without documents. Furthermore, if the initial image containing the document has a height × width of 1100 × 1500, it can be scaled to the nearest multiple of 256 (i.e., 1024 / 1100), resulting in a height × width of 1024 × 1396, thus maximizing the accuracy of the original image after scaling back.

[0079] In order to be able to directly and quickly process the shadow-free image, in another embodiment of the present application, the above-mentioned device also includes a fifth processing unit, and the above-mentioned fifth processing unit is used to binarize the above-mentioned document image when there is no shadow in the above-mentioned initial document image to obtain the above-mentioned target document image.

[0080] In practical applications, the binarization algorithms used in binarization can be roughly divided into global thresholding and local thresholding. The global thresholding method uses a single threshold to divide the document image into two categories: characters (foreground) and background. For example, the Otsu algorithm uses the image's grayscale histogram to select an optimal threshold, maximizing the inter-class variance of foreground and background pixels after threshold segmentation. The local thresholding method (also known as the adaptive thresholding method) uses a sliding window convolution with the document image to set different thresholds for different parts of the image. Algorithms such as Niblack, Sauvola, and Wolf use the grayscale mean and variance within a pixel neighborhood to construct a threshold segmentation surface.

[0081] The document image processing device includes a processor and a memory. The first acquisition unit, determination unit, first processing unit and second processing unit are all stored in the memory as program units. The processor executes the program units stored in the memory to realize corresponding functions.

[0082] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be provided, and the problem of poor image deshading effect in the prior art can be solved by adjusting kernel parameters.

[0083] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0084] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device containing the computer-readable storage medium is controlled to execute the document image processing method.

[0085] An embodiment of the present invention provides a processor, which is used to run a program, wherein the document image processing method is executed when the program is run.

[0086] An embodiment of the present invention provides a device, comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are performed:

[0087] Step S101, obtaining an initial document image and determining whether there is a shadow in the initial document image, wherein the initial document image contains a document;

[0088] Step S102, when the initial document image has a shadow, determining a normal vector of each pixel of the initial document image;

[0089] Step S103: Inputting the initial document image and the normal vector of each corresponding pixel into at least three shadow removal models to obtain at least three preliminary document images, wherein the shadow removal models are models for removing shadows from an image obtained by training a convolutional neural network using a back-propagation loss function, and each shadow removal model has different constraints and corresponds to a different loss function.

[0090] Step S104 , fusing at least three of the prepared document images using a target loss function to obtain a target document image. The target loss function is obtained by weighted summing the loss functions corresponding to at least three shadow removal models.

[0091] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0092] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:

[0093] Step S101, obtaining an initial document image and determining whether there is a shadow in the initial document image, wherein the initial document image contains a document;

[0094] Step S102, when the initial document image has a shadow, determining a normal vector of each pixel of the initial document image;

[0095] Step S103: Inputting the initial document image and the normal vector of each corresponding pixel into at least three shadow removal models to obtain at least three preliminary document images, wherein the shadow removal models are models for removing shadows from an image obtained by training a convolutional neural network using a back-propagation loss function, and each shadow removal model has different constraints and corresponds to a different loss function.

[0096] Step S104 , fusing at least three of the prepared document images using a target loss function to obtain a target document image. The target loss function is obtained by weighted summing the loss functions corresponding to at least three shadow removal models.

[0097] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the above-mentioned units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0099] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0100] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0101] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0102] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0103] 1) Applying the technical solution of the present application, in the method for processing the above-mentioned document image, first, an initial document image is obtained and it is determined whether the above-mentioned initial document image contains a shadow, and the above-mentioned initial document image contains a document; then, in the case that the above-mentioned initial document image contains a shadow, the normal vector of each pixel of the above-mentioned initial document image is determined; then, the above-mentioned initial document image and the above-mentioned normal vector of each corresponding pixel are input into at least three de-shadowing models to obtain at least three preliminary document images, and the above-mentioned de-shadowing model is a model for removing shadows of an image obtained by training a convolutional neural network after back-propagation processing of a loss function, and the constraints of each of the above-mentioned de-shadowing models are different and the corresponding loss functions are different; finally, a target loss function is used to fuse the at least three above-mentioned preliminary document images to obtain a target document image, and the above-mentioned target loss function is obtained by weighted summation of the above-mentioned loss functions corresponding to at least three de-shadowing models. The method inputs an initial document image and a corresponding normal vector into at least three de-shadowing models to obtain at least three preliminary document images, and the de-shadowing model is a model for removing image shadows obtained by training a convolutional neural network after back-propagation processing using a loss function. The target loss function obtained by weighted summation of the above-mentioned loss functions corresponding to the at least three de-shadowing models is used to fuse the at least three preliminary document images, so that the de-shadowing effect of the target document image is improved and the robustness is better, thereby solving the problem of poor image de-shadowing effect in the prior art.

[0104] 2) A device for processing the above-mentioned document image, which obtains an initial document image through the above-mentioned first acquisition unit and determines whether there is a shadow in the above-mentioned initial document image, wherein the above-mentioned initial document image contains a document; determines the normal vector of each pixel of the above-mentioned initial document image through the above-mentioned determination unit when there is a shadow in the above-mentioned initial document image; inputs the above-mentioned initial document image and the above-mentioned normal vector of each corresponding pixel into at least three de-shadowing models through the above-mentioned first processing unit to obtain at least three preliminary document images, wherein the above-mentioned de-shadowing model is a model for removing shadows of an image obtained by training a convolutional neural network after back-propagation processing of a loss function, and each of the above-mentioned de-shadowing models has different constraints and corresponds to a different above-mentioned loss function; and fuses at least three of the above-mentioned preliminary document images using a target loss function through the above-mentioned second processing unit to obtain a target document image, wherein the above-mentioned target loss function is obtained by weighted summation of the above-mentioned loss functions corresponding to at least three de-shadowing models. The device inputs the initial document image and the corresponding normal vector into at least three de-shadowing models to obtain at least three preliminary document images, and the de-shadowing model is a model for removing image shadows obtained by training a convolutional neural network after back-propagation processing of a loss function. The target loss function obtained by weighted summation of the above-mentioned loss functions corresponding to at least three de-shadowing models is used to fuse the at least three preliminary document images, so that the de-shadowing effect of the target document image is improved and the robustness is better, thereby solving the problem of poor image de-shadowing effect in the prior art.

[0105] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for processing a document image, characterized in that: include: Acquire an initial document image and determine whether a shadow exists in the initial document image, wherein the initial document image contains a document; In the case where the initial document image has a shadow, determining a normal vector of each pixel of the initial document image; Inputting the initial document image and the normal vector of each corresponding pixel into at least three shadow removal models to obtain at least three preliminary document images, wherein the shadow removal models are models for removing shadows from images obtained by training a convolutional neural network using a back-propagation loss function, and each shadow removal model has different constraints and corresponds to a different loss function; A target loss function is used to perform fusion processing on at least three of the prepared document images to obtain a target document image. The target loss function is obtained by weighted summation of the loss functions corresponding to at least three shadow removal models.

2. The method according to claim 1, characterized in that In a case where the initial document image has a shadow, determining a normal vector of each pixel of the initial document image includes: The initial document image is input into a pre-trained normal vector prediction model to obtain the normal vector corresponding to the initial document image. The pre-trained normal vector prediction model is a model of the normal vectors of each pixel of the predicted image obtained by convolutional neural network training.

3. The method according to claim 1, characterized in that Before inputting the initial document image and the corresponding normal vectors of each pixel into at least three shadow removal models to obtain at least three preliminary document images, the method further includes: Acquire a plurality of sample images and the normal vectors corresponding to the plurality of sample images; Based on the sample image and the normal vector corresponding to the sample image, training under at least three constraint conditions to obtain at least three preliminary shadow removal models; At least three loss functions are used to perform back-propagation processing on at least three preliminary shadow removal models respectively to obtain at least three shadow removal models.

4. The method according to claim 3, characterized in that Based on the sample image and the normal vector corresponding to the sample image, at least three preliminary shadow removal models are trained under at least three constraint conditions, including: Under the constraint that the data of the predicted sample image is the product of the data of the sample image and the data of the shadow, training the sample image and the normal vector corresponding to the sample image to obtain a first preliminary shadow removal model; Under the constraint that the data of the predicted sample image is a linear superposition of the data of the sample image and the data of the shadow, training the sample image and the normal vector corresponding to the sample image to obtain a second preliminary shadow removal model; The sample image and the normal vector corresponding to the sample image are directly trained to obtain a third preliminary shadow removal model.

5. The method according to claim 4, characterized in that Backpropagation processing is performed on the at least three preliminary shadow removal models using at least three loss functions to obtain at least three shadow removal models, including: Performing backpropagation processing on the first preliminary shadow removal model using a first loss function to obtain a first shadow removal model, where the first loss function is a ReLU function; Performing backpropagation processing on the second preliminary shadow removal model using a second loss function to obtain a second shadow removal model, where the second loss function is a tanh function; A third loss function is used to perform back-propagation processing on the third preliminary shadow removal model to obtain a third shadow removal model, where the third loss function is a sigmoid function.

6. The method according to claim 5, characterized in that The objective loss function is: loss=aloss1+bloss2+closs3, where a, b, and c are all constants greater than zero and a+b+c=1, loss1 is the first loss function, loss2 is the second loss function, and loss3 is the third loss function.

7. The method according to any one of claims 1 to 6, characterized in that Before determining the normal vector of each pixel of the initial document image, the method further includes: The initial document image is subjected to edge expansion processing, wherein the edge expansion processing is to expand the four edges of the initial document image in all directions respectively, a group of two opposite edges of the initial document image are each expanded by a first predetermined value, and another group of two opposite edges are expanded by a second predetermined value, and the expanded area is filled with 0 pixels.

8. The method according to any one of claims 1 to 6, characterized in that Get the initial document image, including: Acquire an initial image, and determine whether the initial image contains a document; When the initial image does not contain a document, obtaining a new initial image; In the case that the initial image contains a document, the height and width of the initial image are calculated, and the initial image is scaled to obtain the initial document image, where the height of the initial document image is an integer multiple of 256 and is closest to the height of the initial image, and the width of the initial document image is the product of the height of the initial document image and the aspect ratio of the initial image.

9. The method according to any one of claims 1 to 6, characterized in that The method further comprises: When the initial document image does not have a shadow, the document image is binarized to obtain the target document image.

10. A device for removing shadows from document images, characterized in that: include: A first acquiring unit is configured to acquire an initial document image and determine whether a shadow exists in the initial document image, wherein the initial document image contains a document; a determining unit, configured to determine a normal vector of each pixel of the initial document image when a shadow exists in the initial document image; a first processing unit, configured to input the initial document image and the normal vector of each corresponding pixel into at least three shadow removal models to obtain at least three preliminary document images, wherein the shadow removal models are models for removing shadows from an image obtained by training a convolutional neural network and subjected to back-propagation processing using a loss function, and each shadow removal model has different constraints and corresponds to a different loss function; The second processing unit is used to use a target loss function to fuse at least three of the prepared document images to obtain a target document image, where the target loss function is obtained by weighted summation of the loss functions corresponding to at least three shadow removal models.

Images