Degenerated document image binarization method and system based on adaptive convolution

Abstract

The invention discloses a degraded document image binarization method and system based on adaptive convolution, and the method comprises the steps: 1, collecting a training degraded document image, and carrying out the preprocessing of the training degraded document image and a corresponding binarization label image; 2, inputting a pre-input image of the degraded document image into the adaptive convolutional network model to obtain a binary image of the degraded document image; 3, calculating to obtain an adversarial loss, a classification loss, a similarity loss, a Jaccard similarity coefficient loss and an average absolute value error; step 4, optimizing parameters of the adaptive convolutional network model to obtain an optimal adaptive convolutional network model; and 5, inputting a pre-input image of the target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image. The method can be applied to degraded document images of various forms of degradation, and high quality and high efficiency of the binarized image are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for binarizing degraded document images based on adaptive convolution. Background Art

[0002] Historical documents are an important part of Chinese history and culture. The images of these historical documents come from the folks, with highly personal styles and writing habits, so there are irregularities in ink marks and strokes, such as different lengths and thicknesses. In addition, degraded document manuscripts are often affected by factors such as moisture, insects, and light, resulting in increased image noise, and the manuscripts themselves may have problems such as character adhesion, typos, and seal coverage.

[0003] With the advancement of digitalization, the binarization of degraded document images has become an important research topic. Traditional methods have limited effect on complex degraded document images such as old, unevenly illuminated, or wrinkled. The rise of deep learning technology provides a new idea for solving such problems. Through large-scale data training and complex model construction, deep learning can effectively improve the quality and clarity of degraded document images while retaining the original information of degraded document images.

[0004] Recently, researchers have applied deep learning methods to the binarization task of historically degraded document images and have made some progress. However, there are still many limitations in dealing with the binarization problem of degraded document images. First, the existing network structure has limitations in dealing with these multiple degradation forms, mainly because the modeling standards in the feature extraction stage are relatively unified and do not fully consider the diverse degradation features, resulting in insufficient robustness and poor generalization of the model when facing different degradation forms. Secondly, the current mainstream neural network-based image binarization methods usually rely on multi-level architectures and additional inputs. The high model complexity makes the model too dependent on the learned inherent data pattern, and the model training is too complicated, which limits its adaptability to degraded document images with multiple degradations and affects subsequent research on degraded document images. Summary of the invention

[0005] In order to at least partially solve the problems of poor generalization, complex training and limited performance of deep learning-based image binarization methods in degraded document images, the present invention provides a degraded document image binarization method and system based on adaptive convolution, constructs an adaptive convolution network model, the encoder of the adaptive convolution network model includes an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolution network model includes a gated pyramid module, so that the adaptive convolution network model can be applied to degraded document images of various forms of degradation, thereby ensuring the high quality and efficiency of the binarized image.

[0006] In order to achieve the above object, the technical solution of the present invention is:

[0007] The first aspect of the present invention proposes a method for binarizing a degraded document image based on adaptive convolution, comprising:

[0008] Step 1: collect degraded document images, scan and annotate the degraded document images to obtain binary label maps of the degraded document images, pre-process the degraded document images and the corresponding binary label maps to obtain pre-input images of the degraded document images and pre-input images of the binary label maps, so as to facilitate training of the adaptive convolutional network model;

[0009] Step 2: inputting the pre-input image of the degraded document image into the adaptive convolutional network model to obtain a binary image of the degraded document image, so as to facilitate training of the adaptive convolutional network model;

[0010] The encoder of the adaptive convolutional network model includes an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolutional network model includes a gated pyramid module, so as to improve the accuracy of the output binary image;

[0011] Step 3: Calculate the adversarial loss based on the identification accuracy of the binary image of the degraded document image and the binary label map of the degraded document image, and use it for training the adaptive convolutional network model;

[0012] The classification loss is calculated based on the difference between the foreground pixel and the background pixel of the binary image of the degraded document image and the binary label map of the degraded document image, and is used for training the adaptive convolutional network model.

[0013] Similarity loss and Jaccard similarity coefficient loss are calculated based on the pixel similarity of the binary image of the degraded document image and the binary label map of the degraded document image, and are used for adaptive convolutional network model training;

[0014] The mean absolute error is calculated based on the pixel difference between the binary image of the degraded document image and the binary label map of the degraded document image, and is used for training the adaptive convolutional network model;

[0015] Step 4: Calculate the total loss based on adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, optimize the adaptive convolutional network model parameters by minimizing the total loss, and obtain the optimal adaptive convolutional network model;

[0016] Step 5: Input the pre-input image of the target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image.

[0017] Furthermore, the preprocessing of the degraded document image and the corresponding binarized label image specifically includes:

[0018] Divide the degraded document images and the corresponding binary label images into training sets and test sets to facilitate the training of the adaptive convolutional network model;

[0019] The degraded document images and the corresponding binary label images in the training set and the test set are cropped to obtain image patches of the degraded document images and image patches of the binary label images, which are convenient for processing by the adaptive convolutional network model and avoid the problem of long processing time caused by too large images.

[0020] The image patch of the degraded document image and the image patch of the binarized label map are input into the edge detection algorithm and the large law algorithm to obtain the pre-input image of the degraded document image and the pre-input image of the binarized label map.

[0021] Further, the adaptive convolution module includes a modulated convolution block, a dynamic weight subnet, a gradient information extraction block and a feature map extraction block;

[0022] The convolution kernel weight of the modulated convolution block is expressed by the following formula:

[0023]

[0024] Among them, W q is the modulated convolution kernel weight, R is the size of the convolution, W p is the convolution kernel weight, : is a placeholder;

[0025] The dynamic weight subnet is expressed by the following formula:

[0026] θ=σ(RELUCon 1×1 (DW 3×3 (PW 1×1 (I))))

[0027] Among them, θ is the proportion of differential information, σ is the Sigmoid function, I is the input feature map, DW 3×3 is a 3×3 depth convolution, PW 1×1 For 1×1 point convolution, RELUCon 1×1 is the ReLU function;

[0028] The gradient information extraction block is expressed as follows:

[0029] F′=W p ⊙F-θ×W q ⊙F

[0030] Among them, F′ is the feature map with gradient information, F is the feature map after convolution, and ⊙ is the element-by-element multiplication operation;

[0031] The feature map extraction block is expressed as follows:

[0032]

[0033] Among them, O is the extracted feature map, is the activation function.

[0034] Furthermore, the attention guiding module is expressed by the following formula:

[0035] W=σ(DW 5×5 (PW 1×1 (S[O,W cat ×O])))

[0036] Among them, W is the final feature weight, σ is the Sigmoid function, DW 5×5 is a 5×5 depth convolution, PW 1×1 is a 1×1 point convolution, S is a pixel rearrangement operation, O is an extracted feature map, and W cat is the mixed domain feature weight.

[0037] Furthermore, the gated pyramid module is expressed by the following formula:

[0038]

[0039] GDConv = gate(Q(I j C ))

[0040] Among them, GDConv is gated deconvolution, gate is the gate function, Q is the bilinear interpolation operation, + is the feature map addition fusion, j is the feature map level, C is the feature map to be fused, P is the fused feature map, I j C is the feature map passed down by downsampling, I j p It is the feature map after the fusion of features of different scales.

[0041] Furthermore, the total loss is calculated based on the adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, specifically including:

[0042]

[0043] Among them, L G is the total loss, L dav To combat losses, is the classification loss, is the similarity loss, is the Jaccard similarity coefficient loss, is the mean absolute error, λ bce is the classification loss weight, λ diceis the similarity loss weight, λ jaccard is the Jaccard similarity coefficient loss weight, λ l1 is the mean absolute error weight, λ adv To counter the loss weight.

[0044] The second aspect of the present invention proposes a degraded document image binarization system based on adaptive convolution, comprising:

[0045] A preprocessing module is used to collect degraded document images, scan and annotate the degraded document images to obtain a binary label map of the degraded document images, preprocess the degraded document images and the corresponding binary label maps to obtain a pre-input image of the degraded document images and a pre-input image of the binary label map, so as to facilitate training of the adaptive convolutional network model;

[0046] An adaptive convolutional network model module, used for inputting a pre-input image of a degraded document image into the adaptive convolutional network model to obtain a binary image of the degraded document image, so as to facilitate training of the adaptive convolutional network model;

[0047] The encoder of the adaptive convolutional network model includes an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolutional network model includes a gated pyramid module, so as to improve the accuracy of the output binary image;

[0048] A loss module, used for calculating the adversarial loss according to the discrimination accuracy of the binary image of the degraded document image and the binary label map of the degraded document image, for training the adaptive convolutional network model;

[0049] The classification loss is calculated based on the difference between the foreground pixel and the background pixel of the binary image of the degraded document image and the binary label map of the degraded document image, and is used for training the adaptive convolutional network model.

[0050] Similarity loss and Jaccard similarity coefficient loss are calculated based on the pixel similarity of the binary image of the degraded document image and the binary label map of the degraded document image, and are used for adaptive convolutional network model training;

[0051] The mean absolute error is calculated based on the pixel difference between the binary image of the degraded document image and the binary label map of the degraded document image, and is used for training the adaptive convolutional network model;

[0052] The training module is used to calculate the total loss based on adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, and optimize the adaptive convolutional network model parameters by minimizing the total loss to obtain the optimal adaptive convolutional network model;

[0053] The output module is used to input the pre-input image of the target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image.

[0054] The third aspect of the present invention proposes an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for binarization of degraded document images based on adaptive convolution as described in the first aspect above is implemented.

[0055] The fourth aspect of the present invention proposes a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a degraded document image binarization method based on adaptive convolution as described in the first aspect above.

[0056] Beneficial effects of the present invention:

[0057] (1) The adaptive convolutional network model proposed in this paper introduces an adaptive convolution module and a guided attention module in the encoder stage. The adaptive convolution module combines local and global features to enhance the transmission of effective information and suppress invalid information. Secondly, the guided attention module marks the degraded area, reduces background interference, and improves the accuracy of feature extraction and the robustness of the model. In addition, the gated pyramid module structure is used to improve feature fusion and enhance the transmission of shallow and high-level semantic information.

[0058] (2) The present invention calculates the total loss through adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, and then trains the adaptive convolutional network model. When calculating the adversarial loss, multiple loss functions are introduced in the generator and the weight setting is optimized. The local discriminator and hinge loss are used in the discriminator to refine the quality of the binary image reconstruction of the degraded document. The overall design is ultra-lightweight and can adaptively process degraded document images that have faded, uneven lighting, water damage, etc., showing high performance, wide applicability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of a method for binarizing a degraded document image based on adaptive convolution is provided in an embodiment of the present invention.

[0060] Figure 2 A schematic diagram of a specific process of a method for binarizing a degraded document image based on adaptive convolution provided in an embodiment of the present invention.

[0061] Figure 3 A schematic diagram of an adaptive convolution module provided in an embodiment of the present invention.

[0062] Figure 4A schematic diagram of an attention guiding module provided in an embodiment of the present invention.

[0063] Figure 5 A schematic diagram of a gated pyramid module provided in an embodiment of the present invention.

[0064] Figure 6 A schematic diagram of visualizing a binary image of a degraded document image provided by an embodiment of the present invention.

[0065] Figure 7 An architecture diagram of a degraded document image binarization system based on adaptive convolution provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Example 1

[0068] like Figure 1 As shown, a method for binarizing a degraded document image based on adaptive convolution is characterized by comprising:

[0069] S101: Collect training degraded document images, scan and annotate the training degraded document images to obtain binary label maps of the training degraded document images, pre-process the training degraded document images and the corresponding binary label maps to obtain pre-input images of the degraded document images and pre-input images of the binary label maps.

[0070] Specifically, the training degraded document images and the corresponding binarized label maps are divided into a training set and a test set at a ratio of 4:1. The degraded document images and the corresponding binarized label maps in the training set and the test set are cropped to obtain 256×256 image patches of the degraded document images and 256×256 image patches of the binarized label maps. The image patches of the degraded document images and the image patches of the binarized label maps are input into the edge detection algorithm and the large law algorithm, respectively, to obtain the pre-input images of the degraded document images and the pre-input images of the binarized label maps.

[0071] S102: inputting the pre-input image of the degraded document image into the adaptive convolutional network model to obtain a binary image of the degraded document image;

[0072] The encoder of the adaptive convolutional network model includes an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolutional network model includes a gated pyramid module.

[0073] S103: Calculating the adversarial loss according to the discrimination accuracy of the binary image of the degraded document image and the binary label map of the training degraded document image;

[0074] The classification loss is calculated based on the difference between the foreground pixel and the background pixel of the binary image of the degraded document image and the binary label map of the training degraded document image;

[0075] The similarity loss and the Jaccard similarity coefficient loss are calculated based on the pixel similarity between the binary image of the degraded document image and the binary label image of the training degraded document image;

[0076] The mean absolute error is calculated based on the pixel difference between the binarized image of the degraded document image and the binarized label map of the training degraded document image.

[0077] S104: Calculate the total loss based on adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, optimize the adaptive convolutional network model parameters by minimizing the total loss, and obtain the optimal adaptive convolutional network model.

[0078] S105: Inputting the pre-input image of the target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image.

[0079] like Figure 2 As shown, the present invention first pre-processes the collected training degraded document images to facilitate the subsequent training of the adaptive convolutional network model. The encoder in the adaptive convolutional network model is composed of an adaptive convolution module and a guided attention module. The adaptive convolution module can perform adaptive feature learning for regions with different degradation forms and degrees, and accurately locate the degraded region according to the global guided attention module. The decoder is composed of a gated pyramid module, which can efficiently restore the local details of the image, improve the reconstruction quality, and facilitate the acquisition of high-quality binary images.

[0080] Example 2

[0081] Based on the above embodiments, Figure 3 As shown, the embodiment of the present invention provides a specific structure of an adaptive convolution module, which specifically includes:

[0082] To solve the multiple degradation problems in degraded document images, an adaptive convolution is designed for feature learning of different degraded regions. The adaptive convolution module is mainly divided into two mechanisms: local and combined global context information. The adaptive convolution module includes a modulated convolution block, a dynamic weight subnet, a gradient information extraction block, and a feature map extraction block. The modulated convolution block and the dynamic weight subnet are used as local information mechanisms for feature learning, and the feature map extraction block is used as a combined global context information mechanism for global feature learning. The modulated convolution block converts the degraded document image into a vector form F∈R C×H×W , after a 3×3 convolution, the convolution kernel weight is W p , the convolution kernel weight modulated by the center intensity information is used as the feature extraction for the next stage. The convolution kernel weight of the modulated convolution block is expressed by the following formula:

[0083]

[0084] Among them, W q is the modulated convolution kernel weight, R is the size of the convolution, W p is the convolution kernel weight, and : is a placeholder.

[0085] The output of the modulated convolution block is input to the gradient information extraction block, in a 3×3 area, the central element of which is denoted by N o,o , whose eight neighborhood sets are recorded as N i,j ∈R, perform sum operation on the two dimensions H and W, and obtain the modulated 1×1 convolution kernel to extract gradient information. Finally, combine the conventional convolution and the modulated convolution to perform subtraction operation, and the feature map with gradient information is recorded as F∈R C×H×W , the gradient information extraction block is expressed as follows:

[0086] F′=W p ⊙F-θ×W q ⊙F

[0087] Among them, F′ is the feature map with gradient information, F is the feature map after convolution, ⊙ is the element-by-element multiplication operation, and θ is the proportion of differential information.

[0088] θ is the output of the dynamic weight subnet. Considering the different degradation sensitivities of different degraded document images and different parts of document images, a dynamic weight subnet is built. Given an I∈R C×H×W As the network input, a set of depth-separable convolutions are used to extract features, and finally an additional 1×1 convolution is performed for feature activation, and finally the Sigmoid function is used to map it to the θ value. The process is expressed as follows:

[0089] θ=σ(RELUCon 1×1 (DW 3×3 (PW1×1 (I))))

[0090] Among them, θ is the proportion of differential information, σ is the Sigmoid function, I is the input feature map, DW 3×3 is a 3×3 depth convolution, PW 1×1 For 1×1 point convolution, RELUCon 1×1 is the ReLU function.

[0091] Finally, the feature map with gradient information output by the gradient information extraction block is input into the feature map extraction block, which uses a gating mechanism to further refine the feature value. First, F′ is divided into two parts, and finally the Sigmoid function is used to map the first part of the feature map into a weight value, and finally the second part is multiplied element by element, denoted as ⊙, and the final extracted feature map formula is as follows:

[0092]

[0093] Among them, O is the extracted feature map, is the activation function.

[0094] Example 3

[0095] Based on the above embodiments, Figure 4 As shown, the embodiment of the present invention provides a specific structure of an attention guiding module, which specifically includes:

[0096] The distribution of handwriting and background pixels in degraded document images is uneven, especially for handwriting on bones, tortoise shells, and damaged paper. By introducing the attention mechanism to guide the model to focus on the degraded area, it is helpful to prevent the model from overfitting and increase the robustness of the model. The guided attention module consists of three parts: spatial attention module, channel attention module, and pixel reordering module, as shown in the attached figure. Figure 2 As shown. The specific process is that in the spatial dimension, global average pooling is used With global maximum pooling And use 1×1 convolution C 1×1 The ReLu function is used to express more complex nonlinear transformations, and finally a 7×7 convolution ReLu is used to expand the receptive field to capture spatial features. In the channel dimension, global average pooling is used And use 3×3 depth convolution DW 3×3 Processed with ReLu function, and finally using 1×1 point convolution PW 1×1 Capture the complex features between channels. Finally, the spatial dimension convolution weight W s and channel dimension convolution weight W c Perform addition operation to obtain the mixed domain feature weight W catThe specific process is expressed by the following formula:

[0097]

[0098] W cat =W s +W c

[0099] Among them, W s is the spatial dimension convolution weight, W c is the channel dimension convolution weight, C 7×7 is a 7×7 convolution, ReLu is a ReLu function, C 1×1 is a 1×1 convolution, Extract feature maps for global average pooling, Extract feature maps for global maximum pooling, PW 1×1 is a 1×1 point convolution, DW 3×3 is a 5×5 depth convolution, W cat is the mixed domain feature weight, [] is the concatenation operation, and s is the spatial dimension for operation.

[0100] After capturing the spatial and channel connections of the feature map, pixel rearrangement is used to learn the attention weights at the pixel level. First, the residual connection idea is used to connect the original input feature map O with the feature map O×W processed by the mixed domain. cat Splicing; then, use Shuffle pixel rearrangement (denoted as S) to rearrange the spliced ​​channels in an alternating manner; then, use depthwise separable convolution for feature integration, and finally use the Sigmoid function to assign weights to each pixel to obtain the final feature weight W. The process is expressed by the following formula:

[0101] W=σ(DW 5×5 (PW 1×1 (S[O,W cat ×O])))

[0102] Among them, W is the final feature weight, σ is the Sigmoid function, DW 5×5 is a 5×5 depth convolution, PW 1×1 is a 1×1 point convolution, S is a pixel rearrangement operation, O is an extracted feature map, and W cat is the total convolution weight of spatial dimension and channel dimension.

[0103] Example 4

[0104] Based on the above embodiments, Figure 5 As shown, the embodiment of the present invention provides a specific structure of a gated pyramid module, which specifically includes:

[0105] The decoder is composed of a feature pyramid network to obtain high-quality visual restoration. For semantic segmentation and image reconstruction, which are pixel-critical visual tasks, the original interpolation upsampling must be improved.

[0106] like Figure 5 As shown in the figure, the high-level information passed down by the encoder is stored in ascending order as C1-C5, and gated deconvolution (GateDeconv) is used for feature extraction, and feature transfer fusion is performed in a cascade manner. Gated deconvolution uses linear interpolation and gated convolution fusion internally, bilinear interpolation expands the image resolution, and then gated deconvolution is used to further select valid pixels for decoding. The specific formula is:

[0107]

[0108] GDConv = gate(Q(I j C ))

[0109] Among them, GDConv is gated deconvolution, gate is the gate function, Q is the bilinear interpolation operation, + is the feature map addition fusion, j is the feature map level, C is the feature map to be fused, P is the fused feature map, I j C is the feature map passed down by downsampling, I j p It is the feature map after the fusion of features of different scales.

[0110] Example 5

[0111] Based on the above embodiment, the embodiment of the present invention provides a training process of an adaptive convolutional network model, which specifically includes:

[0112] The obtained image patches of the degraded document images in the training set, the pre-input images of the degraded document images, and the pre-input images of the binary label images are converted into vector form, and are concatenated in the channel dimension, and the corresponding label images are converted into vector form.

[0113] The spliced ​​image vector is subjected to data preprocessing such as random flipping, random cropping, color jittering, brightness adjustment, and contrast adjustment to increase data diversity. Finally, the spliced ​​image vector and the label image vector are input into the adaptive convolutional network model together for subsequent training.

[0114] The training script was designed using PyCharm and trained in the PyTorch deep learning framework. All experiments used tools and Python scripts in the PyTorch toolbox. The Adma optimizer was used for network training. To address the instability often encountered in GAN training, the learning rate of the generator was set to 3×10-5, while the learning rate of the discriminator was set to 1×10-6. The network was trained for 60 epochs in an environment equipped with RTX 4090.

[0115] The discriminator network is introduced during training, and the hinge loss is used to maximize the adversarial loss. The specific process is as follows: the input received by the discriminator is the generated image (binarized image of the degraded document image) and the real image (binarized label map of the training degraded document image). The discriminator obtains a 16×16 image patch through convolution, and the corresponding label is "true" or "false". Try to introduce the hinge loss function to guide the discriminator to train the discrimination ability. The specific objective function is defined as follows:

[0116]

[0117] Among them, L D is the hinge loss, is the expected value of the binary label map of the training degraded document image, D(t) is the discriminator, G(x) is the generator, is the expected value of the binarized image of the degraded document image.

[0118] The corresponding adversarial loss function is defined as follows, and the optimization direction is given:

[0119]

[0120] argmin G max D L adv (G,D)

[0121] Among them, L adv To combat losses.

[0122] The loss function is introduced into the generator to optimize the image binarization result. For the input image, I otsu ,I sobel ,I origin Spliced ​​into a feature map of shape R∈5×256×256. Different from the general image-to-image transmission task, the image binarization task is mainly to separate foreground pixels from background pixels. Therefore, the binary cross entropy classification loss (BCE) is used, and the BCE loss is defined as follows:

[0123]

[0124] in, is the classification loss, N is the total number of pixels, t i is the binary label map of the training degraded document image, f i is the binarized image of the degraded document image.

[0125] In the image binarization task, some input images have foreground pixels that only account for a very small part, and the similarity loss (Dice Loss) can well handle the situation where the foreground pixels and background pixels are extremely unevenly distributed. Dice Loss can take each pixel into account and calculate the similarity with the pixels of the real image. In addition, the Jaccard similarity coefficient loss (Jasscard) is introduced to deal with degraded document images with more pixel overlap, and the effect is significant. Similarity loss and Jaccard similarity coefficient loss are expressed by the following formula:

[0126]

[0127] in, is the similarity loss, is the Jaccard similarity coefficient loss, ∈ is a constant term.

[0128] While minimizing the pixel difference between the generated image and the target image, we try to introduce the mean absolute value error (L1 loss) to reduce the blurriness of the binary document image. L1 loss is often used in image deblurring tasks. The main causes of blur are edge and detail information. By optimizing L1 loss, the model is more likely to retain the main pixel values ​​and suppress irrelevant pixel values. L1 loss is defined as follows:

[0129]

[0130] in, is the mean absolute error.

[0131] The total loss is expressed as follows:

[0132]

[0133] Among them, L G is the total loss, λ bce is the classification loss weight, λ dice is the similarity loss weight, λ jaccard is the Jaccard similarity coefficient loss weight, λ l1 is the mean absolute error weight, λ adv To counter the loss weight.

[0134] After completing the training of the adaptive convolutional network model through the training set, the test set is used to test it to obtain the optimal adaptive convolutional network model.

[0135] Example 6

[0136] Based on the above embodiment, the embodiment of the present invention provides an adaptive convolutional network model evaluation process, which specifically includes:

[0137] To ensure the fairness of the results, the official tool was used for comprehensive evaluation. The evaluation metrics are as follows: F-measure (FM), which combines precision and recall to give a single performance score, pseudo-F-measure (p-FM), a modified version of F-measure for specific image binarization tasks, peak signal-to-noise ratio (PSNR), which measures the similarity between the original image and the processed image, and distance reciprocal distortion metric (DRD) to evaluate the visual distortion in binary images by considering the importance of each pixel error.

[0138] The present invention proposes a method for binarization of degraded document images based on adaptive convolution, which is composed of lightweight modules. Experiments further prove the advantages of the present invention over existing methods in terms of operating efficiency and model parameter quantity. The experimental comparison schemes are: iterative fine network (DeepOtsu), two-stage gated convolutional network (GDB), conditional generative adversarial network (cGANs), improved conditional generative adversarial network (DE-GAN), and two-stage color document image enhancement and binarization network (2StageGAN) based on generative adversarial neural network.

[0139] The specific experimental results are shown in Table 1. The total number of parameters of the network model proposed in this invention is 26.14M, which is an average of 65.25M less than other methods; the running time is very close to that of the ViT-based method (due to the calculation method of the Transformer architecture itself, the running time is generally faster).

[0140] Table 1 Comparison of total parameters and running time between the present invention and the existing method

[0141] Model Parameters(M) Inter(s) DeepOtsu 93 3956 cGANs 103 982 DEGAN 31 351 2StageGAN 117 1856 GDB 67 465 Ours 26.14 145

[0142] In order to verify the significant performance of the present invention in the degraded document image binarization task, a degraded document image binarization method based on adaptive convolution disclosed in the present invention is compared with the existing methods. The compared methods include: conditional generative adversarial networks (cGANs), improved conditional generative adversarial networks (DE-GANs), two-stage gated convolutional networks (GDB), and two-stage color document image enhancement and binarization networks (2StageGAN) based on generative adversarial neural networks. The data set of this experiment is a degraded document image binarization data set. The experimental results on this data set are highly convincing. The specific experimental results are shown in Table 2.

[0143] Table 2 Performance comparison between the present invention and the existing method

[0144] Model FM p-FM PSNR DRD DeepOtsu 69.67 65.91 15.34 10.12 cGANs 66.51 59.78 13.67 9.97 DEGAN 57.97 49.29 11.64 15.57 2StageGAN 76.12 71.48 16.70 5.54 GDB 71.59 65.93 15.08 7.68 Ours 79.63 77.45 18.36 4.65

[0145] The method proposed in the present invention obtains excellent results in the four image binarization task indicators of FM, p-FM, PSNR, and DRD (the red mark is the optimal value, and the blue mark is the suboptimal value). In the HDIBCO dataset, the average FM, p-FM, PSNR, and DRD of the present invention are 94.60, 95.05, 23.01, and 1.36, respectively. Especially in the HDIBCO2019 dataset, which has a high degree of degradation and extremely complex features, the present invention has achieved the best performance compared with other methods, with an average improvement of 30.82, 30.15, 6.91, and -7.212 in the four indicators of FM, p-FM, PSNR, and DRD (the lower the DRD, the better). The visualization results of the present invention and the existing methods on degraded document images are shown in the figure. Figure 6 shown.

[0146] In summary, the adaptive convolution network model proposed in the present invention combines adaptive convolution and attention mechanism in the feature extraction stage to realize the positioning and adaptive extraction of different degraded areas of degraded document images. The gated pyramid multi-scale fusion mechanism is introduced in the decoder to ensure high-quality image reconstruction. Specifically, adaptive convolution is first used to supplement local detail information with global semantic information and dynamically modulate the weight; secondly, a guided attention layer is designed to focus on the degraded area to prevent overfitting; finally, the gating mechanism and feature pyramid structure are combined to enhance the fusion of high-level and low-level semantic information. This method realizes high-quality binarization of various types of degraded images, and has strong robustness and efficient operation performance. The baseline experimental results show that the model parameter volume is 26.14M, which is 65.25M less than other methods on average; FM, p-FM, PSNR, and DRD are 79.63, 77.45, 18.36, and 4.65, respectively, with the best performance.

[0147] Example 7

[0148] Corresponding to the above method, such as Figure 7 As shown, a degraded document image binarization system based on adaptive convolution includes:

[0149] A preprocessing module is used to collect training degraded document images, scan and annotate the training degraded document images to obtain a binary label map of the training degraded document images, and preprocess the training degraded document images and the corresponding binary label maps to obtain a pre-input image of the degraded document images and a pre-input image of the binary label map;

[0150] An adaptive convolutional network model module, used for inputting a pre-input image of a degraded document image into the adaptive convolutional network model to obtain a binary image of the degraded document image;

[0151] The encoder of the adaptive convolutional network model includes an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolutional network model includes a gated pyramid module.

[0152] A loss module, used for calculating the adversarial loss according to the discrimination accuracy of the binarized image of the degraded document image and the binarized label map of the training degraded document image;

[0153] The classification loss is calculated based on the difference between the foreground pixel and the background pixel of the binary image of the degraded document image and the binary label map of the training degraded document image;

[0154] The similarity loss and the Jaccard similarity coefficient loss are calculated based on the pixel similarity between the binary image of the degraded document image and the binary label image of the training degraded document image;

[0155] The mean absolute error is calculated based on the pixel difference between the binarized image of the degraded document image and the binarized label image of the training degraded document image;

[0156] The training module is used to calculate the total loss based on adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, and optimize the adaptive convolutional network model parameters by minimizing the total loss to obtain the optimal adaptive convolutional network model;

[0157] The output module is used to input the pre-input image of the target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image.

[0158] It should be noted that the adaptive convolution-based degraded document image binarization system provided in an embodiment of the present invention is to implement the adaptive convolution-based degraded document image binarization method in the above embodiment. Its specific functions can be referred to the above method embodiments and will not be repeated here.

[0159] Example 8

[0160] Based on the above embodiments, an embodiment of the present invention provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, a degraded document image binarization method based on adaptive convolution as described in the above embodiments is implemented.

[0161] Preferably, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium comprising a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a degraded document image binarization method based on adaptive convolution as described in the above embodiment.

[0162] In summary, the adaptive convolutional network model proposed in the present invention introduces an adaptive convolution module and a guided attention module in the encoder stage. The adaptive convolution module combines local and global features to enhance the transmission of effective information and suppress invalid information. Secondly, the guided attention module marks the degraded area, reduces background interference, and improves the accuracy of feature extraction and the robustness of the model. In addition, the gated pyramid module structure is used to improve feature fusion and enhance the transmission of shallow and high-level semantic information. The present invention calculates the total loss through adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, and then trains the adaptive convolutional network model. When calculating the adversarial loss, multiple loss functions are introduced in the generator and the weight setting is optimized. The local discriminator and hinge loss are used in the discriminator to refine the quality of the binary image reconstruction of the degraded document. The overall design is ultra-lightweight and can adaptively process degraded document images such as fading, uneven lighting, and water damage, showing high performance, wide applicability and flexibility.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for binarizing degraded document images based on adaptive convolution, characterized in that: include: Step 1: Collect training degraded document images, scan and annotate the training degraded document images to obtain binary label maps of the training degraded document images, pre-process the training degraded document images and the corresponding binary label maps to obtain pre-input images of the degraded document images and pre-input images of the binary label maps; Step 2: inputting the pre-input image of the degraded document image into the adaptive convolutional network model to obtain a binary image of the degraded document image; The encoder of the adaptive convolutional network model includes an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolutional network model includes a gated pyramid module; Step 3: Calculate the adversarial loss based on the discrimination accuracy of the binary image of the degraded document image and the binary label map of the training degraded document image; The classification loss is calculated based on the difference between the foreground pixel and the background pixel of the binary image of the degraded document image and the binary label map of the training degraded document image; The similarity loss and the Jaccard similarity coefficient loss are calculated based on the pixel similarity between the binary image of the degraded document image and the binary label image of the training degraded document image; The mean absolute error is calculated based on the pixel difference between the binarized image of the degraded document image and the binarized label image of the training degraded document image; Step 4: Calculate the total loss based on adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, optimize the adaptive convolutional network model parameters by minimizing the total loss, and obtain the optimal adaptive convolutional network model; Step 5: Input the pre-input image of the target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image.

2. The method for binarizing degraded document images based on adaptive convolution according to claim 1, characterized in that: The preprocessing of the training degraded document image and the corresponding binary label image specifically includes: Divide the training degraded document images and the corresponding binary label images into a training set and a test set; The degraded document images and the corresponding binary label images in the training set and the test set are cropped to obtain image patches of the degraded document images and image patches of the binary label images; The image patch of the degraded document image and the image patch of the binary label map are respectively input into the edge detection algorithm and the large law algorithm to obtain the pre-input image of the degraded document image and the pre-input image of the binary label map.

3. The method for binarizing degraded document images based on adaptive convolution according to claim 1, characterized in that: The adaptive convolution module includes a modulated convolution block, a dynamic weight subnet, a gradient information extraction block and a feature map extraction block; The convolution kernel weight of the modulated convolution block is expressed by the following formula: Among them, W q is the modulated convolution kernel weight, R is the convolution area, W p is the convolution kernel weight, : is a placeholder; The dynamic weight subnet is expressed by the following formula: θ=σ(RELUCon 1×1 (DW 3×3 (PW 1×1 (I)))) Among them, θ is the proportion of differential information, σ is the Sigmoid function, I is the input feature map, DW 3×3 is a 3×3 depth convolution, PW 1×1 For 1×1 point convolution, RELUCon 1×1 is the ReLU function; The gradient information extraction block is expressed as follows: F′=W p ⊙F-θ×W q ⊙F Among them, F′ is the feature map with gradient information, F is the feature map after convolution, and ⊙ is the element-by-element multiplication operation; The feature map extraction block is expressed as follows: Among them, O is the extracted feature map, is the activation function.

4. The method for binarizing degraded document images based on adaptive convolution according to claim 1, characterized in that: The guided attention module is expressed as follows: W=σ(DW 5×5 (PW 1×1 (S[O,W cat ×O]))) Among them, W is the final feature weight, σ is the Sigmoid function, DW 5×5 is a 5×5 depth convolution, PW 1×1 is a 1×1 point convolution, S is a pixel rearrangement operation, O is an extracted feature map, and W cat is the mixed domain feature weight.

5. The method for binarizing degraded document images based on adaptive convolution according to claim 1, characterized in that: The gated pyramid module is expressed by the following formula: GDConv=gate(Q(I j C )) Among them, GDConv is gated deconvolution, gate is the gate function, Q is the bilinear interpolation operation, + is the feature map addition fusion, j is the feature map level, C is the feature map to be fused, P is the fused feature map, I j C is the feature map passed down by downsampling, I j p It is the feature map after the fusion of features of different scales.

6. The method for binarizing degraded document images based on adaptive convolution according to claim 1, characterized in that: The total loss is calculated based on adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, specifically including: Among them, L G is the total loss, L dav To combat losses, is the classification loss, is the similarity loss, is the Jaccard similarity coefficient loss, is the mean absolute error, λ bce is the classification loss weight, λ dice is the similarity loss weight, λ jaccard is the Jaccard similarity coefficient loss weight, λ l1 is the mean absolute error weight, λ adv To counter the loss weight.

7. A degraded document image binarization system based on adaptive convolution, characterized in that: include: A preprocessing module is used to collect training degraded document images, scan and annotate the training degraded document images to obtain a binary label map of the training degraded document images, and preprocess the training degraded document images and the corresponding binary label maps to obtain a pre-input image of the degraded document images and a pre-input image of the binary label map; An adaptive convolutional network model module, used for inputting a pre-input image of a degraded document image into the adaptive convolutional network model to obtain a binary image of the degraded document image; The encoder of the adaptive convolutional network model includes an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolutional network model includes a gated pyramid module. A loss module, used for calculating the adversarial loss according to the discrimination accuracy of the binarized image of the degraded document image and the binarized label map of the training degraded document image; The classification loss is calculated based on the difference between the foreground pixel and the background pixel of the binary image of the degraded document image and the binary label map of the training degraded document image; The similarity loss and the Jaccard similarity coefficient loss are calculated based on the pixel similarity between the binary image of the degraded document image and the binary label image of the training degraded document image; The mean absolute error is calculated based on the pixel difference between the binarized image of the degraded document image and the binarized label image of the training degraded document image; The training module is used to calculate the total loss based on adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss and mean absolute error, and optimize the adaptive convolutional network model parameters by minimizing the total loss to obtain the optimal adaptive convolutional network model; The output module is used to input the pre-input image of the target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image.

8. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a method for binarizing a degraded document image based on adaptive convolution as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The storage medium includes a stored computer program, wherein when the computer program is executed, the device where the storage medium is located is controlled to execute the method for binarizing degraded document images based on adaptive convolution as described in any one of claims 1 to 6.

Images