A method and system for binarization of degraded document images based on adaptive convolution
By using an adaptive convolutional network model, combined with adaptive convolution, guided attention, and gated pyramid modules, the problems of poor generalization and high complexity of deep learning methods in degraded document image binarization are solved, achieving high-quality and robust degraded document image binarization.
Patent Information
- Application Number
- CN202411493584.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-12-09
- Estimated Expiration
- 2044-10-24
AI Technical Summary
Existing deep learning methods suffer from poor generalization, high model complexity, and insufficient robustness when binarizing degraded document images, making it difficult to adapt to document images with various degradation forms.
An adaptive convolutional network model is adopted, which combines an adaptive convolution module, a guided attention module, and a gated pyramid module. The model parameters are optimized by training with adversarial loss, classification loss, similarity loss, Jaccard similarity coefficient loss, and mean absolute error, so as to achieve adaptive processing of different degradation forms.
It improves the binarization quality and efficiency of degraded document images, enhances the robustness and applicability of the model, and can effectively handle document images with various degradation forms such as fading, uneven lighting, and water damage.
Smart Images
Figure CN119444582B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, and in particular to a degradation document image binarization method and system based on adaptive convolution. BACKGROUND
[0002] Historical documents are an important part of Chinese history and culture. The images of these historical documents come from the public and have a high degree of personal style and writing habits, so there is irregularity in the ink and strokes, such as length, thickness, etc. In addition, the original manuscript of the degraded document is often affected by factors such as dampness, insect damage, and light, resulting in an increase in image noise, and the manuscript itself may have problems such as character sticking, misspelling, and seal covering.
[0003] With the advancement of digitalization, the binarization processing of degraded document images has become an important research topic. Traditional methods have limited effect in processing old, unevenly illuminated, or wrinkled complex degraded document images. The rise of deep learning technology provides a new way to solve such problems. Deep learning can effectively improve the quality and clarity of degraded document images while preserving the original information of the degraded document images through large-scale data training and complex model construction.
[0004] Recent researchers have applied deep learning methods to the binarization task of historical degraded document images and have made some progress, but there are still many limitations in processing degraded document image binarization. First, the existing network structure has limitations in processing these multiple degradation forms, mainly because the modeling standard in the feature extraction stage is relatively uniform, and the diversified degradation features are not fully considered, resulting in insufficient robustness of the model when facing different degradation forms and poor generalization. Second, the current mainstream neural network-based image binarization method usually relies on a multi-level architecture and additional input. The high model complexity makes the model too dependent on the inherent data patterns learned, and the model training is too complex, thereby limiting its adaptability to multiple degraded degraded document images and affecting the subsequent research on degraded document images. SUMMARY
[0005] In order to at least partially solve the problems of poor generalization, complex training, and limited performance of deep learning-based image binarization methods in degraded document images, the present application provides a degradation document image binarization method and system based on adaptive convolution, which constructs an adaptive convolution network model. The encoder of the adaptive convolution network model includes an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolution network model includes a gated pyramid module, so that the adaptive convolution network model can be applied to various forms of degraded degraded document images, ensuring the high quality and efficiency of the binarized image.
[0006] In order to achieve the above-mentioned purpose, the technical scheme of the present application is:
[0007] The first aspect of the present application provides a binarization method for degraded document images based on adaptive convolution, comprising:
[0008] Step one: collect degraded document images, scan and label the degraded document images to obtain a binarization label image of the degraded document images, and pre-process the degraded document images and the corresponding binarization label image to obtain a pre-input image of the degraded document images and a pre-input image of the binarization label image, so as to facilitate the training of the adaptive convolution network model;
[0009] Step two: input the pre-input image of the degraded document images into the adaptive convolution network model to obtain a binarization image of the degraded document images, so as to facilitate the training of the adaptive convolution network model;
[0010] The encoder of the adaptive convolution network model comprises an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolution network model comprises a gated pyramid module, so as to improve the accuracy of the output binarization image;
[0011] Step three: calculate the adversarial loss according to the discriminant accuracy of the binarization image of the degraded document images and the binarization label image of the degraded document images, which is used for training the adaptive convolution network model;
[0012] Calculate the classification loss according to the foreground pixel and background pixel difference of the binarization image of the degraded document images and the binarization label image of the degraded document images, which is used for training the adaptive convolution network model;
[0013] Calculate the similarity loss and the Jaccard similarity coefficient loss according to the pixel point similarity of the binarization image of the degraded document images and the binarization label image of the degraded document images, which are used for training the adaptive convolution network model;
[0014] Calculate the mean absolute error according to the pixel difference of the binarization image of the degraded document images and the binarization label image of the degraded document images, which is used for training the adaptive convolution network model;
[0015] Step four: calculate the total loss based on the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, optimize the adaptive convolution network model parameters by minimizing the total loss, and obtain the optimal adaptive convolution network model;
[0016] Step five: input the pre-input image of the target degraded document image into the optimal adaptive convolution network model to obtain the binarization image of the target degraded document image.
[0017] Further, the pre-processing of the degraded document images and the corresponding binarization label image specifically comprises:
[0018] The degraded document image and the corresponding binary label image are divided into a training set and a test set to facilitate training of the adaptive convolutional network model;
[0019] The degraded document images and the corresponding binary label images in the training set and the test set are cropped to obtain image patches of the degraded document images and image patches of the binary label images, so as to facilitate processing of the adaptive convolutional network model and avoid long processing time caused by excessively large images;
[0020] The image patches of the degraded document images and the image patches of the binary label images are input into an edge detection algorithm and a law algorithm to obtain pre-input images of the degraded document images and pre-input images of the binary label images.
[0021] Further, the adaptive convolutional module comprises a modulation convolution block, a dynamic weight subnetwork, a gradient information extraction block and a feature map extraction block;
[0022] The convolution kernel weight of the modulation convolution block is represented by the following formula:
[0023]
[0024] wherein, W q is the modulated convolution kernel weight, R is the size of convolution, W p is the convolution kernel weight, and : is a placeholder;
[0025] The dynamic weight subnetwork is represented by the following formula:
[0026] θ=σ(RELUCon 1×1 (DW 3×3 (PW 1×1 (I))))
[0027] wherein, θ is the size of the proportion of differential information, σ is a Sigmoid function, I is an input feature map, DW 3×3 is a 3×3 deep convolution, PW 1×1 is a 1×1 point convolution, and RELUCon 1×1 is a ReLU function;
[0028] The gradient information extraction block is represented by the following formula:
[0029] F′=W p ⊙F-θ×W q ⊙F
[0030] wherein, F′ is a feature map with gradient information, F is a feature map after convolution, and ⊙ is an element-wise multiplication operation;
[0031] The feature map extraction block is represented by the following formula:
[0032]
[0033] wherein, O is an extracted feature map, is an activation function.
[0034] Further, the guided attention module is expressed by the following formula:
[0035] W = σ(DW 5×5 (PW 1×1 (S[O,W cat ×O])))
[0036] wherein, W is a final feature weight, σ is a Sigmoid function, DW 5×5 is a 5x5 depth convolution, PW 1×1 is a 1x1 point convolution, S is a pixel rearrangement operation, O is an extracted feature map, W cat is a mixed domain feature weight.
[0037] Further, the gating pyramid module is expressed by the following formula:
[0038]
[0039] wherein, GDConv is a gated deconvolution, gate is a gate function, Q is a bilinear interpolation operation, + is a feature map addition fusion, j is a feature map level, C is a to-be-fused feature map, P is a fused feature map, I j C is a feature map transmitted by down-sampling, I j p is a feature map fused by different scale features.
[0040] Further, the total loss is calculated based on the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, and specifically includes:
[0041]
[0042] wherein, L G is a total loss, L dav is an adversarial loss, is a classification loss, is a similarity loss, is a Jaccard similarity coefficient loss, is a mean absolute error, λ bce is a classification loss weight, λ dice is a similarity loss weight, λ jaccard is a Jaccard similarity coefficient loss weight, λ l1 is a mean absolute error weight, λ advThe adversarial loss weight.
[0043] The second aspect of the present application provides a degraded document image binarization system based on adaptive convolution, comprising:
[0044] A preprocessing module is configured to collect degraded document images, scan and label the degraded document images to obtain a binarization label image of the degraded document images, and preprocess the degraded document images and the corresponding binarization label image to obtain a pre-input image of the degraded document images and a pre-input image of the binarization label image, thereby facilitating training of the adaptive convolution network model.
[0045] An adaptive convolution network model module is configured to input the pre-input image of the degraded document images into the adaptive convolution network model to obtain a binarization image of the degraded document images, thereby facilitating training of the adaptive convolution network model.
[0046] The encoder of the adaptive convolution network model comprises an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolution network model comprises a gated pyramid module, thereby facilitating improvement of the accuracy of the output binarization image.
[0047] A loss module is configured to calculate an adversarial loss according to the discriminant accuracy of the binarization image of the degraded document images and the binarization label image of the degraded document images, thereby facilitating training of the adaptive convolution network model.
[0048] A classification loss is calculated according to the difference between the foreground pixels and the background pixels of the binarization image of the degraded document images and the binarization label image of the degraded document images, thereby facilitating training of the adaptive convolution network model.
[0049] A similarity loss and a Jaccard similarity coefficient loss are calculated according to the pixel similarity of the binarization image of the degraded document images and the binarization label image of the degraded document images, thereby facilitating training of the adaptive convolution network model.
[0050] A mean absolute error is calculated according to the pixel difference of the binarization image of the degraded document images and the binarization label image of the degraded document images, thereby facilitating training of the adaptive convolution network model.
[0051] A training module is configured to calculate a total loss based on the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, to optimize the adaptive convolution network model parameters by minimizing the total loss, and to obtain an optimal adaptive convolution network model.
[0052] An output module is configured to input a pre-input image of a target degraded document image into the optimal adaptive convolution network model to obtain a binarization image of the target degraded document image.
[0053] The third aspect of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the adaptive convolution-based degraded document image binarization method according to the first aspect when executing the computer program.
[0054] The fourth aspect of the present application provides a computer-readable storage medium, comprising a stored computer program, wherein the computer program controls the device where the storage medium is located to execute the adaptive convolution-based degraded document image binarization method according to the first aspect when running.
[0055] Advantages of the present application:
[0056] (1) The adaptive convolution network model proposed by the present application introduces an adaptive convolution module and a guided attention module in the encoder stage. The adaptive convolution module combines local and global features, enhances effective information transmission and suppresses invalid information. Secondly, the guided attention module marks the degraded area, reduces background interference, and improves feature extraction accuracy and model robustness. In addition, the feature fusion is improved through the structure of the gated pyramid module, and the transmission of shallow and high-level semantic information is enhanced.
[0057] (2) The present application calculates the total loss by the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, and then trains the adaptive convolution network model. When calculating the adversarial loss, multiple loss functions are introduced in the generator and the weight setting is optimized, and the local discriminator and the hinge loss are used in the discriminator, which refines the reconstruction quality of the degraded document binarization image. The overall design is ultra-light, which can adaptively process degraded document images degraded by fading, uneven illumination, water damage, etc., showing high performance, wide applicability and flexibility. BRIEF DESCRIPTION OF DRAWINGS
[0058] Figure 1 A flowchart of the adaptive convolution-based degraded document image binarization method provided by the embodiment of the present application.
[0059] Figure 2 A schematic diagram of the specific process of the adaptive convolution-based degraded document image binarization method provided by the embodiment of the present application.
[0060] Figure 3 A schematic diagram of the adaptive convolution module provided by the embodiment of the present application.
[0061] Figure 4 A schematic diagram of the guided attention module provided by the embodiment of the present application.
[0062] Figure 5A schematic diagram of the gating pyramid module provided for the embodiment of the present application.
[0063] Figure 6 A schematic diagram of the visualization of the binarization image of the degraded document image provided for the embodiment of the present application.
[0064] Figure 7 An architecture diagram of a degraded document image binarization system based on adaptive convolution provided for the embodiment of the present application. DETAILED DESCRIPTION
[0065] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0066] Embodiment 1
[0067] As shown in Figure 1 A degraded document image binarization method based on adaptive convolution, characterized in that, comprising:
[0068] S101: Collecting training degraded document images, obtaining the binarization label image of the training degraded document images by scanning and labeling, and pre-processing the training degraded document images and the corresponding binarization label images to obtain the pre-input image of the degraded document image and the pre-input image of the binarization label image.
[0069] Specifically, the training degraded document images and the corresponding binarization label images are divided into a training set and a test set according to a ratio of 4:1. The degraded document images and the corresponding binarization label images in the training set and the test set are subjected to cropping processing to obtain 256x256 image patches of the degraded document images and 256x256 image patches of the binarization label images. The image patches of the degraded document images and the image patches of the binarization label images are respectively input into an edge detection algorithm and a grand law algorithm to obtain the pre-input image of the degraded document image and the pre-input image of the binarization label image.
[0070] S102: Inputting the pre-input image of the degraded document image into an adaptive convolution network model to obtain a binarization image of the degraded document image;
[0071] The encoder of the adaptive convolution network model comprises an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolution network model comprises a gating pyramid module.
[0072] S103: Obtain an adversarial loss according to the binary image of the degraded document image and the discriminant accuracy of the binary label image of the training degraded document image;
[0073] Obtain a classification loss according to the foreground pixel and background pixel difference of the binary image of the degraded document image and the binary label image of the training degraded document image;
[0074] Obtain a similarity loss and a Jaccard similarity coefficient loss according to the pixel point similarity of the binary image of the degraded document image and the binary label image of the training degraded document image;
[0075] Obtain a mean absolute error according to the pixel difference of the binary image of the degraded document image and the binary label image of the training degraded document image.
[0076] S104: Calculate a total loss based on the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, optimize the adaptive convolutional network model parameters by minimizing the total loss, and obtain an optimal adaptive convolutional network model.
[0077] S105: Input the pre-input image of the target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image.
[0078] As shown in Figure 2 , the present application first pre-processes the collected training degraded document image, which facilitates the subsequent training of the adaptive convolutional network model. The encoder in the adaptive convolutional network model is composed of an adaptive convolution module and a guided attention module. The adaptive convolution module can perform adaptive feature learning for different degradation forms and different degrees of regions, and accurately locate the degraded regions according to the global guided attention module. The decoder is composed of a gated pyramid module, which can efficiently restore the local details of the image and improve the reconstruction quality, so as to obtain a high-quality binary image.
[0079] Embodiment 2
[0080] On the basis of the above-mentioned embodiments, as shown in Figure 3 , the present application provides a specific structure of an adaptive convolution module, which specifically comprises:
[0081] To solve the problem of multiple degradation in degraded document images, adaptive convolution is designed for feature learning in different degradation areas. The adaptive convolution module mainly includes two mechanisms of local and combined global context information. The adaptive convolution module includes modulation convolution block, dynamic weight subnet, gradient information extraction block and feature map extraction block. The modulation convolution block and dynamic weight subnet are used as local information mechanism for feature learning, and the feature map extraction block is used as combined global context information mechanism for global feature learning. The modulation convolution block converts the degraded document image into a vector form F∈R C×H×W , and after a 3×3 convolution, the convolution kernel weight is W p , and the center intensity information modulated convolution kernel weight is used as the next stage feature extraction. The convolution kernel weight of the modulation convolution block is expressed by the following formula:
[0082]
[0083] , where W q is the modulated convolution kernel weight, R is the size of convolution, W p is the convolution kernel weight, and : is a placeholder.
[0084] The output of the modulation convolution block is input to the gradient information extraction block. In a 3×3 area, the center element is denoted as N o,o , and its eight neighborhood set is denoted as N i,j ∈R. Summation operation is performed in H and W dimensions to obtain the modulated 1×1 convolution kernel for gradient information extraction. Finally, the difference operation is performed between the regular convolution and the modulated convolution, and the feature map with gradient information is obtained, denoted as F∈R C×H×W . The gradient information extraction block is expressed by the following formula:
[0085] F′=W p ⊙F-θ×W q ⊙F
[0086] , where F′ is the feature map with gradient information, F is the feature map after convolution, ⊙ is the element-wise multiplication operation, and θ is the proportion of difference information.
[0087] θ is the output of the dynamic weight subnet. Considering that different degraded document images and different parts of the document image have different degradation sensitivities, a dynamic weight subnet is built. Given I∈R C×H×W as the network input, a set of deep separable convolution is used to extract features, and then an additional 1×1 convolution is used for feature activation. Finally, the Sigmoid function is used to map θ value, which is expressed by the following formula:
[0088] θ=σ(RELUCon 1×1 (DW 3×3 (PW1×1 (I))))
[0089] wherein, θ is the proportion of differential information, σ is the Sigmoid function, I is the input feature map, DW 3×3 is 3x3 deep convolution, PW 1×1 is 1x1 point convolution, RELUCon 1×1 is the ReLU function.
[0090] The feature map with gradient information output by the last gradient information extraction block is input into the feature map extraction block, and the feature map extraction block adopts a gating mechanism to further refine the feature values. First, F' is divided into two parts, and the first part of the feature map is finally mapped into a weight value using a Sigmoid function, and then multiplied element by element with the second part, denoted as, to obtain the final extracted feature map, and the formula is as follows:
[0091]
[0092] wherein, O is the extracted feature map, is the activation function.
[0093] Embodiment 3
[0094] On the basis of the above embodiment, as Figure 4 indicated, the embodiment of the present application provides a specific structure of the guided attention module, which specifically comprises:
[0095] The distribution of handwriting and background pixels in the degraded document image is uneven, especially some handwriting appearing on bones, shells, and damaged paper. By introducing the attention mechanism to guide the model to focus on the degraded area, it is beneficial to prevent model overfitting and increase model robustness. The guided attention module is composed of three parts: a spatial attention module, a channel attention module, and a pixel rearrangement module, as shown in the accompanying Figure 2 As shown in the accompanying and global maximum pooling are used, and 1x1 convolution C 1×1 and ReLu function are used to express more complex nonlinear transformations, and finally 7x7 convolution ReLu is used to expand the receptive field to capture spatial features. In the channel dimension, global average pooling is used, and 3x3 deep convolution DW 3×3 and ReLu function are used for processing, and finally 1x1 point convolution PW 1×1 is used to capture complex features between channels. Finally, the spatial dimension convolution weight W s and the channel dimension convolution weight W c are added to obtain the mixed domain feature weight W cat, the specific process is represented by the following formula:
[0096]
[0097] W cat =W s +W c
[0098] Wherein, W s is a spatial dimension convolution weight, W c is a channel dimension convolution weight, C 7×7 is a 7x7 convolution, ReLu is a ReLu function, C 1×1 is a 1x1 convolution, is a global average pooling feature map, is a global maximum pooling feature map, PW 1×1 is a 1x1 point convolution, DW 3×3 is a 5x5 depth convolution, W cat is a mixed domain feature weight, [] is a splicing operation, and s is a spatial dimension operation.
[0099] After capturing the spatial connection and channel connection of the feature map, pixel rearrangement is used to learn the attention weight at the pixel level. First, the residual connection idea is used to splice the original input feature map O and the feature map OxW cat after mixed domain processing; then, the spliced channel is rearranged in an alternating manner using Shuffle pixel rearrangement (denoted as S); then, feature integration is performed using depth separable convolution; finally, a Sigmoid function is used to assign weights to each pixel point to obtain the final feature weight W, which is represented by the following formula:
[0100] W=σ(DW 5×5 (PW 1×1 (S[O,W cat ×O])))
[0101] Wherein, W is the final feature weight, σ is the Sigmoid function, DW 5×5 is a 5x5 depth convolution, PW 1×1 is a 1x1 point convolution, S is a pixel rearrangement operation, O is an extracted feature map, and W cat is the total weight of spatial dimension and channel dimension convolution.
[0102] Embodiment 4
[0103] On the basis of the above embodiment, as shown in Figure 5 , the embodiment of the application provides a specific structure of a gated pyramid module, which specifically comprises:
[0104] The decoder is composed of a feature pyramid network to obtain high-quality visual recovery. For visual tasks such as semantic segmentation and image reconstruction, which require strict pixel requirements, the original interpolation method must be improved for upsampling.
[0105] As shown in Figure 5 For the high-level information delivered by the encoder downsampling, the level is increased to store C1-C5, and the gate deconvolution (GateDeconv) is used for feature extraction, and the feature transmission and fusion are performed according to the cascading mode. The gate deconvolution uses linear interpolation and gate convolution fusion inside, the bilinear interpolation expands the image resolution, and the gate deconvolution is further used to select effective pixels for decoding. The specific formula is as follows:
[0106]
[0107] GDConv=gate(Q(I j C ))
[0108] Wherein, GDConv is the gate deconvolution, gate is the gate function, Q is the bilinear interpolation operation, + is the feature map addition fusion, j is the feature map level, C is the to-be-fused feature map, P is the fused feature map, I j C is the feature map delivered by downsampling, and I j p is the feature map fused by different scale features.
[0109] Embodiment 5
[0110] On the basis of the above-mentioned embodiments, the training process of the adaptive convolutional network model is provided, and specifically includes:
[0111] The obtained image patches of the degraded document image, the pre-input images of the degraded document image and the pre-input images of the binary label image in the training set are converted into vector form, and are spliced in the channel dimension, and the label image corresponding thereto is converted into a vector form.
[0112] The spliced image vector is executed random flip, random cropping, color jitter, brightness adjustment, contrast adjustment data preprocessing to increase data diversity. Finally, the spliced image vector and the label image vector are jointly input into the adaptive convolutional network model, which is convenient for subsequent training.
[0113] The training script is programmed using PyCharm, and the training is performed in the PyTorch deep learning framework. All experiments use tools in the PyTorch toolbox and Python scripts. The Adma optimizer is used for network training. In order to solve the instability often encountered in GAN training, the learning rate of the generator is set to 3x10 -5 , and the learning rate of the discriminator is set to 1x10 -6 . The network is trained for 60 rounds in an environment equipped with an RTX 4090.
[0114] The discriminator network is introduced during training, and the hinge loss is used to maximize the adversarial loss. The specific process is as follows: the input received by the discriminator is the generated image (the binary image of the degraded document image) and the real image (the binary label image of the training degraded document image). The discriminator obtains 16x16 image patches through convolution, and the corresponding label is "true" or "false". The hinge loss function is introduced to guide the discriminant training ability. The specific objective function is defined as follows:
[0115]
[0116] where L D is the hinge loss, is the expected value of the binary label image of the training degraded document image, D(t) is the discriminator, G(x) is the generator, is the expected value of the binary image of the degraded document image.
[0117] The corresponding adversarial loss function is defined as follows, and the optimization direction is given:
[0118]
[0119] argmin G max D L adv (G,D)
[0120] where L adv is the adversarial loss.
[0121] The loss function is introduced in the generator to optimize the image binarization result. For a given input image, I otsu , I sobel , I origin are spliced into a feature map with a shape of R∈5x256x256. Unlike general image-to-image transmission tasks, the image binarization task mainly separates foreground pixels from background pixels. Therefore, the binary cross-entropy classification loss (BCE) is used, and the BCE loss is defined as follows:
[0122]
[0123] wherein, is a classification loss, N is the total number of pixels, t i is a binary label map of the training degraded document image, f i is a binary image of the degraded document image.
[0124] In the image binarization task, part of the input image has only a very small proportion of foreground pixels, and the similarity loss (Dice Loss) can well handle the case where the foreground and background pixels are extremely unevenly distributed. Dice Loss can take into account each pixel and perform similarity calculation with the pixel points of the real image. In addition, the Jasscard loss is introduced to handle the degraded document image with more pixel overlap, and the effect is significant. The similarity loss and the Jasscard loss are expressed by the following formula:
[0125]
[0126] wherein, is a similarity loss, is a Jasscard loss, and ∈ is a constant term.
[0127] When minimizing the pixel difference between the generated image and the target image, the mean absolute error (L1 loss) is introduced to reduce the ambiguity of the binary document image. L1 loss is commonly used in image deblurring tasks, and the main reason for blurring is the edge and detail information. By optimizing the L1 loss, the model is more likely to retain the main pixel value and suppress irrelevant pixel values. The L1 loss is defined as follows:
[0128]
[0129] wherein, is a mean absolute error.
[0130] The total loss is expressed by the following formula:
[0131]
[0132] wherein, L G is a total loss, λ bce is a classification loss weight, λ dice is a similarity loss weight, λ jaccard is a Jasscard loss weight, λ l1 is a mean absolute error weight, λ adv is an adversarial loss weight.
[0133] After training the adaptive convolutional network model using the training set, the test set is used for testing to obtain the optimal adaptive convolutional network model.
[0134] Example 6
[0135] On the basis of the above-mentioned embodiments, the embodiment of the present application provides a self-adaptive convolutional network model evaluation process, which specifically comprises:
[0136] In order to ensure the fairness of the results, the official tool is used for comprehensive evaluation. The evaluation indexes are as follows: F-measure (FM), which combines precision and recall to give a single performance score, pseudo F-measure (p-FM), a modified version of F-measure for specific image binarization tasks, peak signal-to-noise ratio (PSNR), which measures the similarity between the original image and the processed image, and distance reciprocal distortion metric (DRD), which evaluates the visual distortion in the binary image by considering the importance of each pixel error.
[0137] The self-adaptive convolution-based degraded document image binarization method proposed in the present application is composed of lightweight modules, and experiments further prove the advantages of the present application over existing methods in terms of running efficiency and model parameter quantity. The comparison schemes of the experiment are: iterative fine network (DeepOtsu), two-stage gated convolutional network (GDB), conditional generative adversarial network (cGANs), improved conditional generative adversarial network (DE-GAN), and two-stage color document image enhancement and binarization network based on generative adversarial neural network (2StageGAN).
[0138] The specific experimental results are shown in Table 1. The total parameter quantity of the network model proposed in the present application is 26.14M, which is averagely reduced by 65.25M compared with other methods; and the running time is very small compared with the method based on ViT (due to the calculation method of the Transformer architecture itself, the running time is generally faster).
[0139] Table 1 Comparison of total parameter quantity and running time of the present application and existing methods
[0140] Model Parameters (M) Inter(s) DeepOtsu 93 3956 cGANs 103 982 DEGAN 31 351 2StageGAN 117 1856 GDB 67 465 Ours 26.14 145
[0141] In order to verify the significant performance of the present application in the degraded document image binarization task, the self-adaptive convolution-based degraded document image binarization method disclosed in the present application is compared with existing methods. The comparison methods include: conditional generative adversarial network (cGANs), improved conditional generative adversarial network (DE-GAN), two-stage gated convolutional network (GDB), and two-stage color document image enhancement and binarization network based on generative adversarial neural network (2StageGAN). The data set of this experiment is the degraded document image binarization data set, and the experimental results on this data set are more convincing. The specific experimental results are shown in Table 2.
[0142] Table 2 Performance comparison of the present application and existing methods
[0143] Model FM p-FM PSNR DRD DeepOtsu 69.67 65.91 15.34 10.12 cGANs 66.51 59.78 13.67 9.97 DEGAN 57.97 49.29 11.64 15.57 2StageGAN 76.12 71.48 16.70 5.54 GDB 71.59 65.93 15.08 7.68 Ours 79.63 77.45 18.36 4.65
[0144] The method proposed in the present application obtains very excellent results on the four image binarization task indicators of FM, p-FM, PSNR and DRD (red marks the optimal value, and blue marks the suboptimal value). On the HDIBCO dataset, the average FM, p-FM, PSNR and DRD of the present application are 94.60, 95.05, 23.01 and 1.36, respectively. Especially on the HDIBCO2019 dataset, which has a high degree of degradation and extremely complex features, the present application has the best performance compared with other methods, with an average improvement of 30.82, 30.15, 6.91 and -7.212 (DRD is lower the better) in the four indicators of FM, p-FM, PSNR and DRD. The visualization results of the present application and existing methods on the degraded document image are shown in Figure 6
[0145] In summary, the adaptive convolution network model proposed in the present application combines adaptive convolution and attention mechanism in the feature extraction stage to realize the positioning and adaptive extraction of different degraded regions of the degraded document image. The gated pyramid multi-scale fusion mechanism is introduced in the decoder to ensure high-quality image reconstruction. Specifically, first, adaptive convolution is used to supplement local detail information with global semantic information and dynamically modulate the proportion; second, a guided attention layer is designed to focus on the degraded region to prevent overfitting; and finally, the gated mechanism and feature pyramid structure are combined to enhance the fusion of high-level and low-level semantic information. This method realizes high-quality binarization of multiple types of degraded images, has strong robustness and efficient running performance. The baseline experimental results show that the model parameter amount is 26.14M, which is reduced by an average of 65.25M compared with other methods; the FM, p-FM, PSNR and DRD are 79.63, 77.45, 18.36 and 4.65, respectively, and the performance is optimal.
[0146] Example 7
[0147] Corresponding to the above method, as shown in Figure 7 , a degraded document image binarization system based on adaptive convolution includes:
[0148] A preprocessing module is configured to collect training degraded document images, scan and label the training degraded document images to obtain binarization label images of the training degraded document images, and preprocess the training degraded document images and the corresponding binarization label images to obtain pre-input images of the degraded document images and pre-input images of the binarization label images.
[0149] An adaptive convolutional network model module is configured to input a pre-input image of a degraded document image into an adaptive convolutional network model to obtain a binarization image of the degraded document image.
[0150] The encoder of the adaptive convolutional network model comprises an adaptive convolutional module and a guided attention module, and the decoder of the adaptive convolutional network model comprises a gated pyramid module.
[0151] A loss module is configured to calculate an adversarial loss according to a discrimination accuracy of the binarization image of the degraded document image and a binarization label image of a training degraded document image.
[0152] A classification loss is calculated according to a foreground pixel and background pixel difference of the binarization image of the degraded document image and the binarization label image of the training degraded document image.
[0153] A similarity loss and a Jaccard similarity coefficient loss are calculated according to a pixel point similarity of the binarization image of the degraded document image and the binarization label image of the training degraded document image.
[0154] A mean absolute error is calculated according to a pixel difference of the binarization image of the degraded document image and the binarization label image of the training degraded document image.
[0155] A training module is configured to calculate a total loss based on the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, optimize adaptive convolutional network model parameters by minimizing the total loss, and obtain an optimal adaptive convolutional network model.
[0156] An output module is configured to input a pre-input image of a target degraded document image into the optimal adaptive convolutional network model to obtain a binarization image of the target degraded document image.
[0157] It should be noted that the adaptive convolution-based degraded document image binarization system provided in the embodiments is to realize the adaptive convolution-based degraded document image binarization method in the above embodiments, and the functions can be referred to the above method embodiments, which will not be described here.
[0158] Embodiment 8
[0159] On the basis of the above embodiments, an electronic device is provided, which comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the adaptive convolution-based degraded document image binarization method as described in the above embodiments when executing the computer program.
[0160] Preferably, the embodiment of the present application further provides a computer readable storage medium, which comprises a stored computer program, wherein the computer program controls a device where the storage medium is located to execute the adaptive convolution based binarization method of degraded document image as described in the above embodiment when the computer program is running.
[0161] To sum up, the adaptive convolution network model proposed in the present application introduces an adaptive convolution module and a guided attention module in the encoder stage. The adaptive convolution module combines local and global features, enhances effective information transmission and suppresses invalid information. Secondly, the guided attention module marks the degraded area, reduces background interference, and improves feature extraction accuracy and model robustness. In addition, the feature fusion is improved through the structure of the gated pyramid module, enhancing the transmission of shallow and high-level semantic information. The present application calculates the total loss by using the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, and then trains the adaptive convolution network model. When calculating the adversarial loss, multiple loss functions are introduced in the generator and the weight settings are optimized, and the local discriminator and the hinge loss are used in the discriminator, which refines the reconstruction quality of the degraded document binarization image. The overall design is ultra-light, which can adaptively process faded, unevenly illuminated, water-damaged and other degraded document images, showing high performance, wide applicability and flexibility.
[0162] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method of binarization of degraded document images based on adaptive convolution, characterized in that, The method comprises the following steps: Step 1: Collecting training degraded document images, performing scanning labeling on the training degraded document images to obtain binary label images of the training degraded document images, and pre-processing the training degraded document images and the corresponding binary label images to obtain pre-input images of the degraded document images and pre-input images of the binary label images; Step 2: inputting the pre-input images of the degraded document images into an adaptive convolutional network model to obtain binary images of the degraded document images; The encoder of the adaptive convolutional network model comprises an adaptive convolutional module and a guided attention module, and the decoder of the adaptive convolutional network model comprises a gated pyramid module; Step 3: calculating an adversarial loss according to the discrimination accuracy of the binary images of the degraded document images and the binary label images of the training degraded document images; calculating a classification loss according to the foreground pixel and background pixel difference of the binary images of the degraded document images and the binary label images of the training degraded document images; calculating a similarity loss and a Jaccard similarity coefficient loss according to the pixel point similarity of the binary images of the degraded document images and the binary label images of the training degraded document images; calculating a mean absolute error according to the pixel difference of the binary images of the degraded document images and the binary label images of the training degraded document images; Step 4: calculating a total loss based on the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, optimizing the adaptive convolutional network model parameters by minimizing the total loss to obtain an optimal adaptive convolutional network model; Step 5: inputting pre-input images of a target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image; The adaptive convolutional module comprises a modulation convolutional block, a dynamic weight subnetwork, a gradient information extraction block and a feature map extraction block; The convolution kernel weight of the modulation convolutional block is expressed by the following formula: wherein W q is the modulated convolution kernel weight, R is the area size of the convolution, W p is the convolution kernel weight, and : is a placeholder; The dynamic weight subnetwork is expressed by the following formula: Θ = σ(RELUCon 1×1 (DW 3×3 (PW 1×1 (I)))) wherein θ is the proportion of the differential information, σ is a Sigmoid function, I is an input feature map, DW 3×3 is a 3x3 depthwise convolution, PW 1×1 is a 1x1 pointwise convolution, RELUCon 1×1 is a ReLU function; The gradient information extraction block is expressed by the following formula: F' = W p F - θ x W q F Wherein, F' is a feature map with gradient information, F is a feature map after convolution, and is an element-wise multiplication operation; The feature map extraction block is expressed by the following formula: wherein O is an extracted feature map, is an activation function. The guided attention module is expressed by the following formula: W = σ(DW 5×5 (PW 1×1 (S[O,W cat ×O]))) wherein, W is the final feature weight, σ is the Sigmoid function, DW 5×5 is a 5x5 depthwise convolution, PW 1×1 is a 1x1 pointwise convolution, S is a pixel rearrangement operation, O is the extracted feature map, W cat is the mixed domain feature weight.
2. The method of claim 1, wherein the method is based on an adaptive convolution for binarization of degraded document images. The pre-processing of the training degraded document images and the corresponding binary label images comprises the following steps: dividing the training degraded document images and the corresponding binary label images into a training set and a test set; performing cropping processing on the training set and the test set to obtain image patches of the degraded document images and image patches of the binary label images; inputting the image patches of the degraded document images and the image patches of the binary label images into an edge detection algorithm and a majority algorithm respectively to obtain the pre-input images of the degraded document images and the pre-input images of the binary label images.
3. The method of claim 1, wherein the method is based on an adaptive convolution for binarization of degraded document images, characterized in that, The gated pyramid module is expressed by the following formula: GDConv = gate(Q(I j C )) wherein GDConv is a gated deconvolution, gate is a gating function, Q is a bilinear interpolation operation, + is a feature map addition fusion, j is a feature map level, C is a to-be-fused feature map, P is a fused feature map, I j C is a feature map delivered by down-sampling, I j p is a feature map fused by different scale features.
4. The method of claim 1, wherein the method is based on an adaptive convolution for binarization of degraded document images. The total loss is calculated based on the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, and specifically comprises: wherein, L G is the total loss, L dav is the adversarial loss, is the classification loss, is the similarity loss, is the Jaccard similarity coefficient loss, is the mean absolute error, λ bce is the classification loss weight, λ dice is the similarity loss weight, λ jaccard is the Jaccard similarity coefficient loss weight, λ l1 is the mean absolute error weight, λ adv is the adversarial loss weight, f i is the binarized image of the degraded document image, t i is the binarized label image of the training degraded document image.
5. An adaptive convolution-based document image binarization system, comprising: The method comprises the following steps: The preprocessing module is configured to collect training degraded document images, perform scanning labeling on the training degraded document images to obtain binary label images of the training degraded document images, and perform preprocessing on the training degraded document images and the corresponding binary label images to obtain pre-input images of the degraded document images and pre-input images of the binary label images. The adaptive convolutional network model module is configured to input the pre-input images of the degraded document images into an adaptive convolutional network model to obtain binary images of the degraded document images. The encoder of the adaptive convolutional network model comprises an adaptive convolution module and a guided attention module, and the decoder of the adaptive convolutional network model comprises a gated pyramid module. The loss module is configured to calculate an adversarial loss according to a discrimination accuracy of the binary images of the degraded document images and the binary label images of the training degraded document images. A classification loss is calculated according to a foreground pixel and background pixel difference of the binary images of the degraded document images and the binary label images of the training degraded document images. A similarity loss and a Jaccard similarity coefficient loss are calculated according to a pixel point similarity of the binary images of the degraded document images and the binary label images of the training degraded document images. A mean absolute error is calculated according to a pixel difference of the binary images of the degraded document images and the binary label images of the training degraded document images. The training module is configured to calculate a total loss based on the adversarial loss, the classification loss, the similarity loss, the Jaccard similarity coefficient loss and the mean absolute error, optimize adaptive convolutional network model parameters by minimizing the total loss, and obtain an optimal adaptive convolutional network model. The output module is configured to input pre-input images of a target degraded document image into the optimal adaptive convolutional network model to obtain a binary image of the target degraded document image. The adaptive convolution module comprises a modulation convolution block, a dynamic weight subnetwork, a gradient information extraction block and a feature map extraction block. Convolution kernel weights of the modulation convolution block are expressed by the following formula: wherein W q is the modulated convolution kernel weight, R is the area size of the convolution, W p is the convolution kernel weight, and : is a placeholder; The dynamic weight subnetwork is expressed by the following formula: Θ = σ(RELUCon 1×1 (DW 3×3 (PW 1×1 (I)))) wherein θ is the proportion of the differential information, σ is a Sigmoid function, I is an input feature map, DW 3×3 is a 3x3 depthwise convolution, PW 1×1 is a 1x1 pointwise convolution, RELUCon 1×1 is a ReLU function; The gradient information extraction block is expressed by the following formula: F' = W p F - θ x W q F Wherein, F' is a feature map with gradient information, F is a feature map after convolution, and is an element-wise multiplication operation. The feature map extraction block is expressed by the following formula: wherein O is an extracted feature map, is an activation function; The guided attention module is expressed by the following formula: W = σ(DW 5×5 (PW 1×1 (S[O,W cat ×O]))) wherein, W is the final feature weight, σ is the Sigmoid function, DW 5×5 is a 5x5 depthwise convolution, PW 1×1 is a 1x1 pointwise convolution, S is a pixel rearrangement operation, O is the extracted feature map, W cat is the mixed domain feature weight.
6. An electronic device, comprising: The storage medium comprises a stored computer program, wherein the computer program controls a device where the storage medium is located to perform the adaptive convolution-based degraded document image binarization method when the computer program is running.
7. A computer readable storage medium characterized in that, The storage medium comprises a stored computer program, wherein the computer program controls a device where the storage medium is located to perform the adaptive convolution-based degraded document image binarization method when the computer program is running.