Text image binaryzation method

Abstract

The invention discloses a text image binaryzation method. A MobileViTblock is introduced under a U-Net framework to construct a text image binaryzation model. The left half part of the framework of the U-Net is mainly subjected to operations of convolution, MV2 and Mobile ViT block, and a feature extraction process of down-sampling; a self-attention mechanism module is used at the bottom of the model; the right half part is mainly used for recovering the size of the image through up-sampling convolution operation, and five-time jump connection exists between the left half part and the right half part; and finally, obtaining a final binary grayscale image through a Softmax activation function. According to the model, a lightweight Mobile ViT module is introduced to better learn and extract global information features of a text image, and test results (DIBCO2012 and DIBCO2017) on two data sets reach the current optimal level. The text image binarization method provided by the invention is a simple end-to-end model, and extra image preprocessing or post-processing is not needed.

Description

Technical Field

[0001] The present invention relates to the technical field of text image binarization, and in particular to a text image binarization method. Background Art

[0002] The task of Document Image Binarization (DIB) is to divide the text image into foreground (normal text content) and background (other content). After binarization, the pixel value of the foreground is generally 0, and the pixel value of the background is 255, which is what we usually call black words on white paper. This is an important method to solve the problem of historical document protection and cultural inheritance by digitalizing ancient text materials with a long history. Because manual processing of these large amounts of historical text materials not only takes a lot of time and manpower costs, but is also prone to errors, it is very necessary to use computers to automatically process the images of ancient text materials.

[0003] Therefore, text content analysis and automatic text recognition technologies have emerged. These image processing contents mainly include image preprocessing (denoising), text image binarization, character inspection and recognition, etc. At present, the technology for scanning and recognizing contemporary texts is relatively mature. However, the text recognition effect of ancient text materials with a long history and serious pollution is not very satisfactory. This is because the text recognition of historical text materials is much more difficult than the extraction of modern photo scanned texts. The text presentation forms in ancient texts are extremely rich and diverse, such as Figure 1 as displayed.

[0004] from Figure 1 You can see that there are various text problems in the text image, such as Figure 1 The texture, font size and color depth of the reprinted approximate text in (a) are inconsistent. In ancient texts, there are also distortions in the text content lines, yellowing of the paper, and pollution such as ink stains. There are even damaged and incomplete text materials, severe blurring, and strong interfering background textures in the text area. In short, the analysis and recognition of text information in text images of ancient handwritten texts or early printed products is a very challenging task. Therefore, the research on the task of text image binarization to segment the text is particularly important. Regarding the text image binarization competition, the International Frontier Conference on Handwriting Recognition has been held almost every year since 2009. [1,2,4–12] This shows that the work of text image binarization has very important practical application value and significance.

[0005] Judging from the main research results on text image binarization based on deep learning in recent years [3,13–25], in the field of text image binarization, neural networks do have good expressive ability in the task of segmenting the foreground text and background of text images. This can be seen from the recent champion algorithm for handwritten text image binarization

[26] .

[0006] The champion algorithm

[26] is a typical traditional method that simply uses background estimation and Laplace energy segmentation technology to achieve the task of text image binarization. From the text image binarization results obtained, it can be seen that this traditional binarization method cannot reasonably separate the text and background when the image is contaminated by background textures similar to the text. From the text image binarization results obtained by Rezanezhad et al.

[27] , it can be clearly seen that the text information in the original text image. The binarization results of Rezanezhad et al.

[27] still have some places that are not segmented very well, such as the background information far away from the text information is mistakenly divided into the foreground. However, it can also be clearly seen that the results of text image binarization based on deep learning are far superior to those of traditional methods. Therefore, the text image binarization algorithms in the past three years have basically been studied based on deep learning technology, such as the literature. [16–19,22–25,28] .

[0007] There are two main research directions for text image binarization models based on deep learning technology. One is the model based on convolutional neural network (CNN). [3,13,14,17,19,24,25,28] The other type is a model based on Generative Adversarial Networks (GAN) [16,18,22,23] .

[0008] CNN-based models [3,13,14,17,19,24,25,28] Generally, it has the advantages of CNN itself, namely simple structural design, fewer parameters, and higher computational efficiency, especially friendliness to limited computing resources. Models under the GAN framework [16,18,22,23] It has the characteristics of GAN itself, that is, through an unsupervised generative model, it can produce clearer and more realistic sample results based on a well-trained discriminator. However, GAN also has some disadvantages, such as the possibility of mode collapse and mode oscillation during training, which may lead to unstable quality or lack of diversity of generated text images; GAN training is usually more complicated and requires careful design and adjustment of model parameters, which leads to a longer time to achieve the desired effect.

[0009] From the above analysis, we can see that the essence of text image binarization is to segment the target image (i.e., segment the foreground text and the background). The U-Net network was first proposed by Olaf et al. in 2015.

[29] It was proposed to be used in medical image segmentation tasks and achieved excellent results. It is precisely because of its excellent performance in segmentation tasks that this structure has attracted the attention of binarization related research, and some text image binarization methods based on U-Net have emerged. [27,30–32] . This is due to the two major advantages of the U-Net network itself: the first point is the simple left-right symmetrical convolution model structure, which is essentially the process of encoding and decoding through pure convolution and downsampling and upsampling. These two processes are to compress image features, retain only key information, and restore the image at the same time, ensuring that the output image is the same size as the input image. The second point is that in the encoding and decoding process, there is a jump connection method, which makes the pixel-level features of the image merged with the semantic features, and this fusion of features at different scales is very beneficial for the restored image through upsampling. Therefore, the U-Net network can achieve pixel-level semantic segmentation, so it is very suitable for the task of segmenting text and background of text images. In addition, U-Net has another advantage, that is, by using data enhancement and special network structure, it can also obtain good learning effects with only a small amount of labeled data, which is very suitable for data sets with only a small amount of labeled text images. In short, the pure convolutional network can be easily adjusted and optimized according to different task requirements due to its simple structural design, and can adapt to image processing tasks of various sizes and types. U-Net has these features, while the simple cascaded convolutional neural network cannot, like U-Net, because of its own symmetrical structure and jump connection design structure. It can transmit information between different layers of the network, fuse feature maps of different resolutions, and effectively combine depth and context information to retain more detail information, thereby achieving better text image binarization effect.

[0010] It is precisely because of these advantages of convolution and U-Net network that the model of text image binarization of cascade convolution is [3] And pure U-Net model for experimental exploration. However, there are problems in some cases, such as Figure 2 As shown. Figure 2 In the figure, the binarization results shown in (c) and (d) show that the texture background information similar to the photocopy content and text information in the original text image cannot be segmented reasonably. Figure 2The original text image in (a) is seriously polluted, and even the texture in the background has a strong contrast. Relying solely on simple cascade convolution or U-Net network training cannot fully capture the overall features and achieve the desired "black and white" binarization effect. This problem mainly stems from the limitation of the receptive field range, that is, simple cascade convolution and U-Net networks are difficult to learn a wider range of receptive field features.

[0011] In biological neural networks, the receptive field refers to the input area that neurons "see". In deep learning convolutional neural networks, the calculation of an element in the feature map is affected by a certain area of ​​the input image. This area is the receptive field of the element. There are two specific methods for calculating the receptive field. One is to calculate it step by step starting from the input layer.

[33] , the typical calculation formula is as follows Among them l k ,l k-1 ,f k ,s i denote the receptive field of the kth layer, the receptive field of the k-1th layer, the size of the convolution kernel of the kth layer, and the stride of the ith layer. Another method is proposed by Luo et al.

[34] They proposed to use back propagation to calculate the "effective receptive field". In their paper, the effective receptive field is defined as the area of ​​input pixels that have a certain influence on the central output unit. By analyzing the distribution of effective receptive fields under different weights, they derived the mathematical characteristics of the effective receptive field in the general case of nonlinear activation functions, and the main calculation formulas (1-1), (1-2), as well as the variance formula (1-3) and expectation (1-4).

[0012]

[0013] Where l is the loss function, the pixels of each layer are indexed by (i, j) and their center is (0, 0). The (i, j) pixel of the pth layer is denoted as in is the input of the network, is the output of the nth layer, and the purpose is to measure the contribution of each x to y.

[0014]

[0015] Where g(i,j,p) is the gradient of the p-th layer, and is the gradient of the activation function of pixel (i,j) on the p-th layer. It is the convolution weight value of the convolution kernel at (a, b) of layer p.

[0016]

[0017]

[0018] Using Luo et al.

[34] The proposed effective receptive field calculation method plots the receptive fields of representative RestNet and Transformer before and after training Figure 3 .from Figure 3 The comparison results of (c) and (d) clearly show that after Transformer training, the receptive field of the central element almost covers the entire image, which is significantly better than the effect of ResNe (t pure convolution) (the larger the value, the stronger the impact). This shows that the Transformer architecture model can effectively handle long-distance dependencies through the self-attention mechanism, allowing the model to better understand the relationship between distant positions, thereby better capturing global dependencies. In addition, the flexibility and versatility of the Transformer architecture makes it applicable to various computer vision tasks (Vision Transformer, ViT for short). It is precisely its powerful modeling capabilities that have enabled its structural form to continue to flourish.

[0019] In 2021, Sachin and Mohammed

[35] The proposed Mobile ViT model is a lightweight VisionTransformer architecture. The core components of the model are MV2 and Mobile ViT blocks, which combine the advantages of CNN and Transformer attention mechanisms. The most important module, Mobile ViT block, has the advantage of flexibly combining global self-attention and local feature extraction, and has the following four advantages in balancing performance and computational efficiency. The first is its lightweight structure, which makes it suitable for mobile devices with limited computing resources. The second is hierarchical feature extraction. Mobile ViT block contains multiple sublayers, which can effectively extract features at different abstract levels of the image. The third is cross-layer connection, because it introduces a mechanism of lateral connection or skip connection, which is conducive to improving the efficiency and accuracy of feature transfer. Finally, the feature of multi-scale processing can support inputs of different scales (i.e., input images of different sizes). These advantages make the Mobile ViT model perform well in balancing performance and computational efficiency. Summary of the invention

[0020] The present invention provides a method for binarizing a text image. The model involved is a simple end-to-end model, which not only greatly reduces the number of parameters but also has good robustness.

[0021] The technical solution adopted by the present invention is:

[0022] A text image binarization method is proposed. MobileViTblock is introduced under the U-Net architecture to build a text image binarization model.

[0023] The left half of the U-Net architecture is mainly through convolution, MV2 and Mobile ViT block operations, and downsampling feature extraction process; the bottom of the model uses a self-attention mechanism module; the right half is mainly through upsampling convolution operations to restore the size of the image, there are five jump connections between the left and right halves; finally, the final binary grayscale image is obtained through the Softmax activation function.

[0024] The left half of the U-Net architecture uses three Mobile ViT blocks, and the number of channels in the three Mobile ViTblocks is 64, 80, and 96 respectively.

[0025] The specific process of a method for binarizing a text image is as follows: the left half of the process: input a 512*512 three-channel image, first perform a 3*3 convolution operation and downsample to obtain a 256*256 16-channel feature map, and then perform an MV2 convolution operation to obtain a feature map of the same size; continue to convolve and downsample to obtain a 128*128 24-channel feature map, and perform two MV2 convolution operations in succession; then convolve and downsample to obtain a 64*64 48-channel feature map, and obtain a 64-channel feature map of the same length and width through a Mobile ViT block convolution operation; continue to convolve and downsample to obtain a 32*32 64-channel feature map, and continue to operate the Mobile ViT block convolution operation to obtain an 80-channel feature map of the same length and width; convolve and downsample to obtain a 16*16 80-channel feature map, use the Mobile ViT block convolution operation to obtain a 96-channel feature map of the same length and width, and obtain a 16*16 320-channel feature map through a 1*1 convolution.

[0026] Model bottom process: For the 16*16 320-channel feature map obtained in the left half, the self-attention module is used to obtain a 16*16 64-channel feature map, and convolution is performed to obtain a 16*16 1024-channel feature map.

[0027] The process of the right half is as follows: for the 16*16 1024-channel feature map obtained at the bottom of the model, after upsampling, it is combined with the 32*32 80-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but the number of channels is 512; after upsampling, it is combined with the 64*64 64-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but the number of channels is 256; after upsampling, it is combined with the 128*128 24-channel feature map on the left and convolved 3*3 to obtain the same The feature map has the same length and width but 128 channels; after upsampling, it is combined with the 256*256 16-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but 64 channels; after upsampling, it is combined with the 512*512 3-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but 32 channels; then convolved 3*3 to obtain a feature map with the same length and width but 2 channels; finally, the softmax activation function is used to obtain the final binary image.

[0028] The model of the present invention introduces a lightweight Mobile ViT module to better learn and extract global information features of text images, and can better learn local and global features. The test results on two datasets (DIBCO2012 and DIBCO2017) have reached the current best level. In addition, the text image binarization method proposed in the present invention is a simple end-to-end model, which does not require additional image pre-processing or post-processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A text image and its binarization result image;

[0030] Among them: (a) Original text image, the image comes from the DIBCO dataset [1,2] , (b) Method [3] The binarization result obtained;

[0031] Figure 2 A comparison of a text image and its GT and the results obtained by two binarization methods;

[0032] Among them: (a) original text image, (b) ideal binarization result (GT), (c) Calvo and Gallego [3] Binarization results, (d) Rezanezhad et al.

[37] The binarization result of

[0033] Figure 3 Comparison of the receptive field effects of RestNet and Transformer before and after training for the central element; (the change in color from light yellow to dark green indicates that the receptive field of the area has a greater impact on the central element);

[0034] Figure 4 It is the overall architecture diagram of the present invention;

[0035] Figure 5 For a test image (DIBCO 2017 [5] Dataset, image number 19-2017) and its GT (GroundTruth, the ideal binarization result) and the comparison of the results obtained by different binarization methods;

[0036] Among them: (a)Otsu

[36] , (b) Sauvola

[37] , (c) Xiong et al.

[26] Binarization results, (d) Rezanezhad et al.

[27] Binarization results, (e) Calvo and Gallego [3] Binarization result of, (f) the present invention;

[0037] Figure 6 The FM line graph and PSNR line graph are obtained by using different numbers of channels in different Mobile ViT blocks. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings and experiments.

[0039] Reference Figure 4 A text image binarization method is proposed. The MobileViT block is introduced under the U-Net architecture to build a text image binarization model. The left half of the U-Net architecture mainly extracts features through convolution, MV2 and Mobile ViTblock operations, and downsampling; the bottom of the model uses a self-attention mechanism module; the right half mainly restores the image size through upsampling convolution operations, and there are five jump connections between the left and right halves; finally, the final binary grayscale image is obtained through the Softmax activation function.

[0040] The left half of the U-Net architecture uses three Mobile ViT blocks, and the number of channels in the three MobileViT blocks is 64, 80, and 96, respectively.

[0041] The specific process of a method for binarizing a text image is as follows: the left half of the process: input a 512*512 three-channel image, first perform a 3*3 convolution operation and downsample to obtain a 256*256 16-channel feature map, and then perform an MV2 convolution operation to obtain a feature map of the same size; continue to convolve and downsample to obtain a 128*128 24-channel feature map, and perform two MV2 convolution operations in succession; then convolve and downsample to obtain a 64*64 48-channel feature map, and obtain a 64-channel feature map of the same length and width through a Mobile ViT block convolution operation; continue to convolve and downsample to obtain a 32*32 64-channel feature map, and continue to operate the Mobile ViT block convolution operation to obtain an 80-channel feature map of the same length and width; convolve and downsample to obtain a 16*16 80-channel feature map, use the Mobile ViT block convolution operation to obtain a 96-channel feature map of the same length and width, and obtain a 16*16 320-channel feature map through a 1*1 convolution.

[0042] Model bottom process: For the 16*16 320-channel feature map obtained in the left half, the self-attention module is used to obtain a 16*16 64-channel feature map, and convolution is performed to obtain a 16*16 1024-channel feature map.

[0043] The process of the right half is as follows: for the 16*16 1024-channel feature map obtained at the bottom of the model, after upsampling, it is combined with the 32*32 80-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but the number of channels is 512; after upsampling, it is combined with the 64*64 64-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but the number of channels is 256; after upsampling, it is combined with the 128*128 24-channel feature map on the left and convolved 3*3 to obtain the same The feature map has the same length and width but 128 channels; after upsampling, it is combined with the 256*256 16-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but 64 channels; after upsampling, it is combined with the 512*512 3-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but 32 channels; then convolved 3*3 to obtain a feature map with the same length and width but 2 channels; finally, the softmax activation function is used to obtain the final binary image.

[0044] The five jump connections between the right and left halves allow the image to be fused with features and semantic features at the pixel level, and the fusion is five times at different scales, which helps to restore the image after upsampling.

[0045] The above downsampling is 2 times downsampling.

[0046] Experimental verification of the present invention

[0047] 1. Experimental description

[0048] In order to objectively evaluate the performance of the model proposed in this invention, the same method as Rezanezhad et al.

[27] The same dataset is used for experimental training. Specifically, the DIBCO dataset from recent years is used. [1,2,4,5,7–11] , Bali Palm Leaf Dataset from the 15th International Conference on Frontiers in Handwriting Recognition

[39] and PHIBC of the Persian Heritage Text Image Binarization Competition

[40] In order to conduct comparative experimental analysis, DIBCO 2012

[10] , DIBCO 2017 [5] and DIBCO 2018 [2] The rest of the datasets were used as training datasets, which is similar to Rezanezhad et al.

[27] Models and other models [15,17,21,23,38] Be consistent.

[0049] The training image size, learning rate and number of epochs of the image binarization model of the present invention are obtained through a large number of experimental parameter adjustments. The specific training model process is divided into two stages: the first stage does not perform a block operation on the image of the training data set, the purpose is to learn the model from the overall structural features, and the complete image information after data enhancement processing is used for training, and the training epoch is set to 30; the second stage focuses on the learning ability of local detail information. In this stage, the original text image is cut into pieces, resulting in an increase in the amount of training data to more than four times that of the first stage, so the training epoch is set to 20.

[0050] Generally, the loss function for text image binarization model training is F value (FM), as shown in the literature [3,27] As described in. The F value (FM) of the image is defined as in formula (2-1).

[0051]

[0052] in The three quantities TO, FP, and FN respectively represent the correct positive values ​​(pixels of the text foreground are correctly classified as text), the wrong positive values ​​(pixels of the background are wrongly classified as text foreground), and the wrong negative values ​​(pixels of the text foreground are wrongly classified as background) obtained in the experiment. The specific operation is to set the pixel value of the text foreground in the true binary label (GT) to 1 ("positive value") and the background pixel value to 0 ("negative value") during the training process. Substituting the calculation method of Precision and Recall into formula (3-4), the simplified objective function formula is:

[0053]

[0054] In the process of model training, an important problem is the insufficient amount of training data. This is because the DIBCO dataset in recent years [1,2,4–11] There are only 11 datasets, containing a total of 146 pairs of original text images and their true binary label images (GT), plus the Bali Palm Leaf dataset

[39] 50 pairs and PHIBC dataset

[40] There are only 212 pairs of valid images and labels in total. If the training target is the DIBCO2017 dataset [5] , then there are only 172 pairs of images and their GT that can be used for training (excluding DIBCO2017 [5] 20 pairs, also do not contain DIBCO 2019 [6] Because the text images in this dataset are more complex, the type of pollution and the text are very different from the characteristics shown in other datasets, and the other models used for comparison [15,17,21,23,27,38] The training process did not use the above two data [5,6] ). This is not enough for neural network model training, so similar to Rezanezhad et al.

[27] The existing dataset was enhanced by [20,24] , mainly including image rotation, scale transformation, brightness adjustment and image slicing operations. In the specific training process, for DIBCO2012

[10] , DIBCO 2017 [5] and DIBCO 2018 [2] The model training selected an input image size of 512×512. The training process is divided into two stages. In the first stage, the learning rate value is 1e-3 or 8e-4 (adjusted according to the curve and results during training). In the second stage, a learning rate timer is used, and the learning rate is halved every five iterations. The initial learning rate size is the same as the first stage.

[0055] In summary, through the comparative analysis of experimental results, the present invention, in DIBCO2012

[10] and DIBCO2017[5] datasets (compared to Rezanezhad et al.

[27] The model of text image binarization performed better in all four indicators. [2] The results on the dataset are not very satisfactory, and this is analyzed. In addition, this invention is also presented at DIBCO 2019 [6]The validation and comparison experiments were conducted on the dataset (because the images in this dataset are severely damaged and the image features are different from the training dataset), and the results exceeded those of the model with the same parameters.

[27] and other traditional methods. Next, the experimental results will be analyzed and discussed from both qualitative and quantitative aspects.

[0056] 2. Qualitative analysis

[0057] The experiments of the present invention were conducted at DIBCO2012

[10] ,DIBCO2017 [5] and DIBCO 2018 [2] The text image binarization model proposed in this paper is compared with the model based on deep learning. [15,17,21,23,27,38] And the classic traditional method [26,36,37] In these three data sets [2,5,10] Comparison is performed by listing the visual comparison diagrams of the binarization results and calculating the relevant evaluation indicators. First, Figure 5 The visual comparison results of different methods on different datasets are shown.

[0058] The present invention Figure 5 A completely satisfactory text image binarization result is obtained. Figure 5 The traditional method in (Otsu et al.

[36] , Sauvola et al.

[37] and Xiong et al.

[26] ) can be seen from the binarization results that for the shadows similar to text information in the background, these methods are difficult to correctly segment as background content.

[27] Binarization result of Calvo and Gallego [3] In comparison, the background content can be segmented well near the text, but the photocopy pollution area far away from the text area cannot be segmented correctly. This is because Rezanezhad et al.

[27] The model has limited ability to learn the features of global information of text images. The advantage of the model of the present invention is that it can effectively fuse global information with local information features extracted by convolution using a lightweight MobileViTblock module.

[0059] 3. Quantitative analysis

[0060] There are four popular evaluation indicators for the quantitative numerical evaluation of the binarization results of text images: image F-measure FM (F-measure), pseudo F-measure pFM (pseudo F-measure)

[40] , Peak Signal to Noise Ration (PSNR) and Distance Reciprocal Distortion (DRD)

[41] These four evaluation indicators show the quality of the binary image obtained from different aspects, and they are obviously better than the simple accuracy.

[0061] (1)FM

[0062] Before giving the definitions of the evaluation indicators FM and pFM required by the present invention, the following four concepts are first introduced, namely, TP, FP, TN, and FN, which respectively represent the correct positive value, the wrong positive value, the correct negative value, and the wrong negative value obtained in the experiment. The FM calculation formula (2-1) is also given in the experimental description of the present invention, and the accuracy rate (Accuracy) is defined as follows:

[0063]

[0064] In the binarization process, Precision is the correct probability of being judged as 1, and Recall is the probability of being truly 1 to being judged as 1. From formula (2-1) and formula (2-3), we can see that FM is more reasonable than Accuracy in evaluating text image binarization. Because the proportion of text in the binary text is much smaller than the pixels in the blank part, using FM is more scientific than simply looking at the accuracy of the pixels of the entire image.

[0065] (2) pFM

[0066] The pseudo F value (pFM) of an image is defined as:

[0067]

[0068] The precision is the same as that in FM, and pRecall represents the percentage of the character structure in the binarized image and the standard image. It can be clearly seen from the definitions of FM and pFM above (2-1, 2-4) that the values ​​of these two indicators are positively correlated with the quality of the text image binarization result.

[0069] (3)PSNR

[0070] Peak signal-to-noise ratio (PSNR) is an indicator related to mean square error for evaluating image quality. It is an objective evaluation indicator that mainly represents the difference between the result after image processing and the real image. In the present invention, it is used to evaluate the difference between the result after text image binarization and the real text image binarization image. The specific calculation is as follows:

[0071]

[0072] MSE refers to the mean square error between the binarized result and the real binarized image. For the general 8-bit image representation method, n=8 and the peak value is 255=2. 8 -1. The unit of PSNR is dB. The larger the value, the more similar the binarization result is to the real binarized text image. This is an objective indicator widely used by the public to evaluate image quality.

[0073] (4)DRD

[0074] Distance Reciprocal Distortion Metric DRD

[41] It is to objectively express the visual perception distortion of the binary text image through the distance between pixels. The specific definition is as follows:

[0075]

[0076] Among them, DRD k is the distortion of the kth flipped pixel, and NUBM is the number of non-uniform (not all black or white) blocks in the real binary image. The larger the DRD value, the greater the visual distortion of the binarization result, so the smaller the DRD value, the better the binarization result of the obtained text.

[0077] The following are listed in DIBCO2012

[10] ,DIBCO2017 [5] and DIBCO 2018 [2] Methods on three datasets [15,17,21,23,26,27,36–38] The average values ​​of the four indicators of the binarization results obtained by the present invention are shown in Table 1, Table 2 and Table 3.

[0078] From Table 1 and Table 2, we can see that the simple end-to-end model of the present invention outperforms other traditional methods in the mean values ​​of four common text image binarizations. [26,36,37] And the deep learning model under the same training data set conditions [15,17,21,23,27,38] In particular, the performance of the single model of the present invention also exceeds that of Rezanezhad et al.

[27] The integrated model shows the absolute superiority of the performance of the proposed model. However, as can be seen from Table 3, the proposed model does not exceed Jemni et al. on the DIBCO 2018 dataset.

[23] and Rezanezhad et al.

[27] The model will also be experimentally analyzed.

[0079] From Table 3, we can see that the model of the present invention is effective for DIBCO 2018 [2] The results of the four indicators on the dataset are not as good as the model [23,27] In order to find out the specific reasons, the DIBCO 2018 obtained by the model [2] The values ​​of the four evaluation indicators of the binarization results of the ten text images in the dataset are listed in Table 4.

[0080] Table 1 Different binarization methods and the present invention in DIBCO2012

[10] Comparison of results on the dataset

[0081]

[0082] Table 2 Different binarization methods and the present invention in DIBCO2017 [5] Comparison of results on the dataset

[0083]

[0084] Table 3 Different binarization methods and the present invention in DIBCO2018 [2] Comparison of results on the dataset

[0085]

[0086] The specific values ​​in Table 4 show that the PSNR values ​​of binarization of images 2-2018 and 10-2018 are low and the FM values ​​are also small, which directly leads to the poor performance of the proposed model for DIBCO 2018. [2] The average result on the dataset is not satisfactory.

[0087]

[0088] 4. Robustness Verification

[0089] The model and other methods of the present invention are described below. [26,27,36,37] Further experimental comparison is carried out on DIBCO 2019, where the image damage is more serious. [6] The experiments were conducted on the DIBCO 2017 dataset. [5]The model is obtained by using the minimum number of training dataset images (a total of 162 pairs of labeled text images and their true binarization results). This paper also uses these four binarization indicators PSNR, FM, pFM and DRD in DIBCO 2019 [6] The means of the twenty results in the data set are quantitatively compared, and the specific means of various methods are shown in Table 5.

[0090] In Table 5, in order to make the measurement model more objective and fair

[27] The robustness of the model of the present invention is demonstrated by

[27] Because Rezanezhad et al.

[27] The model provided is an integration of several trained models, and its parameter count is about 37.0M, while the parameter count of the model of the present invention is about 8.9M. The parameter counts of the two models are not in the same order of magnitude.

[42] In the paper, detailed experiments were conducted and it was verified that the good performance of the original model can be maintained by adjusting the network scale accordingly in depth.

[42] The method proposed in (Reference

[42] (d) of the study, i.e. the method of reducing the number of parameters on the depth parameter) for Rezanezhad et al.

[27] The parameter amount of the model is adjusted to be about 9.0M, which is on the same order of magnitude as the parameter amount of the model of the present invention. At this time, when other training conditions are consistent, model training is performed.

[0091] Table 5 Different binarization methods and the present invention at DIBCO 2019 [6] Comparison of results on the dataset. The best and second best indicator values ​​are marked in red and blue respectively.

[0092]

[0093] From Table 5 we can see that Otsu

[36] The method has the worst numerical performance in the binarization indicators PSNR and DRD. Although Xiong et al.

[26] Methods at DIBCO2018 [2] The algorithm won the championship in the dataset, but in DIBCO 2019 [6] The numerical results of FM and pFM on the dataset are the worst, indicating that the robustness of the method is not very good.

[27] -37.0M in Table 5 for DIBCO 2019 [6]The four average indicators of the images in the dataset are all optimal. The main reason for this is that the huge number of parameters of the model makes its robust performance optimal. [6] The average values ​​of PSNR, FM and DRD of all images in the dataset are second, and only the pFM index is not as good as the model

[27] -9.0M. It can be said that among the models with the same number of parameters, the model of the present invention is superior to that of Rezanezhad et al.

[27] The DIBCO 2019 obtained by the model of the present invention [6] It is found that there is an image (8-2019-a) whose binarization evaluation index value obtained by the model of the present invention is particularly low. Its pFM value is lower than that of the model

[27] -9.0M is about 33 smaller, which directly leads to the model of the present invention being selected in DIBCO 2019 [6] Lower pFM metric on the dataset.

[0094] 5. Ablation experiment

[0095] In this section, we analyze the role of the core modules in the proposed model and the impact of their corresponding channel settings on network performance. All ablation experiments were conducted at DIBCO 2017 [5] Validated on the dataset.

[0096] 5.1 Experiments on the performance of MobileViT block

[0097] The first set of ablation experiments is to explore the number of Mobile ViT blocks in the model of the present invention. The specific experiment is to replace the three MobileViTblocks in the network with MV2 in sequence by changing the number of the modules while keeping other variables the same. Zero, one, two and three Mobile ViT blocks are used to train the model of the present invention, and the four obtained models are recorded as M0, M1, M2 and M3 (the present invention). These four models are used to train the DIBCO 2017 [5] The data sets are binarized and their respective average evaluation index (FM, pFM, PSNR, DRD) values ​​are calculated. The results are recorded in Table 6.

[0098] Table 6 uses zero, one, two and one (the present invention) Mobile ViT The model obtained by the block module is in the DIBCO 2017 dataset [5] Comparison of results on

[0099] method PSNR FM <![CDATA[ pFM ]]> DRD M0 18.56 91.57 94.91 2.94 M1 18.84 92.45 95.37 2.55 M2 18.88 92.39 95.12 2.75 M3 19.29 93.23 95.99 2.22

[0100] In Table 6, it can be observed that the average value of PSNR gradually increases from M0 through M1, M2, to M3. This shows that as the number of Mobile ViT blocks increases, the similarity between the binarization result obtained by network training and the real binarized text increases. From M0 to M1, then to M2, and finally to M3, the values ​​of FM and pFM have experienced a process from small to large, then to small, and finally to the largest (the change trend of DRD is similar). Finally, the values ​​of these four evaluation indicators reach the optimal level in M3 (model of the present invention). Therefore, the three Mobile ViT blocks selected by the present invention are reasonable.

[0101] 5.2 Experiments on the number of channels in MobileViT block

[0102] The second ablation experiment is to analyze the impact of the number of channels in the three MobileViTblocks on the model performance. The specific operation is to use 32, 64 and 96 channels for the first Mobile ViT block, 40, 80 and 96 channels for the second, and 48, 96 and 144 channels for the third to train the model. The numbers of channels used in the present invention are 64, 80 and 96, respectively. In the experiment, the number of channels in the first two modules is controlled, and different numbers of channels are used in the third module. The average FM and PSNR of the final binarization results obtained by different channel numbers are shown in their respective line graphs. Figure 6 Displayed in.

[0103] from Figure 6 It can be seen that, for both the FM index value and the PSNR value, in each Mobile ViTblock, when the number of three channels is selected as the middle value, the two evaluation indicators reach the best level. Therefore, the middle number of channels selected by the model of the present invention is reasonable.

[0104] The present invention is a simple end-to-end model, compared to the similar Rezanezhad et al.

[27] The model has a 76% reduction in parameters. It has shown even better performance on the DIBCO 2012 and DIBCO 2017 [5, 10] datasets, with significant improvements in common evaluation indicators such as FM, pFM, PSNR, and DRD. To verify the robustness of the model, Rezanezhad et al.

[27] The two parameter models and the classic traditional method and the present invention are presented in DIBC02019 [6]The text image binarization process was performed on the data set. By comparing the mean values ​​of four evaluation indicators of the binarization results obtained by different methods, it was confirmed that the model proposed in the present invention has good robustness.

[0105] References

[0106] [1]Pratikakis I,Gatos B,Ntirogiannis K.Icdar 2013document imagebinarization contest(dibco 2013)[C].Proceedings of 201312th International Conference on Document Analysis and Recognition,1471–1476,2013.

[0107] [2]Pratikakis I,Zagori K,Kaddas P,et al.Icfhr 2018competition onhandwritten document image binarization(h-dibco 2018)[C].Proceedings of 201816th International Conference on Frontiers in Handwriting Recognition(ICFHR),489–493,2018.

[0108] [3]Calvo-Zaragoza J, Gallego A JA selectional auto-encoder approach for document image binarization[J].Pattern Recognition, 2019,86:37–47.

[0109] [4]Pratikakis I, Zagoris K, Barlas G, et al.Icfhr2016 handwrittendocument image binarization contest(h-dibco 2016)[C].Proceedings of 201615thInternational Conference on Frontiers in Handwriting Recognition(ICFHR),619–623,2016.

[0110] [5]Pratikakis I,Zagoris K,Barlas G,et al.Icdar2017 competition ondocument image binariza-tion(dibco 2017)[C].Proceedings of 201714th IAPRInternational Conference on Document Analysis and Recognition(ICDAR),volume01,1395–1403,2017.

[0111] [6]Pratikakis I,Zagoris K,Karagiannis X,et al.Icdar 2019competitionon document image binarization(dibco 2019)[C].Proceedings of2019International Conference on DocumentAnalysis and Recognition(ICDAR),1547–1556,2019.

[0112] [7]Gatos B,Ntirogiannis K,Pratikakis I.Icdar 2009document imagebinarization contest(dibco 2009)[C].Proceedings of 200910th InternationalConference on Document Analysis and Recognition,1375–1382,2009.

[0113] [8]Pratikakis I,Gatos B,Ntirogiannis K.H-dibco 2010-handwrittendocument image bina-rization competition[C].Proceedings of 201012thInternational Conference on Frontiers in Handwriting Recognition,727–732,2010.

[0114] [9]Pratikakis I,Gatos B,Ntirogiannis K.Icdar 2011document imagebinarization contest(dibco 2011)[C].Proceedings of 2011InternationalConference on Document Analysis and Recog-nition,1506–1510,2011.

[0115]

[10] Pratikakis I,Gatos B,Ntirogiannis K.Icfhr 2012 competition onhandwritten document image binarization(h-dibco 2012)[C].Proceedings of 2012International Conference on Frontiers in Handwriting Recognition,817–822,2012.

[0116]

[11] Ntirogiannis K,Gatos B,Pratikakis I.Icfhr2014 competition onhandwritten document im-age binarization(h-dibco 2014)[C].Proceedings of201414th International Conference on Frontiers in Handwriting Recognition,809–813,2014.

[0117]

[12] Seuret M,Nicolaou A,Stutzmann D,et al.Icfhr 2020 competition onimage retrieval for historical handwritten fragments[C].Proceedings of202017th International Conference on Frontiers in Handwriting Recognition(ICFHR),216–221,2020.

[0118]

[13] Pastor-Pellicer J, -Boquera S,Zamora-Martínez F,etal.Insights on the use of con-volutional neural networks for document imagebinarization[C].In:Rojas I,Joya G,Catala A,(eds.).Proceedings of Advances inComputational Intelligence,115–126,Cham,2015.Springer InternationalPublishing.

[0119]

[14] Calvo-Zaragoza J,Vigliensoni G,Fujinaga I.Pixel-wise binarizationof musical documents with convolutional neural networks[C].Proceedings of2017 Fifteenth IAPR International Conference on Machine Vision Applications(MVA),362–365,2017.

[0120]

[15] Zhao J,Shi C,Jia F,et al.Document image binarization withcascaded generators of condi-tional generative adversarial networks[J].Pattern Recognition,2019,96:106968.

[0121]

[16] Souibgui M A,Kessentini Y.De-gan:A conditional generativeadversarial network for docu-ment enhancement[J].IEEE Transactions on PatternAnalysis and Machine Intelligence,2022,44(3):1180–1191.

[0122]

[17] Kang S,Iwana B K,Uchida S.Complex image processing with lessdata-document image binarization by integrating multiple pre-trained u-netmodules[J].Pattern Recognition,2021,109:107577.

[0123]

[18] Kumar A,Ghose S,Chowdhury P N,et al.Udbnet:Unsupervised documentbinarization net-work via adversarial game[C].Proceedings of 2020 25thInternational Conference on Pattern Recognition(ICPR),7817–7824.IEEE,2021.

[0124]

[19] Xiong W,Jia X,Yang D,et al.Dp-linknet:A convolutional network forhistorical document image binarization[J].KSII Transactions on Internet andInformation Systems(TIIS),2021,15(5):1778–1797.

[0125]

[20] Nicolaou A,Christlein V,Riba E,et al.Tormentor:Deterministicdynamic-path,data aug-mentations with fractals[C].Proceedings of Proceedingsof the IEEE / CVF Conference on Computer Vision and Pattern Recognition,2707–2711,2022.

[0126]

[21] Souibgui M A,Biswas S,Jemni S K,et al.Docentr:An end-to-enddocument image enhance-ment transformer[C].Proceedings of 202226thInternational Conference on Pattern Recogni-tion(ICPR),1699–1705.IEEE,2022.

[0127]

[22] Suh S,Kim J,Lukowicz P,et al.Two-stage generative adversarialnetworks for binarization of color document images[J].Pattern Recognition,2022.108810.

[0128]

[23] Jemni S K,Souibgui M A,Kessentini Y,et al.Enhance to read better:A multi-task adver-sarial network for handwritten document image enhancement[J].Pattern Recognition,2022,123:108370.

[0129]

[24] Dey A,Das N,Nasipuri M.Variational augmentation for enhancinghistorical document im-age binarization[J].arXiv preprint arXiv:2211.06581,2022.

[0130]

[25] Yang Z,Xiong Y,Wu G.Gdb:Gated convolutions-based documentbinarization[J].arXiv preprint arXiv:2302.02073,2023.

[0131]

[26] Xiong W,Zhou L,Yue L,et al.An enhanced binarization framework fordegraded historical document images[J].EURASIP Journal on Image and VideoProcessing,2021,2021(1):13.

[0132]

[27] Vahid R,Konstantin B,Clemens N.A hybrid cnn-transformer model forhistorical document image binarization[C].Proceedings of HIP’23:7thInternational Workshop on Historical Document Imaging and Processing,San JoseCA USA,79–84,2023.

[0133]

[28] Xiong W,Yue L,Zhou L,et al.Fd-net:A fully dilated convolutionalnetwork for historical document image binarization[C].Proceedings of PatternRecognition and Computer Vision:4th Chinese Conference,PRCV 2021,Beijing,China,October29–November 1,2021,Pro-ceedings,Part I 4,518–529.Springer,2021.

[0134]

[29] Ronneberger O,Fischer P,Brox T.U-net:Convolutional networks forbiomedical image segmentation[C].In:Navab N,Hornegger J,Wells W M,et al.,(eds.).Proceedings of Medical Image Computing and Computer-AssistedIntervention–MICCAI 2015,234–241,Cham,2015.Springer International Publishing.

[0135]

[30] Nikitin F,Dokholyan V,Zharikov I,et al.U-net based architecturesfor document text detec-tion and binarization[Z],10,2019.

[0136]

[31] Bezmaternykh P,Ilin D,Nikolaev D.U-net-bin:hacking the documentimage binarization contest[J].Computer Optics,2019,43:825–832.

[0137]

[32] Detsikas N,Mitianoudis N,Papamarkos N.A dilated multires visualattention u-net for historical document image binarization[J].SignalProcessing:Image Communication,2024,122:117102.

[0138]

[33] Dumoulin V,Visin F.A guide to convolution arithmetic for deeplearning[Z],2018.

[0139]

[34] Ioffe S,Szegedy C.Batch normalization:Accelerating deep networktraining by reducing internal covariate shift[C].Proceedings of Internationalconference on machine learning,448–456.pmlr,2015.

[0140]

[35] Mehta S,Rastegari M.Mobilevit:Light-weight,general-purpose,andmobile-friendly vision transformer[J].ArXiv,2021.

[0141]

[36] Otsu N.A threshold selection method from gray-level histograms[J].IEEE Transactions on Systems,Man,and Cybernetics,1979,9(1):62–66.

[0142]

[37] Sauvola J, M.Adaptive document image binarization[J].Pattern Recognition,2000,33(2):225–236.

[0143]

[38] Tensmeyer C,Martinez T.Document image binarization with fullyconvolutional neural net-works[C].Proceedings of 201714th IAPR internationalconference on document analysis and recognition(ICDAR),volume 1,99–104.IEEE,2017.

[0144]

[39] Burie J C,Coustaty M,Hadi S,et al.Icfhr2016 competition on theanalysis of handwritten text in images of balinese palm leaf manuscripts[C].Proceedings of 201615th International Conference on Frontiers in HandwritingRecognition(ICFHR),596–601,2016.

[0145]

[40] Ayatollahi S M,Ziaei Nafchi H.Persian heritage image binarizationcompetition(phibc2012)[C].Proceedings of 2013 First Iranian Conference onPattern Recognition and Image Analysis(PRIA),1–4,2013.

[0146]

[41] Lu H,Kot A,Shi Y.Distance-reciprocal distortion measure forbinary document images[J].IEEE Signal Processing Letters,2004,11(2):228–231.

[0147]

[42] Tan M,Le QV.Efficientnet:Rethinking model scaling forconvolutional neural networks[J].CoRR,2019,abs / 1905.11946。

Claims

1. A method for text image binarization, introducing MobileViTblock under the U-Net architecture to build a text image binarization model.

2. The method for binarizing a text image according to claim 1, characterized in that: The left half of the U-Net architecture is mainly through convolution, MV2 and MobileViTblock operations, and downsampling feature extraction process; the bottom of the model uses a self-attention mechanism module; the right half is mainly through upsampling convolution operations to restore the size of the image, there are five jump connections between the left and right halves; finally, the final binary grayscale image is obtained through the Softmax activation function.

3. The method for binarizing a text image according to claim 2, characterized in that: The left half of the U-Net architecture uses three MobileViTblocks, and the number of channels in the three MobileViTblocks is 64, 80, and 96 respectively.

4. The method for binarizing a text image according to claim 2, characterized in that: The specific process of this method is as follows: The left half of the process: input a 512*512 three-channel image, first perform a 3*3 convolution operation and downsample to obtain a 256*256 16-channel feature map, and then perform an MV2 convolution operation to obtain a feature map of the same size; continue to convolve and downsample to obtain a 128*128 24-channel feature map, and perform two MV2 convolution operations in succession; then convolve and downsample to obtain a 64*64 48-channel feature map, and use the MobileViTblock convolution operation to obtain a 64-channel feature map of the same length and width; continue to convolve and downsample to obtain a 32*32 64-channel feature map, and continue the MobileViTblock convolution operation to obtain an 80-channel feature map of the same length and width; convolve and downsample to obtain a 16*16 80-channel feature map, use the Mobile ViTblock convolution operation to obtain a 96-channel feature map of the same length and width, and use the convolution 1*1 to obtain a 16*16 320-channel feature map; The bottom process of the model: For the 16*16 320-channel feature map obtained in the left half, the self-attention module is used to obtain a 16*16 64-channel feature map, and convolution is performed to obtain a 16*16 1024-channel feature map; The process of the right half is as follows: for the 16*16 1024-channel feature map obtained at the bottom of the model, after upsampling, it is combined with the 32*32 80-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but the number of channels is 512; after upsampling, it is combined with the 64*64 64-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but the number of channels is 256; after upsampling, it is combined with the 128*128 24-channel feature map on the left and convolved 3*3 to obtain the same The feature map has the same length and width but 128 channels; after upsampling, it is combined with the 256*256 16-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but 64 channels; after upsampling, it is combined with the 512*512 3-channel feature map on the left and convolved 3*3 to obtain a feature map with the same length and width but 32 channels; then convolved 3*3 to obtain a feature map with the same length and width but 2 channels; finally, the softmax activation function is used to obtain the final binary image.

Images