Improved document image super-resolution method and system based on Real-ESRGAN

Abstract

The invention discloses an improved document image super-resolution method and system based on Real-ESRGAN, and belongs to the technical field of picture processing. The super-resolution method comprises the following steps: S1, constructing a training set: determining pictures used for training, preprocessing each picture to obtain high-resolution pictures and low-resolution pictures used for model training, and forming the training set; s2, model design: replacing a residual dense connection module in the Real-ESRGAN model with an edge detail enhancement module to obtain an edge detail enhanced Real-ESRGAN model; s3, training the Real-ESRGAN model of which the edge details are enhanced by using the training set to obtain a document image super-resolution model; and S4, performing super-resolution processing on a to-be-processed document image by using the trained document image super-resolution model. According to the method, the super-resolution precision of the document image can be effectively improved, and the phenomena of artifacts, wrong features, line distortion and the like are avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a document image super-resolution method and system based on Real-ESRGAN improvement. Background Art

[0002] With the rapid development of the Internet, document images often have a mismatch between image resolution and demand during transmission. Therefore, it is necessary to reconstruct images from low resolution to high resolution. Due to the complex degradation process that images suffer during transmission, it is difficult to obtain pixel-aligned high-resolution and low-resolution images. To solve this problem, Real-ESRGAN simulates low-resolution images through complex degradation models to form data pairs for training, and has achieved good super-resolution results in natural image super-resolution tasks. However, when Real-ESRGAN is used to process document image super-resolution, the super-resolution results will have artifacts, erroneous features, and line distortion, which makes it unable to meet the requirements of document images in Internet transmission. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the above-mentioned related art at least to a certain extent.

[0004] To this end, the purpose of the present invention is to provide a document image super-resolution method and system based on Real-ESRGAN, which can effectively improve the accuracy of document image super-resolution and avoid the occurrence of artifacts, erroneous features, line distortion and the like.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0006] The embodiment of the present invention provides a document image super-resolution method based on Real-ESRGAN improvement, and the super-resolution method includes:

[0007] S1. Construct training set: determine the images to be used for training, pre-process each image, obtain high-resolution images and low-resolution images for model training, and form a training set;

[0008] S2. Model design: Replace the residual dense connection module in the Real-ESRGAN model with the edge detail enhancement module to obtain the Real-ESRGAN model with edge detail enhancement;

[0009] S3, using the training set constructed in S1 to train the Real-ESRGAN model with edge detail enhancement obtained in S2, to obtain a document image super-resolution model;

[0010] S4. Use the trained document image super-resolution model to perform super-resolution processing on the document image to be processed.

[0011] In addition, the document image super-resolution method based on Real-ESRGAN improvement according to the present invention may also have the following additional technical features:

[0012] In some implementations, the content of preprocessing the image includes:

[0013] S11, dividing the image into a number of small blocks of the same size;

[0014] S12, enlarging the segmented small blocks to a preset size as high-resolution images for model training;

[0015] S13. Perform image degradation processing on the high-resolution image used for model training to obtain a low-resolution image used for model training.

[0016] In some of the embodiments, the size of each small block divided in S11 is 128*128;

[0017] The size of high-resolution images in S12 is 512*512;

[0018] The size of low-resolution images in S13 is 128*128.

[0019] In some of the embodiments, the image degradation is processed by using an n-order model including n repeated degradation processes for degradation, and each degradation process uses a classical degradation model with the same process but different hyperparameters.

[0020] In some of the embodiments, the framework structure of the Real-ESRGAN model for edge detail enhancement in step S2 includes a convolutional layer, a plurality of edge detail enhancement modules, a convolutional layer, an upsampling layer, and a plurality of convolutional layers arranged in sequence.

[0021] In some of the embodiments, the edge detail enhancement module includes an edge enhancement module and a detail enhancement module.

[0022] In some of the embodiments, the edge enhancement module adopts a Scharr filter and uses a center difference convolution with a Laplacian kernel, and has horizontal difference convolution, vertical difference convolution and center difference convolution.

[0023] In some of the embodiments, the content of the detail enhancement module includes: performing an average pooling operation on the input and then performing a 1*1 convolution operation to generate a weight matrix, and controlling the size of the output feature map to be the same as the input feature map by padding the size; performing convolution on each channel of the input feature to extract features, and finally multiplying the weights by the features extracted by the convolution to obtain the weighted receptive field spatial features.

[0024] In some of the embodiments, the edge enhancement module and the detail enhancement module are connected in a densely connected manner to form the edge detail enhancement module.

[0025] The embodiment of the present invention further provides a document image super-resolution system based on Real-ESRGAN improvement, which can implement the document image super-resolution method based on Real-ESRGAN improvement as described above; the system includes:

[0026] Training set construction module: used to determine the images used for training, pre-process each image, obtain high-resolution images and low-resolution images for model training, and form a training set;

[0027] Model design module: used to build the model used for subsequent model training. The model replaces the residual dense connection module in the Real-ESRGAN model with the edge detail enhancement module to obtain the Real-ESRGAN model with edge detail enhancement.

[0028] Document image super-resolution model acquisition module: uses the training set constructed by the training set construction module to train the edge detail enhanced Real-ESRGAN model obtained by the model design module to obtain a document image super-resolution model;

[0029] Document image super-resolution module: used to determine the parameters of the document image super-resolution model, and use the document image super-resolution model to perform super-resolution processing on the document image to be processed.

[0030] Compared with the prior art, the present invention has at least the following beneficial effects:

[0031] In the embodiment of the present invention, the document image super-resolution method based on the improved Real-ESRGAN is designed specifically according to the characteristics of the image. For example, there are many characters in the document image, so an edge enhancement module is added to enhance the edge of the image. The detail processing of the Chinese characters with complex strokes is not detailed enough, so a detail enhancement module is added to enhance some details of the image, so that the present invention can achieve effective super-resolution processing for the document image.

[0032] In an embodiment of the present invention, the document image super-resolution method based on Real-ESRGAN improvement is provided. In the data preprocessing stage, the operation of first cropping and enlarging and then degrading is performed. Compared with the original image preprocessing technology, such an operation can make the training data closer to the real document image.

[0033] The document image super-resolution system based on Real-ESRGAN improvement of the present invention can implement the steps of the document image super-resolution method based on Real-ESRGAN improvement, and thus at least has all the features and advantages of the document image super-resolution method based on Real-ESRGAN improvement, which will not be described in detail here. Additional aspects and advantages of the present invention will be partially given in the following description, and part will become obvious from the following description, or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a document image super-resolution method based on Real-ESRGAN improvement disclosed in one embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a Real-ESRGAN model framework disclosed in one embodiment of the present invention;

[0036] Figure 3 Schematic diagram of a differential operator disclosed in an embodiment of the present invention; wherein (a) is a horizontal differential operator, (b) is a vertical differential operator, and (c) is a central differential operator;

[0037] Figure 4 A schematic diagram of the structure of an edge enhancement module disclosed in one embodiment of the present invention;

[0038] Figure 5 A schematic diagram of the connection between an edge enhancement module and a detail enhancement module disclosed in one embodiment of the present invention;

[0039] Figure 6 A structural comparison diagram of a Real-ESRGAN model improved by an edge detail enhancement module disclosed in an embodiment of the present invention and a traditional Real-ESRGAN model;

[0040] Figure 7 A comparison diagram of super-resolution inference results of different models disclosed in an embodiment of the present invention; wherein (a), (b), (c) and (d) are comparison diagrams of inference results of different models for four document images, respectively. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The embodiments of the present invention are described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0043] See also Figure 1 As shown, in some embodiments of the present invention, a document image super-resolution method based on Real-ESRGAN improvement is provided, and the steps of the algorithm include:

[0044] Step 1: Data Preprocessing

[0045] 1) Crop the image into 128*128 blocks;

[0046] 2) Enlarge the cropped image block to a 512*512 size image and use it as a high-resolution image in the model training data;

[0047] 3) Use the same image degradation method as Real-ESRGAN, that is, an n-order model contains n repeated degradation processes, where each degradation process adopts a classic degradation model with the same process but different hyperparameters. The 512*512 size image is degraded to obtain a 128*128 size image, and the processed image is used as the low-resolution image in the model training data.

[0048] The reason why 128*128 is selected in this embodiment of the present invention is that most of the degraded documents seen in daily life are fuzzy documents of the same size as the original documents. In order to obtain a document image that is closer to the real scene, it is also a method adopted to adapt to the degradation process of the original model.

[0049] The 128*128 sized square cropped in 1) above cannot be used directly in the present invention, but needs to be enlarged and then degraded. The reason is that the degradation effect of the original model degradation process is too high, and it is mainly aimed at natural images of large scenes. When it is directly used on a small document image, the text and other detailed information on the document image will be blurred and unrecognizable, and it will be difficult to identify with the naked eye even if it is enlarged. This type of image is rare in daily life, so the method of enlarging and then degrading is adopted.

[0050] Step 2: Model Design

[0051] 1) The overall framework of the model of the present invention follows the framework of Real-ESRGAN, and the residual dense connection module is replaced by a new edge detail enhancement module. The specific framework is as follows: Figure 2 shown.

[0052] 2) The edge detail enhancement module innovatively uses the edge enhancement module and the detail enhancement module. The edge enhancement module is an improved module based on the Sobel filter - the Scharr filter. On its basis, the central differential convolution using the Laplace kernel is innovatively added to form horizontal differential convolution (HDC), vertical differential convolution (VDC) and central differential convolution (CDC). The convolution kernels are denoted as K x , K y and K lap ,like Figure 3 shown.

[0053] The edge enhancement module formed by the connection is as follows Figure 4 shown.

[0054] The algorithm of the present invention innovatively adds three parameters w x 、w y and w lap It is used to control the weights of the three convolution results and dynamically adjust them according to the input data so that the network can learn the importance of each convolution result. Specifically, as shown in the following formula:

[0055]

[0056] Among them, I represents the input feature map of the edge enhancement module, ReLU represents the activation function used, and F edge Represents the output feature map of the edge enhancement module.

[0057] 3) The detail enhancement module used in the edge detail enhancement module generates a weight matrix by performing an average pooling operation on the input and then performing a 1×1 convolution operation. The convolution kernel is The padding size is used to control the size of the output feature map to be the same as the size of the input feature map. On the other hand, convolution is performed on each channel of the input feature to extract the feature. The convolution kernel is Finally, the weights are multiplied by the convolutional features to obtain the weighted receptive field spatial features. The specific calculation process can be expressed by the following formula:

[0058]

[0059] Among them, AvgPool represents the average pooling operation, and Norm represents the normalization operation.

[0060] 4) The edge enhancement module and the detail enhancement module are connected using a dense connection method, such as Figure 5 shown.

[0061] 5) The connection in 4) is connected again using the dense connection method used by Real-ESRGAN, and finally the edge detail enhancement module in the model framework is obtained, such as Figure 6 as shown

[0062] After improving the edge detail enhancement module, the structure diagram of the model compared with the original Real-ESRGAN is as Figure 6 shown

[0063] Step 3: Experimental process

[0064] Set the experimental parameters: Use an A100 to train the model, and set the batch size to 12; Initialize the learning rate to 2×10 -4 ; The training is carried out for 25K iterations, and the learning rate is decayed by 2 times every 2 mini-batch updates ×10 -5 ; The generator is trained using a combination of L1 loss, perceptual loss, and GAN loss, with weights {1, 1, 0.1} respectively; Use the activation in the VGG19 network to calculate the perceptual loss, and set the learning rate to 1×10 -4 .

[0065] 1) The model proposed in the present invention was compared with advanced PSNR-oriented methods, namely EDSR, SRCNN, and the model results after training on a self-built dataset of Real-ESRGAN. The results are shown in Table 1. The average PSNR of the inference results of the proposed model is 0.38 dB higher than that of the original model. The improved model makes the super-resolution results have less distortion and fewer adhesions between strokes. The fonts in the super-resolution images are clear and the strokes are accurate. For images with a high degree of blurriness, the fonts in the images also have higher辨识度.

[0066] Table 1

[0067]

[0068] As Figure 7 (a) shown, the PSNR value of the inference result of the improved model for this example image is nearly 0.9 dB higher than that of the old model. It can also be seen from the figure that the improved model makes the font more recognizable, such as the character "实" in the experiment.

[0069] From Figure 7 (b) it can be seen that the inference result of the new model increases the PSNR of this image by nearly 1.4 dB, the font strokes are more complete, and the deformation degree of the book title brackets is reduced; The improved model not only has a good improvement effect on Chinese document content, but also has a good improvement effect on English documents and documents containing calligraphy fonts.

[0070] As Figure 7 (c) shown, the inference result of the new model increases the PSNR of this image by about 1.8 dB. Among them, the coherence of English letters becomes better and the distortion degree weakens.

[0071] From Figure 7 As can be seen from (d), the new model makes the brush strokes in the calligraphy image clearer and sharper, and the overall PSNR value of the calligraphy image is improved by about 0.8dB.

[0072] 2) Ablation experiment

[0073] The improved model is implemented by using the PyTorch framework, and the ablation experiment is performed on the model using the NVIDIA A100 GPU. First, the training data uses a dataset containing 1537 HR-LR image pairs. The images can be converted from paper documents downloaded from the Internet; 1400 of them are selected as the training dataset and 137 images are selected as the test set. Each experiment uses a model with the same parameters but different modules to train 195 epochs, a total of 25k cycles, and finally calculates the average PSNR value of the inference results of each model. As shown in Table 2, the experimental results show that the two additional modules, the edge enhancement module and the detail enhancement module, have an improvement effect on the model inference results. The PSNR values ​​are increased by 0.12dB and 0.2dB respectively. When they are combined and added to the model, they have better inference results, and the PSNR value is increased by 0.38dB, which proves that the idea of ​​improving the model is feasible and effective.

[0074] Table 2

[0075]

[0076] The switched dataset is screen content images, which are similar to document images and also include LR and HR data pairs. A portion of the data is randomly selected and cropped, and the cropped dataset is expanded to the dataset built by the present invention, so that the number of training datasets increases to 9585, 9000 of which are selected as training sets and 585 as test sets. The experimental results in Table 3 show that the improved model still effectively improves the inference results of images, and the average PSNR is increased by 0.33dB.

[0077] Table 3

[0078]

[0079] In the table of the present invention, a check mark means that the module (edge ​​enhancement, detail enhancement) corresponding to the leftmost column is used, and a cross means that the module (edge ​​enhancement, detail enhancement) corresponding to the leftmost column is not used.

[0080] It should be noted that, regarding the detail enhancement module, the prior art uses an attention mechanism to implement it, such as: a method called residual channel attention network (RCAN), in which the residual channel attention block (RCAB) enables the network to focus on channels with richer information; a channel attention (CA) and spatial attention (SA) module to utilize the relationship between channels and spaces of feature maps; a balanced attention mechanism (BAM), in which the channel attention mechanism aims to suppress extreme noise in large-scale feature maps, while the spatial attention mechanism retains high-frequency texture details; a holistic attention network (HAN), which combines the channel attention mechanism and the inter-layer attention mechanism to retain more image details. This attention aims to refine the final feature map based on the interaction of features at adjacent positions. For the edge enhancement module, the prior art usually adopts a method of extracting edge features and then adding the final result of the original model for training, and no improvement has been made to the algorithm for extracting edge features. Although the above existing methods can be used in document image super-resolution, the comprehensive effect is not as good as the model algorithm of the present invention, and the purpose of the present invention cannot be achieved.

[0081] Parts of the present invention that are not described in detail are well known to those skilled in the art.

[0082] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A document image super-resolution method based on Real-ESRGAN improvement, characterized in that: The super-resolution method comprises: S1. Construct training set: determine the images to be used for training, pre-process each image, obtain high-resolution images and low-resolution images for model training, and form a training set; S2. Model design: Replace the residual dense connection module in the Real-ESRGAN model with the edge detail enhancement module to obtain the Real-ESRGAN model with edge detail enhancement; S3, using the training set constructed in S1 to train the Real-ESRGAN model with edge detail enhancement obtained in S2, to obtain a document image super-resolution model; S4. Use the trained document image super-resolution model to perform super-resolution processing on the document image to be processed.

2. The document image super-resolution method based on Real-ESRGAN improvement according to claim 1, characterized in that: The contents of image preprocessing include: S11, dividing the image into a number of small blocks of the same size; S12, enlarging the segmented small blocks to a preset size as high-resolution images for model training; S13. Perform image degradation processing on the high-resolution image used for model training to obtain a low-resolution image used for model training.

3. The document image super-resolution method based on Real-ESRGAN improvement according to claim 2, characterized in that: The size of each small block divided in S11 is 128*128; The size of high-resolution images in S12 is 512*512; The size of low-resolution images in S13 is 128*128.

4. The document image super-resolution method based on Real-ESRGAN improvement according to claim 2, characterized in that: The image degradation is processed by using an n-order model containing n repeated degradation processes, and each degradation process uses a classical degradation model with the same process but different hyperparameters.

5. The document image super-resolution method based on Real-ESRGAN improvement according to claim 1, characterized in that: The framework structure of the Real-ESRGAN model for edge detail enhancement in step S2 includes a convolutional layer, several edge detail enhancement modules, a convolutional layer, an upsampling layer, and several convolutional layers arranged in sequence.

6. The document image super-resolution method based on Real-ESRGAN improvement according to any one of claims 1 to 5, characterized in that: The edge detail enhancement module includes an edge enhancement module and a detail enhancement module.

7. The document image super-resolution method based on Real-ESRGAN improvement according to claim 6, characterized in that: The edge enhancement module adopts a Scharr filter and uses a central differential convolution with a Laplace kernel, and has horizontal differential convolution, vertical differential convolution and central differential convolution.

8. The document image super-resolution method based on Real-ESRGAN improvement according to claim 6, characterized in that: The content of the detail enhancement module includes: performing an average pooling operation on the input and then performing a 1*1 convolution operation to generate a weight matrix, and controlling the size of the output feature map to be the same as the input feature map by padding the size; performing convolution on each channel of the input feature to extract features, and finally multiplying the weights with the features extracted by the convolution to obtain the weighted receptive field spatial features.

9. The document image super-resolution method based on Real-ESRGAN improvement according to claim 6, characterized in that: The edge enhancement module and the detail enhancement module are connected in a densely connected manner to form the edge detail enhancement module.

10. A document image super-resolution system based on Real-ESRGAN, characterized in that: A document image super-resolution method based on Real-ESRGAN improvement as claimed in any one of claims 1 to 9 can be implemented; the system comprises: Training set construction module: used to determine the images used for training, pre-process each image, obtain high-resolution images and low-resolution images for model training, and form a training set; Model design module: used to build the model used for subsequent model training. The model replaces the residual dense connection module in the Real-ESRGAN model with the edge detail enhancement module to obtain the Real-ESRGAN model with edge detail enhancement. Document image super-resolution model acquisition module: uses the training set constructed by the training set construction module to train the edge detail enhanced Real-ESRGAN model obtained by the model design module to obtain a document image super-resolution model; Document image super-resolution module: used to determine the parameters of the document image super-resolution model, and use the document image super-resolution model to perform super-resolution processing on the document image to be processed.

Images