A super-resolution method for text images

Abstract

The present invention belongs to the technical field of image processing, and discloses a super-resolution method for text images, including: generating a text super-resolution dataset, identifying and converting the target text area in the image into a rectangle for input into the super-resolution network, extracting the features of the image to be super-resolved to obtain a feature map X0, generating an attention map A using a text prior extraction module, performing dimensionality increase on the feature map X0 and the attention map A through multiple residual blocks with attention mechanisms to obtain a feature map Xn, performing pixel and channel recombination on the feature map Xn to generate a super-resolved image SR, calculating the text sequence loss of the SR image l ctc and the inter-pixel loss l 2 , and obtaining the total loss l . By iteratively optimizing the network parameters and evaluating the model performance, inputting the SR image into a text recognition model, obtaining the predicted text, splicing it to the original image after inverse transformation, enhancing the visual effect, and effectively solving the problem that the traditional super-resolution model has limited improvement effect on the quality of text images or even produces a negative optimization effect

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a super-resolution method for text images. Background Art

[0002] With the rapid development of information technology, text images are increasingly used in daily life and work. However, due to the limitations of imaging equipment or the loss of image transmission, the acquired text images often have low resolution, making the text content difficult to recognize. Therefore, how to effectively improve the resolution of text images has become an urgent problem to be solved in the field of image processing.

[0003] Traditional super-resolution methods are mainly based on algorithms such as interpolation and reconstruction. Although the interpolation method is simple and easy to use, it is easy to lose high-frequency information when processing images, resulting in blurred edges and loss of details in the generated images. The reconstruction method requires the use of information from multiple low-resolution images to improve the resolution through fusion and denoising, but the process is complex, the amount of calculation is large, and it is difficult to apply to real-time processing scenarios. The traditional super-resolution model has limited effect on improving the quality of text images and even produces a negative optimization effect.

[0004] Therefore, a super-resolution method for text images is proposed to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a super-resolution method for text images to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a super-resolution method for text images, comprising the following steps:

[0007] S1: Generate text super-resolution dataset;

[0008] S2: using a first preset algorithm to obtain a polygon of a target text area in the image, and using a perspective transformation to convert the polygon into a rectangle for input into a text super-resolution network;

[0009] S3: Extract features of the image LR that needs super-resolution and obtain a feature map X0;

[0010] S4: Use the text prior extraction module to perform text prediction on the image LR and generate an attention map A;

[0011] S5: The feature map X0 obtained in S3 and the attention map A obtained in S4 are subjected to channel dimension upgrading through n residual blocks with channel and spatial attention mechanisms connected in series, during which feature maps X1, ..., Xn are generated in sequence;

[0012] S6: Perform pixel reorganization and channel reorganization on the dimension-upgraded feature map Xn to generate a super-resolution image SR;

[0013] S7: Pass the SR image through the text recognition module to obtain the predicted probability of each character at each time step pred ,calculate pred and the ground-truth text label gt The text sequence loss CTCLoss between l ctc ; Use the preset function to calculate the pixel loss between the SR image and the LR image to obtain l 2 ,calculate l ctc and l 2 The weighted sum l , the optimal model parameters of the network are iteratively solved using the second preset algorithm, and finally, the super-resolution quantitative indicators PSNR and SSIM and the recognition accuracy of the text are calculated to evaluate the model performance;

[0014] S8: Input the SR image into the text recognition model to obtain the predicted text, perform inverse transformation according to the perspective transformation parameters used in S2 to restore the SR image to the original polygon shape, and re-stitch it into the original image to enhance the visual effect.

[0015] Preferably, the S1 specifically comprises the steps of:

[0016] According to the preset super-resolution magnification, interpolation changes are made to the text recognition dataset to generate a text super-resolution dataset.

[0017] Preferably, the S3 specifically comprises the steps of:

[0018] The image LR that requires super-resolution is input into the convolutional neural network for feature extraction to obtain the feature map X0.

[0019] Preferably, the S4 specifically comprises the steps of:

[0020] S41: inputting the image LR requiring super-resolution into a text prior extraction module for text prediction;

[0021] S42: Generate t attention maps according to the predicted sequence length t, and add up the attention maps to obtain the attention map A of the entire sequence.

[0022] Preferably, the S6 specifically comprises the steps of:

[0023] S61: preprocess the feature map Xn using pixel reorganization Pixel Shuffle;

[0024] S62: Then input the upsampled convolution layer to perform channel reorganization to generate a super-resolution image SR.

[0025] Preferably, the text super-resolution dataset comprises a text super-resolution dataset I of high-resolution scale images. HR Text super-resolution dataset for low-resolution and scaled images I LR。

[0026] Preferably, the first preset algorithm includes a DBNet text detection algorithm.

[0027] Preferably, the preprocessing includes channel dimension reduction and pixel dilation.

[0028] Preferably, the preset function includes a mean square loss function MSELoss.

[0029] Preferably, the second preset algorithm includes an Adam optimization algorithm.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention improves the text recognition accuracy by improving the resolution of low-quality text images. By introducing channel and spatial attention mechanisms and text prior feature extraction on the basis of the traditional single image super-resolution model, the model's responsiveness to text features in the image is improved. The text-aware loss function is used to enable the model training process to effectively train the text super-resolution network, which effectively solves the problem that the traditional super-resolution model has limited effect on improving the quality of text images and even produces a negative optimization effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the residual block of the original TSRN of the present invention;

[0033] Figure 2 Schematic diagram of residual block adding channel and spatial attention to the present invention;

[0034] Figure 3 Schematic diagram of the SAR network and its 2D attention of the present invention;

[0035] Figure 4 This is a schematic diagram of SAR attention visualization of the present invention;

[0036] Figure 5 Schematic diagram of a text prior embedding method of the present invention;

[0037] Figure 6 The network structure diagram after adding text perception loss to the present invention;

[0038] Figure 7 Schematic diagram of the influence of text perception loss weight of the present invention; DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] See also Figure 1-7 As shown, the present invention provides the following technical solutions:

[0041] A super-resolution method for text images comprises the following steps:

[0042] S1: According to the preset super-resolution magnification, interpolation changes are performed on the text recognition dataset to generate a text super-resolution dataset, wherein the text super-resolution dataset includes a text super-resolution dataset I of a high-resolution scale image. HR Text super-resolution dataset for low-resolution and scaled images I LR ;

[0043] S2: Use the DBNet text detection algorithm to obtain the polygon of the target text area in the image, and use perspective transformation to convert the polygon into a rectangle to input into the text super-resolution network;

[0044] S3: Input the image LR that needs super-resolution into the convolutional neural network for feature extraction to obtain the feature map X0;

[0045] S4: Input the image LR that needs super-resolution into the text prior extraction module for text prediction;

[0046] S5: Generate t attention maps according to the predicted sequence length t, and add up the attention maps to obtain the attention map A of the entire sequence;

[0047] S6: The feature map X0 obtained in S3 and the attention map A obtained in S4 are subjected to channel dimension upgrading through n residual blocks with channel and spatial attention mechanisms connected in series, during which feature maps X1, ..., Xn are generated in sequence;

[0048] S7: Use pixel shuffle to perform channel dimension reduction and pixel expansion on the feature map Xn;

[0049] S8: Then input the up-sampled convolutional layer to perform channel reorganization to generate a super-resolution image SR;

[0050] S9: Pass the SR image through the text recognition module to obtain the predicted probability of each character at each time step pred ,calculate predand the ground-truth text label gt The text sequence loss CTCLoss between l ctc ; Use the mean square loss function MSELoss to calculate the pixel loss between the SR image and the LR image l 2 ,calculate l ctc and l 2 The weighted sum l , the Adam optimization algorithm is used to iteratively solve the optimal model parameters of the network. Finally, the super-resolution quantitative indicators PSNR and SSIM as well as the recognition accuracy of the text are calculated to evaluate the model performance;

[0051] S10: Input the SR image into the text recognition model to obtain the predicted text, perform inverse transformation according to the perspective transformation parameters used in S2 to restore the SR image to the original polygon shape, and re-stitch it into the original image to enhance the visual effect.

[0052] In addition, in the present invention, if Figure 1 As shown in the figure, for the feature map from the convolutional layer, the original TSRN treats each channel and each position in space equally. For the complex and changeable text images in the text super-resolution task, the TSRN network ignores the features from different channels, has weak discriminative learning ability across feature channels, fails to effectively distinguish between text areas and background areas, and has difficulty focusing on important features of the image and suppressing unnecessary regional responses.

[0053] In order to cope with complex and changeable text images, flexibly treat features from different channels, and enhance the discriminative learning ability across feature channels, this patent improves the residual block based on TSRN, introduces CBAM channels and spatial attention modules, and applies different attention to different locations and channels in the space to help the network focus on capturing the text area.

[0054] And in the present invention, the residual block after adding the channel and spatial attention mechanism is as follows Figure 2As shown. For the feature map from the convolutional layer, the channel attention mechanism (Chanel Attention Module) compresses the feature map in the spatial dimension, performs average pooling (Average Pooling) and maximum pooling (Max Pooling) on ​​the feature map, and obtains a one-dimensional vector, which is the channel attention of the feature map, and then pays attention to the feature map by multiplying it with the original feature map. Average pooling and maximum pooling can be used to aggregate the spatial information of the feature map, send it to a shared network, compress the spatial dimension of the input feature map, and sum it element by element to produce a channel attention map. Average pooling has feedback for every pixel on the feature map, while when performing gradient back propagation calculation for maximum pooling, only the place with the largest response in the feature map has gradient feedback. For a single image, channel attention focuses on which content on this image is important. The channel attention mechanism can be expressed as:

[0055] ;

[0056] Where MLP stands for multi-layer perceptron, W1 and W0 are the weights of the multi-layer perceptron, and represents the activation function Sigmoid.

[0057] The Spatial Attention Module performs channel compression on the feature maps from the channel attention module, that is, it performs average pooling and maximum pooling in the channel dimension to obtain two single-channel feature maps. Then the extracted feature maps are merged to obtain a dual-channel feature map, and then the feature map is fused into a single-channel feature map through the convolution layer to obtain the spatial attention map, and finally the attention is applied to the feature map from the previous module. The spatial attention mechanism can be expressed as:

[0058] ;

[0059] Where f7x7 represents the convolution layer operation with a convolution kernel of 7x7.

[0060] In addition, most existing super-resolution methods treat scene text images as natural scene images for super-resolution processing, ignoring the important category information brought by the text content in the image. TSRN improves the super-resolution network by introducing recurrent neural networks into the super-resolution network. However, the recurrent neural network in the residual block mainly extracts sequence features, and TSRN still finds it difficult to distinguish the specific characters contained therein.

[0061] In order to enable the network to accurately restore the text information in the LR image, this patent introduces text prior information into the network to guide the super-resolution process. It is difficult to accurately segment the text area in low-quality text images, and it is also not easy to locate the key points of the text. The attention map (Attention Map), an intermediate product of the text recognition method based on the attention mechanism, represents the importance of each position in the space or sequence to the decoding result, contains the position information of the characters, and can be used as a representation of text prior information. Therefore, this patent uses the text recognition module SAR based on two-dimensional attention as a text prior information extraction module.

[0062] The network structure and attention mechanism of SAR are as follows Figure 3 shown.

[0063] And, regarding the above-mentioned SAR module:

[0064] The SAR module first uses CNN to extract a two-dimensional feature map, then converts the feature map into a one-dimensional sequence and sends it to LSTM to encode the global feature (Holistic Feature), which is used as the initial state of the decoder to start decoding step by step. Finally, the attention mechanism is applied with the decoding result as the query and the feature map as the key and value to obtain the weighted feature map, so that the character of this step can be predicted. Since the feature map retains the spatial information of the image and uses spatial attention, which contains the information about the distribution of characters in space, SAR is suitable for processing irregular text.

[0065] Since each attention map generated during the SAR recognition process represents the attention of a single character to each pixel in space, it is necessary to add the attention of each time step to obtain a spatial attention map representing the entire text sequence. The attention map can reflect the spatial distribution information of the text to a certain extent. Figure 4 The visualization effect of SAR attention is shown, from which we can see that it contains the location information of the text.

[0066] Since the size of the attention map does not necessarily match the LR input size of the super-resolution network, the feature map needs to be sampled and mapped to the LR input size by bicubic interpolation before it can participate in the forward propagation process of the super-resolution network. Since bicubic interpolation sampling is a differentiable operation, it can ensure that the SAR network with unfrozen weights can be optimized together with the super-resolution network during training.

[0067] After extracting the prior information, the next step is to embed the prior information into the super-resolution module. In order to make the entire backbone network guided by the text prior information, the prior information is input into each residual block. In the convolution, 2D Attention (TP) represents the attention from the SAR module, and TP stands for Text Prior, i.e. text prior.

[0068] In addition to LR images and HR images, the text image super-resolution dataset also provides corresponding text labels, and the perceptual loss of TSRN does not use text labels for loss calculation. Therefore, based on TSRN, this patent uses HR images to extract text prior information during the model training process, or directly uses text labels as priors, and designs a loss function based on priors to help the network learn more favorable representations.

[0069] In order to perceive the text in the SR image, a pre-trained text recognition network is added after the super-resolution network to perceive the text semantics of the SR image. The network parameters are frozen. It accepts the SR image as input and outputs the predicted probability of each character at each time step. Then, according to different loss functions, the loss between the output and the encoded text label is calculated. The network structure after adding text perception loss is as follows: Figure 6 shown.

[0070] In addition, in the present invention, during the back propagation process, the gradients related to the text perception loss are accumulated in the super-resolution network. When optimizing the network, in addition to improving SSIM and PSNR, the goal is to improve the SR image recognition accuracy. The following are two different formulas for text perception loss:

[0071]

[0072]

[0073] Where pred represents the character probability map output by inputting the SR image into the perception module. Where the flatten operation flattens the vector in the time step dimension, and the flattened vector is used to calculate the cross entropy loss.

[0074] Similar to other super-resolution tasks, constructing SR images is inseparable from inter-pixel losses such as MSELoss or MAELoss. The text-aware loss function cannot be used alone, but as an auxiliary part of the overall loss function. Therefore, the overall loss function is defined as:

[0075]

[0076] In the formula, is the inter-pixel loss, and are the text-aware loss and its weight respectively.

[0077] Due to differences in networks and training data sets, different recognition models have different preferences for input images. CRNN is a CTC-based method that uses CTCLoss; Aster and Moran are based on a sequence-to-sequence model and use sequence cross entropy loss.

[0078] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A super-resolution method for text images, characterized by: The steps include: S1: Generate text super-resolution dataset; S2: using a first preset algorithm to obtain a polygon of a target text area in the image, and using a perspective transformation to convert the polygon into a rectangle for input into a text super-resolution network; S3: Extract features of the image LR that needs super-resolution and obtain a feature map X0; S4: Use the text prior extraction module to perform text prediction on the image LR and generate an attention map A; S5: The feature map X0 obtained in S3 and the attention map A obtained in S4 are subjected to channel dimension upgrading through n serially connected residual blocks with channel and spatial attention mechanisms, during which feature maps X1, ..., Xn are generated in sequence. The spatial attention mechanism is for the feature map from the convolution layer. The channel attention mechanism compresses the feature map in the spatial dimension, performs average pooling and maximum pooling on the feature map respectively, obtains a one-dimensional vector as the channel attention of the feature map, and then applies attention to the feature map by multiplying it with the original feature map; S6: Pixel reorganization and channel reorganization are performed on the feature map Xn after dimension upgrading to generate a super-resolution image SR; S7: Pass the SR image through the text recognition module to obtain the predicted probability of each character at each time step pred ,calculate pred and the ground-truth text label gt The text sequence loss CTCLoss between l ctc ; Use the preset function to calculate the pixel loss between the SR image and the LR image to obtain l 2 ,calculate l ctc and l 2 The weighted sum l , the preset function includes the mean square loss function MSELoss, and the optimal model parameters of the network are iteratively solved using the second preset algorithm, the second preset algorithm includes the Adam optimization algorithm. Finally, the super-resolution quantitative indicators PSNR and SSIM and the recognition accuracy of the text are calculated to evaluate the model performance; S8: Input the SR image into the text recognition model to obtain the predicted text, perform an inverse transformation according to the perspective transformation parameters used in S2 to restore the SR image to the original polygon shape, and re-stitch it into the original image.

2. The super-resolution method for text images according to claim 1, characterized in that: The S1 specifically comprises the steps of: According to the preset super-resolution magnification, interpolation changes are made to the text recognition dataset to generate a text super-resolution dataset.

3. The super-resolution method for text images according to claim 2, characterized in that: The S3 specifically includes the steps of: The image LR that requires super-resolution is input into the convolutional neural network for feature extraction to obtain the feature map X0.

4. The super-resolution method for text images according to claim 3, characterized in that: The S4 specifically comprises the steps of: S41: inputting the image LR requiring super-resolution into a text prior extraction module for text prediction; S42: Generate t attention maps according to the predicted sequence length t, and add up the attention maps to obtain the attention map A of the entire sequence.

5. The text image super-resolution method according to claim 4, characterized in that: The S6 specifically comprises the steps of: S61: preprocess the feature map Xn using pixel reorganization Pixel Shuffle; S62: Then input the upsampled convolution layer to perform channel reorganization to generate a super-resolution image SR.

6. The super-resolution method for text images according to claim 5, characterized in that: The text super-resolution dataset includes a text super-resolution dataset I of high-resolution scale images. HR Text super-resolution dataset for low-resolution and scaled images I LR。 7. The super-resolution method for text images according to claim 6, characterized in that: The first preset algorithm includes a DBNet text detection algorithm.

8. The super-resolution method for text images according to claim 7, characterized in that: The preprocessing includes channel dimension reduction and pixel expansion.

9. The super-resolution method for text images according to claim 8, characterized in that: The preset function includes the mean square loss function MSELoss.

10. The text image super-resolution method according to claim 9, characterized in that: The second preset algorithm includes the Adam optimization algorithm.

Images