Text prior guided scene text image super-resolution method

Abstract

The super-resolution of the scene text image is the key to enhance the resolution and visual quality of the low-resolution scene text image, and the text recognition capability can be remarkably enhanced. According to an existing STISR method, a text image is usually regarded as a natural scene image, and inherent unique classification information of a text is ignored. A new framework is provided, classified text priori is integrated into STISR model training, and the framework is called as a two-stage text priori guided super-resolution (TTPSR) framework. A character probability sequence extracted from a text recognition model is used as text prior, and robust classification guidance is provided for reconstruction of a high-resolution text image. The TTPSR framework initially applies a parallel context attention network to restore low resolution images without text priori. And then a multi-head cross attention mechanism is utilized to fuse the text prior and the image features so as to guide the generation of a final high-resolution text image. The TTPSR enhances the visual quality of a scene text image, and is superior to the existing STISR method in the aspect of text recognition precision.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and mainly relates to text image super-resolution, and specifically is a scene text image super-resolution method based on text prior guidance, which is used to combine the text prior information of a low-resolution image and the low-resolution image itself to generate a more accurate high-resolution text image. Background Art

[0002] Text in pictures is an important channel for us to obtain daily information, and we extract and understand it according to different needs. However, during the shooting process, scene text images often suffer from quality degradation, such as reduced resolution and blurred structure. These problems seriously affect the results of advanced recognition tasks such as scene text detection, optical character recognition (OCR) and scene text recognition. Therefore, it is particularly important to improve the resolution of scene text images and enhance their visual quality.

[0003] In recent years, in order to improve the quality of text images, researchers have developed many scene text image super-resolution (STISR) techniques based on deep learning, and these methods have achieved remarkable achievements. Using datasets of degraded text and original text image pairs, deep convolutional neural networks (CNNs) can be trained. These networks can extract a variety of prior knowledge from the data due to their powerful learning ability and show excellent performance. Dong et al. proposed the SRCNN model to reconstruct high-resolution images. It is the first deep learning method that surpasses traditional methods and has made breakthrough progress in image reconstruction. Subsequently, VDSR proposed to use a deep network with residual learning to capture high-frequency details, and FSRCNN improved computational efficiency by processing in a low-resolution space before upscaling. The EDSR model achieves state-of-the-art performance by removing batch normalization layers and using residual expansion in a very deep architecture. SRGAN first introduced a generative adversarial network to produce visually attractive images through adversarial training. SR3 uses a denoising diffusion probability model to iteratively refine image resolution and generate high-fidelity images. Zhang proposed a text image super-resolution reconstruction method based on diffusion model, which combines the image diffusion model (IDM), text diffusion model (TDM) and hybrid multimodal module (MoM) to restore high-fidelity and real text images from degraded low-resolution input. Noguchi proposed a text super-resolution method based on text conditional diffusion model, which significantly improved the performance of text image super-resolution model by generating low-resolution and high-resolution text image pairs.

[0004] However, the above methods cannot effectively guarantee the accuracy of low-resolution image text, and ignore the text information of the scene text image itself. The current representative text prior-based methods, TPGSR and TATT models, use existing recognition models as text semantic extraction modules. Since the details and clarity of low-resolution images are limited, and most of the existing recognition models are designed for high-quality, high-resolution images, directly using the recognition model as the TPG module has certain limitations in understanding image text information. In the part of merging text priors and image features, TPGSR uses simple deconvolution to allow text priors and image features to interact, which limits the guidance of text priors on the text reconstruction process. TATT uses a TP module based on a transformer structure and adopts global attention operations to capture the correlation between them. Due to the use of the transformer structure, the computational complexity of the network increases exponentially with the length of the text in the image, and there are problems of large models and high computational complexity.

[0005] In summary, the existing super-resolution methods based on deep learning ignore the text information contained in the image itself, and while pursuing high-precision performance, they lead to problems such as large models and low generation efficiency. Summary of the invention

[0006] In view of the shortcomings and problems existing in the prior art, the present invention proposes a method that can integrate text information into a super-resolution process and can guide the super-resolution process, so as to generate more accurate high-resolution images in text information reconstruction.

[0007] The present invention is a text prior-guided image super-resolution method, and its main process comprises the following steps:

[0008] (1) Establish a scene text image dataset: collect low-resolution images and corresponding high-resolution images. Each low-resolution image corresponds to a high-resolution image to form an image pair. The dataset is divided into a training set, a validation set, and a test set.

[0009] (2) Construct an unguided image reconstruction network: Construct a low-resolution reconstruction network based on a parallel contextual attention mechanism. The network adopts a dual-branch structure to construct visual feature dependencies in the vertical and horizontal directions in parallel, and then uses channel attention to adaptively weight the previously obtained features. Then, the orthogonal texture perception module is used to obtain the output features after texture perception enhancement, and finally the upsampling layer is used to obtain the initial restored image.

[0010] (3) Constructing a text prior-guided image reconstruction network: A low-resolution reconstruction network based on text image prior guidance is constructed. The network uses the preliminary restored image obtained in the previous stage as input. The text prior generator first generates a text prior probability sequence. This sequence is a length sequence composed of classification probability vectors of size |A|, with a length of 37, representing a character set consisting of 0 to 9, A to Z, and blank classes. Subsequently, the text prior is passed to the STP interpreter, combined with the image features to generate TP features, and then passed to the text-guided super-resolution module together with the preliminary restored image. This module includes five text-guided blocks, which gradually fuse image features and text features, and then pass through the final pixel reconstruction layer to obtain the final high-resolution image.

[0011] The present invention has the following advantages compared with the prior art:

[0012] (1) Improved reconstruction accuracy of text content in high-resolution images: Compared with existing super-resolution methods, the present invention constructs a text prior-guided super-resolution network to add the text semantic information of the image to the reconstruction process, providing semantic guidance for the model, so that the model can learn the semantic information of the image, thereby being able to more accurately reconstruct the text content in the image, thereby improving the super-resolution effect.

[0013] (2) Accelerated reconstruction efficiency of super-resolution images: The present invention uses cross-attention to fuse text prior and image feature information. Compared with the traditional transformer structure, the self-attention process of the input image is omitted, so that the model of the present invention can be reduced in terms of parameter quantity. At the same time, the entire process does not undergo iterative refinement. Compared with the original multiple iterative process, it is more efficient in terms of generation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is the overall flow chart of the present invention

[0015] Figure 2 The STP module of the present invention

[0016] Figure 3 The final image result comparison of the present invention is DETAILED DESCRIPTION

[0017] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0018] Combined with Figure 1 The implementation method of the present invention is described in detail.

[0019] The overall flow chart of the present invention is as follows: Figure 1As shown in the figure, the input is a low-resolution text image (LR). The network is mainly divided into two stages. In the first stage processing path, the low-resolution image is input into the No-Guided SRModule to directly generate a super-resolution (SR) image. In the second stage processing path, first, the low-resolution image is extracted through the convolution layer (Conv) to extract image features f I ∈Rh×w×c. Then the image features fi and TP features are input into the text prior guided super-resolution module (TP-Guided SR Module) to generate high-resolution output.

[0020] The two-stage process also includes a text prior module (TP Module), which uses the SR input image to generate a text prior. The text prior generator (TP Generator) generates a text prior probability sequence based on the input SR image. This process can be expressed as fp = TP(X). fp is an l-length sequence consisting of classification probability vectors of size |A|. A represents the character set consisting of '0' to '9', 'A' to 'z' and a blank class (37 in total). Subsequently, the text prior is passed to the STP Interpreter and combined with the image features to generate the TP feature f TM (TP feature), which is used to guide the final super-resolution text restoration.

[0021] Finally, TP is mapped to f TM and image features f I Passed to the text-guided super-resolution module. This module consists of 5 text-first guided blocks (tpgb) that gradually fuse f TM and f I , and a final Pixel-Shuffle layer to increase the resolution. The 5 tpgb first adds f TM and f I The high-resolution image features are merged and then reconstructed using a Sequential-Recurrent Block (SRB). The output of this module is the final super-resolution (SR) text image.

[0022] One of the key processes of text prior guided image restoration is the generation of text prior. The purpose of the TP Generator module is to generate text prior to guide the subsequent image restoration process. The accuracy of the text prior directly affects the correctness of the semantic guidance. Since the details and clarity in low-resolution images are limited, most recognition algorithms are designed for high-quality, high-resolution images. Directly using the recognition model as a TPG module has certain limitations in understanding the image text information. Therefore, we designed a two-stage super-resolution module. In the first stage, the unguided super-resolution module performs preliminary super-resolution processing on the low-resolution image to generate a preliminary high-resolution image. Then, in the second stage, the generated high-resolution image is used as input to generate a more accurate text prior through the TP Generator.

[0023] Finally, the image features and TP features are fused and processed by the TP-GuidedSRModule to generate high-quality super-resolution text image output (SR output). This design effectively improves the accuracy of text priors, thereby enhancing the correctness and effect of semantic guidance in the image restoration process.

[0024] Combined with Figure 2 The implementation method of the STP module of the present invention is described in detail.

[0025] Another key structure in the architecture of text prior guided image restoration is the TP interpreter. The TP interpreter applies semantic guidance to the image at relevant spatial locations. Its main function is to integrate text priors and image features, associate and understand the relationship between images and text, and achieve richer data interpretation. A more direct multimodal fusion method is to use a Transformer-based algorithm, adopt an Encoder-Decoder structure, and use self-attention mechanism and cross-attention mechanism to achieve the fusion of text priors and image features. However, due to the limitations of the TPG module, the obtained text probability prior contains a lot of noise and irrelevant information, and the encoder structure using the self-attention mechanism depends on the quality of the contextual information. The self-attention mechanism may be distracted to unimportant parts and cannot accurately capture important patterns or relationships.

[0026] Therefore, we designed an STP interpreter based only on the multi-head cross-attention mechanism to obtain the global correlation of text priors and image features. Compared with the Encoder and Decoder structure of Transformer, our TP interpreter has fewer parameters and computational complexity, and can provide more effective text semantic guidance.

[0027] In the STP interpreter, the input consists of the text prior f T and image features f I First, we have f I Adding recurrent position encoding (RPE), we get f I' , this encoding can effectively capture the dependencies of image feature sequences in the horizontal direction, thus helping the model to more accurately locate text semantic features during the cross-attention process. This position encoding has the same shape as the image feature, is learnable, and is specifically used to encode the dependencies of adjacent contexts in the sequence. Next, the image feature f after position encoding is added I' The output of TP Generator T In the STP module, we apply a multi-head cross attention (MCA) layer, which is used in f T and f I' Specifically, we first convert f T and f I' It is divided into n subgroups in the channel dimension, and then a criss-cross attention operation is performed on each group. For the i-th group, the criss-cross attention operation C is and

[0028] For the i-th group, the cross attention operation C is and On, where and Represents f T and f I' In this process, we use three sets of linear projection parameters To convert the corresponding features, and then normalize them through the Softmax operation.

[0029] The specific operation is as follows: First, map each set of features to the query, key, and value spaces respectively:

[0030]

[0031] Next, the dot product similarity between the query and the key is calculated and normalized by a Softmax operation to obtain a weight distribution:

[0032]

[0033] where d k is the dimension of the key vector, used to scale the dot product result.

[0034] After completing the cross-attention operation for each subgroup, we integrate the results of all subgroups through channel concatenation and process them through linear projection WOWO to form the output of MCA:

[0035] MCA(f T ,f′ I )=W o [Attention(Q1,K1,V1);Attention(Q2,K2,V2)…;Attention(Q n ,K n ,V n )],

[0036] Where [;] indicates a connection operation.

[0037] This output is then fed into a feed-forward neural network (FFN) for further feature refinement and finally reshaped into a text-position map f TM Through this MCA mechanism, the text prior f T It can effectively interact with the image features f'I and associate each semantic element with a spatial location. TM In the proposed method, semantically meaningful regions are enhanced, which helps to condition image features when performing semantically specific text reconstruction.

[0038] This design not only enhances the expressive power of feature mapping, but also improves the capture and reconstruction of image details, thereby improving the performance and robustness of the model as a whole. Through the MCA layer, the interaction between text priors and image features is optimized, making it possible to more accurately capture and utilize semantic information in the process of high-resolution image restoration.

[0039] Combined with Figure 3 The final image result comparison of the present invention is described in detail.

[0040] like Figure 3 As shown in , we visualize the results of different current STISR methods on the text image super-resolution task. These methods include BICUBIC, TPGSR, TPGSR-3, TATT, TTPSR, and high-resolution reference images (HR). TPGSR and TATT are still unstable when restoring spatially deformed images under TP guidance. Figure 3 As shown, compared with current scene image super-resolution methods, our proposed network performs better in recovering text semantics in samples of all cases with clearer character structures.

[0041] In summary, the super-resolution method based on text prior guidance proposed in this paper adds the text semantic information of the image to the reconstruction process, provides semantic guidance for the model, enables the model to learn the semantic information of the image, and thus can reconstruct the text content in the image more accurately, thereby improving the final super-resolution image text restoration effect.

Claims

1. A super-resolution method based on text priors, characterized in that: The following steps are involved: S1: Data preprocessing: collect low-resolution images and corresponding high-resolution images. Each low-resolution image corresponds to a high-resolution image to form an image pair. The data set is divided into training set, validation set and test set. S2: Construct an unguided image reconstruction network and a low-resolution reconstruction network based on a parallel contextual attention mechanism. The network adopts a dual-branch structure to construct visual feature dependencies in the vertical and horizontal directions in parallel, and then uses channel attention to adaptively weight the previously obtained features. Then, the orthogonal texture perception module is used to obtain the output features after texture perception enhancement, and finally the upsampling layer is used to obtain the preliminary restored image. S3: Construct an image reconstruction network guided by text priors. Construct a low-resolution reconstruction network guided by text image priors. The network uses the preliminary restored image obtained in the previous stage as input. First, a text prior probability sequence is generated through a text prior generator. This sequence is a length sequence composed of classification probability vectors of size |A|, with a length of 37, representing a character set consisting of 0 to 9, A to Z, and blank classes. Subsequently, the text prior is passed to the STP interpreter, combined with the image features to generate TP features, and then passed to the text-guided super-resolution module together with the preliminary restored image. This module includes five text-guided blocks, which gradually fuse image features and text features, and then pass through the final pixel reconstruction layer to obtain the final high-resolution image. The specific operation steps of the STP module are as follows: In the STP interpreter, the input consists of the text prior f T and image features f I First, we have f I Adding recurrent position encoding (RPE), we get f I′ , this encoding can effectively capture the dependencies of image feature sequences in the horizontal direction, thus helping the model to more accurately locate text semantic features during the cross-attention process. This position encoding has the same shape as the image feature, is learnable, and is specifically used to encode the dependencies of adjacent contexts in the sequence. Next, the image feature f after position encoding is added I′ The output of TP Generator is T In the STP module, we apply a multi-head cross attention (MCA) layer, which is used in f T and f I′ Specifically, we first convert f T and f I′ It is divided into n subgroups in the channel dimension, and then a criss-cross attention operation is performed on each group. For the i-th group, the criss-cross attention operation C is and On, where and Represents f T and f I′ In this process, we use three sets of linear projection parameters To convert the corresponding features, and then normalize them through the Softmax operation. The specific operation is as follows: First, map each set of features to the query, key, and value spaces respectively: Next, the dot product similarity between the query and the key is calculated and normalized by a Softmax operation to obtain a weight distribution: where d k is the dimension of the key vector, used to scale the dot product result. After completing the cross-attention operation for each subgroup, we integrate the results of all subgroups through channel concatenation and process them through linear projection WOWO to form the output of MCA: MCA(f T ,f I ′)=W O [Attention(Q1,K1,V1);Attention(Q2,K2,V2)...;Attention(Q n ,K n ,V n )], Where [;] indicates a connection operation. This output is then fed into a feed-forward neural network (FFN) for further feature refinement and finally reshaped into a text-position map f TM Through this MCA mechanism, the text prior f T Can effectively combine with image features f I ′ interacts with each semantic element and associates it with a spatial location. TM In the proposed method, semantically meaningful regions are enhanced, which helps to condition image features when performing semantically specific text reconstruction. This design not only enhances the expressive power of feature mapping, but also improves the capture and reconstruction of image details, thereby improving the performance and robustness of the model as a whole. Through the MCA layer, the interaction between text priors and image features is optimized, making it possible to more accurately capture and utilize semantic information in the process of high-resolution image restoration.

Images