Image Super-Resolution Method and System Based on Semantic Reasoning and Cross-Convolution

Abstract

This invention belongs to the field of image super-resolution technology and proposes an image super-resolution method and system based on semantic reasoning and cross-convolution. The method includes: acquiring a low-resolution scene text image; preprocessing the image to obtain shallow features and the original probability sequence; inputting the acquired shallow features and the original probability sequence into a model based on semantic reasoning and multi-scale cross-convolution to obtain a super-resolution image; the model includes a text semantic reasoning module, a reconstruction module, and a text structure preservation module connected in sequence; the text semantic reasoning module is a Transformer-based encoder-decoder structure used to generate text prior features with high character readability. The encoder of this invention combines a self-attention mechanism and a Transformer-based language model to achieve bi-branch semantic reasoning. The improved encoder adaptively learns highly accurate semantic features and exhibits good generalization ability on scene images with learnable language rules.

Description

Technical Field

[0001] This invention belongs to the field of image super-resolution technology, and particularly relates to an image super-resolution system based on semantic reasoning and cross-convolution. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Scene text images contain a large amount of text information required for computer vision tasks such as scene text recognition. However, scene text images often degrade to varying degrees during the imaging process due to limitations such as occlusion, equipment, and lighting conditions. Many studies have proposed applying image super-resolution techniques to recover degraded scene text images as preprocessing for subsequent scene text recognition tasks. Image super-resolution involves upscaling low-resolution (LR) images to high-resolution (HR) images using specific algorithms.

[0004] Existing scene text image super-resolution methods can be broadly classified into two categories: early general methods and text-oriented methods. Early methods directly used neural network models proposed in the field of scene image super-resolution to recover visually high-quality images. However, these methods have limitations: (1) they learn pixel-level mapping relationships, ignoring text-specific attributes; (2) when experiments are conducted on synthetic data, the trained models show limited generalization ability on text images in real-world scenes.

[0005] To overcome the limitations of earlier general-purpose methods, text-oriented methods and the real-world scene dataset TextZoom have been proposed. Unlike earlier general-purpose methods, text-oriented methods utilize text attributes, which helps enhance the model's representation ability in text regions. Current text-oriented methods have two research directions. One is the modification of the main architecture for modeling contextual dependencies between characters. The introduction of BLSTM enables the model to capture sequence information in both horizontal and vertical directions, which makes capturing sequence information in arbitrary directions challenging. With the excellent performance of the Transformer model in modeling long-distance dependencies, Transformer has been successfully applied to scene text image super-resolution models, further improving the recognition accuracy of reconstructed images. The other is the focus on the text itself, such as text location, text content, and structural information. Existing text-oriented methods focus on the characteristics of text content. In recent semantic cue-guided methods, global attention mechanisms have been introduced into the text prior interpreter to capture the correlation between text priors and image features.

[0006] While these methods have made significant progress in image quality, they still have the following gaps and shortcomings when faced with extremely blurry images: (1) Recent Transformer-based text prior interpreters are insufficient to capture contextual relevance to recover extremely blurry images, which largely depends on the blurriness of the original image; (2) Text in images of real-world scenes has practical meaning, and text with practical meaning has learnable language rules. Past methods have not made full use of this characteristic; (3) When recovering blurry images, existing methods focus on improving the accuracy of text priors, ignoring the learning of text structural similarity. The reconstructed image faces the problem of interlaced character structures, which affects the further improvement of recognition accuracy. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, this invention provides a text image super-resolution method and system based on a multi-scale cross-convolution model.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of this invention provides an image super-resolution method based on semantic reasoning and cross-convolution, comprising:

[0010] Acquire the text image of the low-resolution scene to be upgraded;

[0011] The low-resolution scene text image is preprocessed to obtain the shallow features of the scene text image and the original probability sequence of the characters.

[0012] The acquired shallow features and the original probability sequence are input into a semantic reasoning and multi-scale cross-convolution model to obtain a super-resolution image.

[0013] The semantic reasoning and multi-scale cross-convolution model includes a text semantic reasoning module, a reconstruction module, and a text structure preservation module connected in sequence.

[0014] The text semantic reasoning module is a Transformer-based encoder-decoder structure. The encoder includes a first branch, a second branch, and a fully connected layer. The first branch uses a Transformer language model to model the interdependence between characters and language rules, generating a first probability sequence. The second branch uses a global attention mechanism to model long-distance dependencies between characters, generating a second probability sequence. The first and second probability sequences are concatenated by the fully connected layer and input to the decoder. The decoder uses a global attention mechanism to explore the corresponding positional relationship between the encoder's output and shallow features, obtaining semantic features with high character readability.

[0015] Shallow features and semantic features are input into the reconstruction module, concatenated, and then input into the text structure preservation module to obtain the attention weights of features at each level.

[0016] Super-resolution images are generated by sequentially passing the features at each level with attention weights through normalization layers and sub-pixel convolutional layers.

[0017] A second aspect of the present invention provides a text image super-resolution system based on a multi-scale cross-convolution model, comprising:

[0018] The image acquisition module is configured to acquire low-resolution scene text images that need to be upscaled.

[0019] The image preprocessing module is configured to: preprocess the acquired low-resolution scene text image to obtain the shallow features of the scene text image and the original probability sequence of the characters;

[0020] The reconstruction module is configured to input the acquired shallow features and the original probability sequence into a semantic reasoning and multi-scale cross-convolution model to obtain a super-resolution image.

[0021] The semantic reasoning and multi-scale cross-convolution model includes a text semantic reasoning module, a reconstruction module, and a text structure preservation module connected in sequence.

[0022] The text semantic reasoning module is a Transformer-based encoder-decoder structure. The encoder includes a first branch, a second branch, and a fully connected layer. The first branch uses a Transformer language model to model the interdependence between characters and language rules, generating a first probability sequence. The second branch uses a global attention mechanism to model long-distance dependencies between characters, generating a second probability sequence. The first and second probability sequences are concatenated by the fully connected layer and input to the decoder. The decoder uses a global attention mechanism to explore the corresponding positional relationship between the encoder's output and shallow features, obtaining semantic features with high character readability.

[0023] Shallow features and semantic features are input into the reconstruction module, concatenated, and then input into the text structure preservation module to obtain the attention weights of features at each level.

[0024] High-resolution images are generated by sequentially passing the features at each level with attention weights through normalization layers and subpixel convolutional layers.

[0025] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of an image super-resolution method based on semantic reasoning and cross-convolution as described in the first aspect of the present invention.

[0026] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of an image super-resolution method based on semantic reasoning and cross-convolution as described in the first aspect of the present invention.

[0027] The above one or more technical solutions have the following beneficial effects:

[0028] (1) The text semantic reasoning module proposed in this invention combines the attention mechanism and language rule-based semantic reasoning to process the initial text prior in parallel, and adaptively learns semantic features with high character readability to guide super-resolution. The proposed text structure preservation module explores the correlation of features at each level, emphasizes key features, and uses multi-scale cross-convolutional blocks to focus on global structural features and local feature details in a hierarchical and progressive manner.

[0029] (2) The proposed model focuses on semantic features with high character readability, extraction of multi-scale structural features and learning of structural similarity. This enables the proposed model to generate text images with high character readability and high structural similarity.

[0030] (3) This invention constructs a text semantic reasoning module based on the Transformer encoder-decoder module and improves the encoder. The encoder combines a self-attention mechanism and a Transformer-based language model to achieve two-branch semantic reasoning. The improved encoder can adaptively learn highly accurate semantic features and exhibits good generalization ability on scene images with learnable language rules.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] Figure 1 The flowchart shows a text image super-resolution method based on a multi-scale cross-convolution model, as shown in the first embodiment.

[0034] Figure 2 This is a structural diagram of the text semantic reasoning module in the first embodiment.

[0035] Figure 3 The module structure diagram is reserved for the text structure of the first embodiment.

[0036] Figure 4This is a comparison chart of the results of different reconstruction methods on the TextZoom dataset for the first embodiment.

[0037] Figure 5 This is a comparison chart of the results of different reconstruction methods on the scene text recognition dataset in the first embodiment. Detailed Implementation

[0038] Example 1

[0039] like Figure 1 As shown, this embodiment discloses an image super-resolution method based on semantic reasoning and cross-convolution, including:

[0040] Step 1: Acquire low-resolution scene text images;

[0041] Step 2: Preprocess the low-resolution scene text image to obtain the shallow image features of the scene text image and the original probability sequence;

[0042] Step 3: Input the acquired shallow image features and the original probability sequence into a model based on semantic reasoning and multi-scale cross-convolution to obtain a super-resolution image.

[0043] In step 1, the resolution range of the low-resolution scene text image is less than or equal to 16×64. Before being input into the network, the data is processed and interpolated to 16×64.

[0044] In step 2, before preprocessing the image, a step of inputting a low-resolution image into a text alignment module for correction is also included;

[0045] Due to limitations of the camera and shooting conditions, images processed by the network may exhibit misalignment. The text alignment module employs Spatial Transformer Networks (STNs) to learn and correct the given low-resolution image. The transformation operation uses a TPS transform to minimize the objective function involving multiple sets of acquired image matching point pairs.

[0046] Preprocessing of the corrected low-resolution image includes:

[0047] Step 201, Low-resolution image After passing through a convolutional layer T with a kernel size of 9×9 SF (·) Perform shallow image feature extraction:

[0048] F I =T SF (I LR (1)

[0049] in (h, w, c represent F respectively)I The height, width, and channel dimensions represent the extracted shallow image features. Shallow image features are close to the input and have low semantic meaning.

[0050] Step 202: The recognition network TPG (Text Prior Generator) in this embodiment is an image-based sequence recognition network (Convolutional Recurrent Neural Network, CRNN). Low-resolution image I LR As input to the pre-trained recognition network TPG, it outputs a probability sequence of length l. Probability sequence F R Depend on Composed of a probability vector of dimension, where Representing categories 0 to 9 and a to z, this process can be represented as F. R =R(I LR ).

[0051] In step 3, the model based on semantic reasoning and multi-scale cross-convolution includes a Text Semantic Reasoning Module (TSRM), a reconstruction module, and a Text Structure Preservation Module (TSPM) connected in sequence.

[0052] Step 301: Identify the original probability sequence output by the network. and shallow image features output after a 9×9 convolutional layer As input to the text semantic reasoning module, it generates prior text features that guide the reconstruction module in reconstructing the text content. F TM =T TSRM (F R ,F I ), where T TSRM This represents the text semantic reasoning module.

[0053] The Text Semantic Reasoning (TSRM) module is a Transformer-based encoder-decoder architecture. The encoder adaptively learns highly readable character features by combining self-attention mechanisms with a Transformer-based language model. The decoder utilizes a global attention mechanism to explore the encoder's output against shallow image features F. I The corresponding positional relationship between them.

[0054] Step 302: The text prior features F output by the text semantic reasoning module TMThe semantic cues are input into the reconstruction module. The reconstruction module consists of five Text Prior Guided Blocks (TPGBs). Each TPGB includes sequentially connected horizontal and vertical BLSTM layers. In this embodiment, each TPGB has two inputs; the first TPGB uses shallow features... and text prior features F TM As input; the (i+1)th (i = 1, 2, 3, 4)th TPGB uses the feature map output by the ith TPGB. and text prior features F TM As input, in order to capture greater local semantic information, each TPGB concatenates the two inputs above through a concatenate layer, and then learns features representing contextual relationships and high-frequency information through horizontal and vertical BLSTM layers.

[0055] F1 = GRU V (GRU H (cat(F I F TM (2)

[0056] F i+1 =GRU V (GRU H (cat(F i F TM )), i = 1, 2, 3, 4 (3)

[0057] Where cat represents the concatenation operation, GRU H Represents a horizontal BLSTM layer, GRU V This represents the vertical BLSTM layer.

[0058] Step 303: Features F1 and F2 from each level of TPGB i+1 (i = 1, 2, 3, 4) are concatenated using a concatenation layer to obtain the feature map. Attention weights are obtained through the text structure storage module and are used to represent the structural correlation between features at each level and the corresponding high-resolution image. (Attention weights and feature maps) Multiplication yields a fine-tuned feature map. The fine-tuned feature map output by the text structure preservation module is then fused into a feature map of constant size and c channels through convolutional and normalization layers. Since the shallow feature map contains sufficient original information, a residual operation is performed to preserve this information. An element-wise summation operation is then performed between the shallow feature map and the fused feature map to obtain a new feature map, which is input into a sub-pixel convolutional layer. The final output is a high-resolution image of 32×128. The entire process can be represented as follows:

[0059] Fout =T PS (T BN (Conv(T TSPM (F H ))+F I (4)

[0060] in, Represents a high-resolution image; T TSPM Represents a text structure preservation module; Conv represents a convolutional layer; T BN Indicates the normalization layer; T PS This indicates a subpixel convolutional layer.

[0061] The text semantic reasoning module proposed in this embodiment generates text prior features to guide super-resolution based on a Transformer encoder-decoder structure. The encoder models long-distance dependencies between characters and interdependencies between characters and language rules.

[0062] Low-quality semantic information negatively impacts the performance of super-resolution models. Therefore, the encoder aims to maximize the accuracy of recognizing probability sequences from low-resolution images. A Transformer-based encoder-decoder approach was introduced, with improvements made to the encoder to address these requirements. When faced with the challenge of low-resolution images, humans autonomously extract text from the images and utilize their vocabulary for bidirectional semantic reasoning. Inspired by human behavior, a language model that has learned language rules is used to correct word spelling.

[0063] To enhance the location information, a sinusoidal position code F is used. PE With the original probability sequence Add at the element level; probability sequence As input to the encoder, it passes through two parallel branches, resulting in two probability sequences. and (l is the length of the probability sequence, c is 64).

[0064] like Figure 2 As shown, the text semantic reasoning module is a Transformer-based encoder-decoder structure. The encoder includes a first branch, a second branch, and a fully connected layer. In the first branch, the original probability sequence F... R By modeling the interdependence between characters and language rules using a Transformer-based language model, a first probability sequence is generated.

[0065] The language model utilizes a pre-trained Bidirectional Cloze Network (BCN) model. This model is a variant of the Transformer Decoder, consisting of four Transformer Decoder layers. Each layer is sequentially connected to a multi-head mask attention (MHA) layer, a layer-norm (LN) normalization layer, a feed-forward network (FFN) layer, and another layer-norm (LN). The BCN model specifies an attention mask to prevent attention from being focused on the current character. Training this model performs a task similar to a cloze test.

[0066] The formula for the first branch is as follows:

[0067]

[0068]

[0069] F P =T LN (T FFN (T LN (F MHA ))) (7)

[0070] in, Indicates an attention mask; T represents a linear mapping matrix; LN T FFN These represent the normalization layer and the feedforward neural network layer, respectively. This represents the probability sequence generated by the first branch. The pre-trained BCN model reduces the number of model parameters in this embodiment, accelerating model convergence. To enhance the language model's learning ability for cloze tests, the language model is fine-tuned during training using the KL divergence function:

[0071] L pre =k1||R(I HR )-LM(R(I LR ))||+k2KL(R(I HR )-LM(R(I LR ))) (8)

[0072] The second branch consists of a multi-head self-attention (MSA) layer, a layer-norm (LN) layer, feed-forward network layers (FFN), and another layer-norm (LN) layer connected in sequence. This branch utilizes a global attention mechanism to model long-distance dependencies between characters and generates a second probability sequence. This means that the second branch can establish connections between characters at different positions to capture their semantic relationships; where the multi-head self-attention block MSA of the second branch can be represented as:

[0073]

[0074] in, This represents the linear mapping matrix. Similar to the first branch, the output F of the multi-head self-attention block MSA... MSA The second probability sequence is obtained by sequentially passing through a normalization layer (LN), a feedforward neural network layer (FFN), and another normalization layer (LN). It is expressed as follows:

[0075] F T =T LN (T FFN (T LN (F MSA ))) (10)

[0076] The first and second probability sequences are concatenated by a concatenation layer and then input into a fully connected layer to adaptively select features with high character readability. This encoder uses a fully connected layer to process the concatenated probability sequences, rather than element-wise addition of the two-branch probability sequences. The features output by the encoder, along with the shallow image features extracted by a 9×9 convolutional layer, are input into the decoder. The decoder then passes through a multi-head cross-attention (MCA) layer, a normalization layer, a feedforward neural network layer, and another normalization layer to capture the corresponding positional relationships of the probability sequences within the image features, thus obtaining prior text features. (L is 1024), which is ultimately reshaped into new textual prior features. (c is 64). Text prior features It can be used as a semantic clue input into the reconstruction module to guide the reconstruction of text content.

[0077] like Figure 3As shown, the text structure preservation module is used to alleviate the problem of character structure interleaving in blurred image processing. The text structure preservation module consists of a first 1×1 convolutional layer, a multi-scale cross convolution block (MCCB), a channel attention layer (CA), a first 3×3 convolutional layer, a ReLU layer, a second 3×3 convolutional layer, a second 1×1 convolutional layer, and a sigmoid activation function layer, connected sequentially. Since structural information is sensitive to scale changes, the multi-scale cross convolution block utilizes different receptive fields and the reuse of local features to explore multi-scale structural information, which is beneficial for simultaneously focusing on global structural information and local feature details.

[0078] Features at each level after splicing First, a 1×1 convolutional layer is used to compress the channels to reduce redundant feature information, resulting in a feature map. Then feature map F X Layered progressive exploration is performed using multi-scale cross-convolutional blocks; each multi-scale cross-convolutional block consists of multiple parallel branches, each branch employing a residual structure. The first branch of the multi-scale cross-convolutional block includes a cross-convolutional block (CCB-(m,n)) and a channel attention layer connected in sequence; except for the first branch, each branch includes a 3×3 convolutional layer, a cross-convolutional block (CCB-(m,n)), and a channel attention layer connected in sequence, and the cross-convolutional blocks (CCB-(m,n)) between adjacent branches are progressively connected;

[0079] The output of the cross-convolutional block (CCB-(m,n)) in each branch is passed through the channel attention layer in that branch, and then the low-level structural features are fused through residual connections to obtain the representation features of each branch.

[0080] Specifically, the multi-scale cross-convolutional block in this embodiment includes three branches, each branch adopting a residual structure. The first branch of the multi-scale cross-convolutional block includes a cross-convolutional block CCB-(m,n) and a channel attention layer connected in sequence. The second and third branches include a 3×3 convolutional layer, CCB-(m,n) and a channel attention layer connected in sequence.

[0081] The cross-convolutional block CCB-(m,n) consists of convolutional layers with parallel asymmetric kernels, which are dedicated to preserving structural information in a specific direction. The idea behind cross-convolution is to use convolutional layers with parallel asymmetric kernels to capture edge information in both the horizontal and vertical directions, with the outputs of the two branches added pixel-by-pixel. The two parallel asymmetric kernels are n×m and m×n, respectively. Convolutional layers with parallel asymmetric kernels preserve more structural information than sequentially connected layers. Unlike previous methods that ignored filter size, in the hierarchical CCB-(m,n), the asymmetric kernels m and n take different values, obtaining multi-scale edge information through different receptive fields. The sizes of the horizontal convolutional kernels are 1×3, 3×3, and 3×5, while the values ​​of m and n for the vertical asymmetric kernels are reversed. The ReLU activation function is used after each asymmetric convolution to effectively alleviate the gradient vanishing problem.

[0082] The first branch Represented as:

[0083]

[0084] Where, k n×m k m×n This represents a convolution kernel with values ​​of n×m and m×n, used to extract spatial information; f represents convolution; b is the bias term. relu Represents the ReLU activation function layer; F X This represents the characteristics of each level after channel compression. The output of CCB-(m, n) in the first branch... By using channel attention layers and leveraging residual connections to focus on the original low-level structural features, this branch can better learn the details of local structural features.

[0085]

[0086] in, The channel attention layer represents the first branch; The representation characteristics of the first branch; This represents the low-level structural features of the first branch, i.e., the features F at each level after a 1×1 convolution operation. X .

[0087] As the receptive field expands, the structural features of adjacent layers help recover high-frequency information. The hierarchical CCB-(m, n) are connected in a progressive manner. This progressive approach allows for the acquisition of information from adjacent layers. and The correlation of (j=1,2) enables the fusion and reuse of multi-scale structural features. In the input... Before, The output characteristics and F XBy concatenating features through a concatenation layer and then passing them through a 3×3 convolutional layer, features containing rich high-frequency information can be adaptively learned. (j = 1, 2).

[0088] As the input to CCB-(m, n) in the second and third branches, it can be represented as:

[0089]

[0090] The second and third branches (j = 1, 2) is represented as:

[0091]

[0092] Similarly, in addition to the 3×3 convolutional layer and CCB-(m,n), the second and third branches also include channel attention layers, which emphasize key spatial information in the branches. At the same time, residual learning is introduced to enable feature reuse in the branches, which is more conducive to focusing on the details of local features.

[0093]

[0094] in, This represents the low-level structural features of the (j+1)th branch, i.e., the inputs of CCB-(m, n) in the second and third branches.

[0095] The progressive connectivity based on multi-scale feature fusion proposed in this embodiment allows multi-scale cross-convolutional blocks (MCCBs) to focus more on global information and local details. Three layers of multi-scale structural information are concatenated through a concatenation layer and then passed through a channel attention layer. Channel self-attention layers enable information exchange between channels to enhance key features. A series of subsequent convolutions, ReLU, and sigmoid operations are performed to obtain attention weights, enabling this module to learn to assign attention weights to features at each level.

[0096] To verify the generation performance of the model in this embodiment, the following comparative verification is performed:

[0097] (1) The results were compared with existing scene text image super-resolution models on the TextZoom dataset, and the results are shown in Table 1.

[0098] This section provides a brief introduction to the dataset used and common evaluation metrics in scene image super-resolution.

[0099] The TextZoom dataset consists of a training set containing 17,367 LR-HR image pairs and a test set containing 4,373 LR-HR image pairs. The LR-HR image pairs were captured in real-world scenes using cameras with different focal lengths, making them more challenging than synthetic data. Based on the digital camera's focal length, the test set is split into three subsets: 1,411 images in the easy subset, 1,619 in the medium subset, and 1,343 in the hard subset.

[0100] Scene Text Recognition Datasets: The benchmark text recognition datasets used are the ICDAR2015, ICDAR2013, and SVT datasets. The ICDAR2013 dataset consists of 1,095 real-world text images. The SVT dataset consists of 647 text images from street scenes, some of which are difficult to recognize due to low resolution, blurriness, and noise. The ICDAR2015 dataset consists of 2,077 captured text images; due to limitations in the shooting conditions, most of the text is blurry and multi-directional. 425 samples were selected as the test set based on a resolution lower than 16×64, with 391 images from the ICDAR15 dataset, 14 from the ICDAR13 dataset, and 20 from the SVT dataset. The test set was used to evaluate the proposed model's ability as a scene text recognition preprocessor.

[0101] Evaluation metrics: Evaluation metrics for scene text image super-resolution include text recognition accuracy and image quality metrics. Recognition accuracy is calculated using fixed-parameter recognizers (CRNN, MORAN, and ASTER). A higher average recognition accuracy across the three subsets indicates better model representation ability. Image quality is measured by PSNR and SSIM.

[0102] On the TextZoom dataset, the model in this embodiment was compared with two existing state-of-the-art scene text image super-resolution methods, including general methods and text-oriented methods. Table 1 shows that compared with the general method that treats low-resolution text images as ordinary images, the model in this embodiment significantly improves recognition accuracy, indicating that utilizing text attributes enhances the model's generalization ability. In the text-oriented method, both the model proposed in this embodiment and the PCAN model focus on recovering high-frequency information. The image reconstructed by the model in this embodiment improves the recognition accuracy of CRNN, ASTER, and MORAN recognition models by 5.2%, 2.6%, and 1.6%, respectively.

[0103] Furthermore, compared to the TATT model, the proposed model in this embodiment improves the recognition accuracy on the ASTER and MORAN recognition models by 0.5% and 0.6%, respectively. Both of these models embed Transformer-based modules to provide semantic cues. These experimental results demonstrate that the method in this embodiment significantly outperforms currently popular methods in terms of structural similarity and information integrity.

[0104] Table 1

[0105]

[0106]

[0107] (2) Using benchmark scene text recognition datasets, the proposed model is studied as a preprocessor for scene text recognition tasks.

[0108] Text images were selected from the ICDAR2015, ICDAR2013, and SVT datasets, with a resolution less than 16×64. 425 samples (391 from the ICDAR15 dataset, 14 from the ICDAR13 dataset, and 20 from the SVT dataset) were selected as the test set. The images in this test set were resized to 16×64 and used as the original low-resolution images input to the model. To simulate ill-posed degradation during imaging, Gaussian blur, Gaussian noise, and contrast processing were applied to the low-resolution images. To demonstrate the superior performance of the model in reconstructing extremely blurred images, the radius of the Gaussian blur kernel was set to 5. The contrast processing is represented as follows: Where x and These represent the original low-resolution image and the text image after contrast processing, respectively.

[0109] Table 2 compares the recognition accuracy of the method in this embodiment with other methods under each type of degradation, where "O", "GB", "GN", and "C" represent the original low-resolution image, Gaussian blur, Gaussian noise, and contrast, respectively. As shown in Table 2, the proposed method in this embodiment outperforms other methods in all types of degradation. Specifically, under Gaussian blur processing, the model in this embodiment shows a significant improvement in recognition accuracy, increasing by 23.3% on the CRNN recognizer, 14.1% on the MORAN recognizer, and 16.3% on the ASTER recognizer. This result confirms that the model in this embodiment has a greater advantage in processing extremely blurred real text images, making it more beneficial for subsequent text recognition tasks.

[0110] Table 2

[0111]

[0112] (3) Using the TextZoom test set, we studied the impact of different designs of the text semantic reasoning module on the super-resolution performance of the model. The experimental content is as follows:

[0113] To verify the rationality of the text semantic reasoning module design, a network was constructed by removing the structural information preservation module and the text semantic reasoning module; this network is called TINet. The TPGSR module providing text priors consists of three sequentially connected deconvolutions with a stride of (2,2) and one deconvolution with a stride of (2,1) to transform the spatial dimension of the text priors. The TPI encoder utilizes a global attention mechanism to enhance the contextual dependencies between text priors.

[0114] As shown in Table 3, compared to the TPGSR module which provides text priors, the global attention mechanism improves the average recognition accuracy by 1.9%. To better address the challenges of blurred images, the method in this embodiment attempts to leverage the power of strong language rules. "TPI+LM" indicates that the global attention mechanism and the Transformer-based language model are designed as two horizontal branches, processing the original text prior in parallel. The probability sequences output by the two branches are added element-wise. This module improves the TINet of the TPI module by 0.3%. TSRM also consists of a self-attention mechanism and a two-branch model that has learned language rules. Specifically, the probability sequences output by the two branches are concatenated and fed into a fully connected layer. Unlike direct element-wise addition, TSRM adaptively learns a more accurate text prior by using a fully connected layer. As shown in Table 3, the recognition accuracy of "TINet+TSRM" is further improved.

[0115] (4) During training, the pre-trained language model is fine-tuned under the supervision of the KL divergence function to more accurately recover character semantic information. This is achieved by setting λ... PRE This ensures a balance between semantic and structural information in our proposed model, thereby improving its performance.

[0116] In order to study λ PRE The impact of different values ​​of λ on model performance was investigated in this embodiment, with comparative experiments performed for different values. PRE Values ​​are taken from the set {0.2, 0.5, 1}, as shown in Table 4:

[0117] Table 4

[0118]

[0119] As can be seen from Table 4, with λ PRE As λ increases, the image quality index of the reconstructed image continuously improves, indicating a significant improvement in the quality of the reconstructed image.PRE When = 1, and λ PRE Compared to the method with λ=0.2, this model improves PSNR by 0.44 dB and SSIM by 0.66%. This means that λ PRE The model performs best when the value is 1.

[0120] Example 2

[0121] This embodiment discloses an image super-resolution system based on semantic reasoning and cross-convolution, including:

[0122] The image acquisition module is configured to acquire low-resolution scene text images that need to be upscaled.

[0123] The image preprocessing module is configured to: preprocess the acquired low-resolution scene text image to obtain the shallow features of the scene text image and the original probability sequence of the characters;

[0124] The reconstruction module is configured to input the acquired shallow features and the original probability sequence into a semantic reasoning and multi-scale cross-convolution model to obtain a super-resolution image.

[0125] The semantic reasoning and multi-scale cross-convolution model includes a text semantic reasoning module, a reconstruction module, and a text structure preservation module connected in sequence.

[0126] The text semantic reasoning module is a Transformer-based encoder-decoder structure. The encoder includes a first branch, a second branch, and a fully connected layer. The first branch uses a Transformer language model to model the interdependence between characters and language rules, generating a first probability sequence. The second branch uses a global attention mechanism to model the long-distance dependency between characters, generating a second probability sequence. The first and second probability sequences are concatenated by the fully connected layer and then input to the decoder. The decoder uses a global attention mechanism to explore the corresponding positional relationship between the encoder's output and shallow features, obtaining text prior features with high character readability.

[0127] Shallow features and prior text features are input into the reconstruction module, concatenated, and then input into the text structure preservation module to obtain the attention weights of features at each level.

[0128] High-resolution images are generated by sequentially passing the features at each level with attention weights through normalization layers and subpixel convolutional layers.

[0129] Example 3

[0130] The purpose of this embodiment is to provide a computer-readable storage medium.

[0131] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of an image super-resolution method based on semantic reasoning and cross-convolution as described in Embodiment 1 of this disclosure.

[0132] Example 4

[0133] The purpose of this embodiment is to provide an electronic device.

[0134] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in an image super-resolution method based on semantic reasoning and cross-convolution as described in Embodiment 1 of this disclosure.

[0135] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0136] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0137] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An image super-resolution method based on semantic reasoning and cross-convolution, characterized in that, include: Acquire the low-resolution scene text image to be upgraded; Low-resolution scene text images are preprocessed to obtain shallow features of the image and the original probability sequence; The acquired shallow features and the original probability sequence are input into a semantic reasoning and multi-scale cross-convolution model to obtain a high-resolution image. The semantic reasoning and multi-scale cross-convolution model includes a text semantic reasoning module, a reconstruction module, and a text structure preservation module connected in sequence. The text semantic reasoning module is a Transformer-based encoder-decoder structure. The encoder includes a first branch, a second branch, and a fully connected layer. The first branch uses the Transformer language model to model the interdependence between characters and language rules, generating a first probability sequence. The second branch uses a global attention mechanism to model the long-distance dependency between characters, generating a second probability sequence. After the first and second probability sequences are concatenated and processed through a fully connected layer, they are input into the decoder. The decoder uses a global attention mechanism to explore the corresponding positional relationship between the encoder's output and shallow features, thereby obtaining text prior features with high character readability. Shallow features and prior text features are input into the reconstruction module, concatenated, and then input into the text structure preservation module to obtain the attention weights of features at each level. The reconstruction module includes multiple sequentially connected text prior guidance blocks. Shallow features and text prior features are input into the reconstruction module for concatenation, including: Each text prior guidance block consists of BLSTM layers connected in the horizontal and vertical directions in sequence; each text prior guidance block has two inputs. The first text prior guidance block takes shallow features and text prior features as inputs; the subsequent text prior guidance block takes the output of the previous text prior guidance block and text prior features as inputs. The text prior guidance block concatenates two inputs through a concatenate layer, and then learns features representing contextual relationships and high-frequency information through horizontal and vertical BLSTM layers, resulting in feature maps of different levels output by multiple text prior guidance blocks; The feature maps at different levels are concatenated through a concatenation layer to obtain the final feature map output by the reconstruction module. The text structure preservation module includes a multi-scale cross-convolution block; the feature map output by the reconstruction module is first compressed through a convolutional layer and then input into the multi-scale cross-convolution block for hierarchical progressive exploration. The multi-scale cross-convolutional block includes multiple branches, each branch adopts a residual structure, and the first branch of the multi-scale cross-convolutional block includes a cross-convolutional block and a channel attention layer connected in sequence; except for the first branch, each of the remaining branches includes a 3×3 convolutional layer, a cross-convolutional block and a channel attention layer connected in sequence, and the cross-convolutional blocks between adjacent branches are connected in a progressive manner. The output of the cross-convolutional block in each branch is passed through the channel attention layer in that branch, and then the low-level structural features are fused through residual connections to obtain the representation features of each branch. High-resolution images are generated by sequentially passing the features at each level with attention weights through normalization layers and subpixel convolutional layers.

2. The image super-resolution method based on semantic reasoning and cross-convolution as described in claim 1, characterized in that, The original probability sequence of the character is a probability sequence after sinusoidal position encoding.

3. The image super-resolution method based on semantic reasoning and cross-convolution as described in claim 2, characterized in that, The language model is a pre-trained BCN model, which contains 4 Transformer Decoder-Layers. Each layer includes a multi-head mask attention layer, a normalization layer, a feedforward neural network layer, and a normalization layer connected in sequence. The first probability sequence generated by the first branch for: in, Indicates an attention mask; The output of the multi-head masked attention block (MHA); Represents the original probability sequence; Indicates the channel dimension; Represents the sine wave position code; Represents a linear mapping matrix; , These represent the normalization layer and the feedforward neural network layer, respectively. This represents the first probability sequence generated by the first branch.

4. The image super-resolution method based on semantic reasoning and cross-convolution as described in claim 3, characterized in that, The second branch includes a multi-head self-attention layer, a normalization layer, a feedforward neural network layer, and another normalization layer connected in sequence; The second probability sequence generated by the second branch for: in, , , Represents a linear mapping matrix. This is the output of the multi-head self-attention block (MSA).

5. The image super-resolution method based on semantic reasoning and cross-convolution as described in claim 4, characterized in that, The cross-convolutional block includes parallel asymmetric convolutional layers, which use parallel asymmetric convolutional kernels to capture edge information in both horizontal and vertical directions. Each branch of the convolutional layer is followed by a ReLU activation function layer.

6. An image super-resolution system based on semantic reasoning and cross-convolution, employing an image super-resolution method based on semantic reasoning and cross-convolution as described in any one of claims 1-5, characterized in that: include: The low-resolution image acquisition module is configured to acquire low-resolution scene text images; The low-resolution image preprocessing module is configured to: preprocess the acquired low-resolution scene text image to obtain the shallow features of the scene text image and the original probability sequence of the characters. The reconstruction module is configured to input the acquired shallow features and the original probability sequence into a semantic reasoning and multi-scale cross-convolution model to obtain a super-resolution image. The semantic reasoning and multi-scale cross-convolution model includes a text semantic reasoning module, a reconstruction module, and a text structure preservation module connected in sequence. The text semantic reasoning module is a Transformer-based encoder-decoder structure. The encoder includes a first branch, a second branch, and a fully connected layer. The first branch uses the Transformer language model to model the interdependence between characters and language rules, generating a first probability sequence. The second branch uses a global attention mechanism to model the long-distance dependency between characters, generating a second probability sequence. The first probability sequence and the second probability sequence are concatenated by a fully connected layer and then input to the decoder. The decoder uses a global attention mechanism to explore the corresponding positional relationship between the encoder output and shallow features, and obtains text prior features with high character readability. Shallow features and prior text features are input into the reconstruction module, concatenated, and then input into the text structure preservation module to obtain the attention weights of features at each level. Super-resolution images are generated by sequentially passing the features at each level with attention weights through normalization layers and sub-pixel convolutional layers.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps of an image super-resolution method based on semantic reasoning and cross-convolution as described in any one of claims 1-5.

8. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the image super-resolution method based on semantic reasoning and cross-convolution as described in any one of claims 1-5.

Images