Scene image super-resolution method based on code prediction and text prior guidance

Abstract

The invention relates to the technical field of computer vision and image processing, and discloses a scene image super-resolution method based on code prediction and text prior guidance, which comprises the following steps of: constructing a discrete feature space and a quantitative auto-encoder, converting continuous feature mapping into discrete codebook index prediction, relieving the uncertainty of a traditional method, and improving the image super-resolution efficiency. The quality stability and prediction accuracy of the reconstructed image are improved; in combination with a multi-scale code prediction module and a scale fusion attention mechanism, local and global features are captured, the text recognition accuracy and edge fidelity are improved, and discrete space priori knowledge is effectively utilized; multi-source text priori knowledge is introduced, collaborative optimization is carried out by means of hierarchical and multi-level cross attention mechanisms, character-level and word-level recognition accuracy is improved, and generalization ability is enhanced; through multilayer feature fusion and text structure consistency loss, semantic consistency is ensured, average and text readability is improved in a comprehensive test set, and robustness and generalization are excellent.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and more specifically, to a scene image super-resolution method based on code prediction and text prior guidance. Background Technology

[0002] With the ever-increasing demand for digital information processing, especially in fields such as smart education, intelligent transportation, and smart healthcare that require optical character recognition (OCR), the application of scene text image super-resolution technology is becoming increasingly widespread. Although traditional super-resolution reconstruction methods have made some progress in backbone network architecture design, prior knowledge extraction, and feature fusion techniques, they still have not systematically solved core technical challenges such as the uncertainty of low-resolution to high-resolution mapping, the effective utilization of prior knowledge in discrete feature space, and the deep fusion of prior information from multiple text sources. These challenges make it difficult to meet the generalization requirements of different scenarios, fonts, and text types.

[0003] Therefore, how to construct a scene text super-resolution method that can effectively integrate prior knowledge from multiple sources of text, accurately predict high-quality feature codes, and has strong robustness and generalization ability has become an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides a scene image super-resolution method based on code prediction and text prior guidance, which solves the technical problems in the prior art of constructing a discrete feature space, effectively integrating multi-source text prior knowledge, accurately predicting high-quality feature codes, and having strong robustness and generalization ability.

[0005] This invention provides a scene image super-resolution method based on code prediction and text prior guidance, comprising the following steps:

[0006] Construct a discrete feature space, collect high-resolution text images from public datasets, and use vector quantization techniques to map text image features onto a discrete codebook to form a shared basic representation of high-quality features;

[0007] Based on the discrete feature space, a quantized autoencoder is trained and the ViT-VQGAN architecture is adopted, which includes an encoder, a learnable discrete codebook and a decoder. The codebook utilization is optimized and the image quality is reconstructed through split decoding and L2 normalization techniques.

[0008] Based on a quantized autoencoder, a multi-scale code prediction mechanism is constructed. Given low-quality image features, a non-autoregressive Transformer encoder with a multi-scale attention mechanism predicts the index of the corresponding high-quality features in the codebook at different scales.

[0009] Based on the code prediction mechanism, multi-source text prior guidance is introduced, and the CRNN text recognition model, technical documents, user manuals, and programming forum multi-source text prior knowledge are comprehensively utilized, and the multi-source text prior knowledge is fused into the code prediction mechanism through a multi-level cross attention mechanism;

[0010] In combination with high-quality code features and a decoder, a super-resolution backbone network is constructed, high-quality code features obtained through the multi-scale code prediction mechanism are subjected to multi-layer feature fusion processing, and each SRB module of the super-resolution backbone network is introduced, and finally, super-resolution reconstruction is completed.

[0011] As a preferred scheme of the present application, the training quantization autoencoder comprises:

[0012] Based on the discrete codebook, the encoder divides the input image into blocks and performs linear projection to obtain a feature vector, and the feature vector is input into the encoder after adding position encoding to obtain a feature vector;

[0013] The feature is mapped to a low-dimensional latent space through decomposition and normalization operations, the nearest neighbor discrete vector in the codebook is searched, and the feature is mapped back to a high-dimensional space after quantization;

[0014] The quantized feature is input into the decoder to reconstruct the image, so as to provide an encoding-decoding reference for the code prediction mechanism;

[0015] The encoder outputs a high-quality text image feature representation through a loss function, and the loss function comprises a reconstruction loss, an adversarial training mechanism, a perception loss, and a codebook space optimization loss.

[0016] As a preferred scheme of the present application, the multi-scale code prediction comprises:

[0017] Based on the quantization autoencoder, a non-autoregressive Transformer encoder with a multi-scale attention mechanism is used as the code prediction mechanism;

[0018] Feature extraction branches are constructed on different scale levels to capture local detail features and global structure features, respectively;

[0019] Different scale feature information is integrated through a scale fusion attention mechanism;

[0020] The input feature is mapped to the codebook quantity dimension through a multi-scale classification head, and a weighted cross-entropy loss is used for training;

[0021] A multi-scale L2 loss is introduced to constrain the encoder output, so that it is close to the high-resolution image encoder output at each scale, and accurate code prediction results are provided for text prior guidance and feature fusion.

[0022] As a preferred scheme of the present application, the multi-source text prior guidance comprises:

[0023] Based on the multi-scale code prediction result, the scaled low-resolution image input is input into the learnable CRNN to obtain logits values, and the basic text priori knowledge is obtained through the softmax operation;

[0024] Collect technical documents, user manuals, programming forum Q&A multi-source text resources related to the target code, and extract semantic features through a pre-trained language model;

[0025] A hierarchical attention fusion mechanism is adopted, including intra-source attention and inter-source attention, to weight and fuse the multi-source text resources;

[0026] The high-resolution image is input into the pre-trained CRNN to obtain reference priori knowledge;

[0027] The multi-source KL divergence loss is used to make the source priori distribution close to the reference distribution;

[0028] In the code prediction mechanism, the fused multi-source text priori knowledge is used as the value and key vector, and the feature token output by the self-attention is used as the query vector, and the multi-level cross-attention mechanism is used to fuse the text priori knowledge, so that the code prediction accuracy is improved, and high-quality code features are provided for super-resolution reconstruction.

[0029] As a preferred scheme of the present application, the multi-layer feature fusion processing comprises:

[0030] The high-quality code features of the multi-scale code prediction mechanism are input into the decoder to extract multi-layer intermediate features;

[0031] The intermediate features are introduced into each SRB module of the super-resolution backbone network through multi-head cross-attention;

[0032] The high-quality code features and the intermediate features are concatenated and introduced into the multi-head cross-attention, so that the intermediate features of the super-resolution network use the high-quality code features for feature extraction;

[0033] The scale adaptive mechanism is introduced at different levels, the fusion weight is dynamically adjusted according to the feature resolution of the current layer and the multi-scale prediction result, and the final super-resolution reconstruction is completed.

[0034] As a preferred scheme of the present application, it further comprises a text structure consistency loss, which comprises:

[0035] The multi-source text priori mechanism and the text structure consistency loss are introduced into the super-resolution backbone network;

[0036] The multi-source text priori feature map is extracted and fused into the image features through the multi-source TP-Interpreter;

[0037] The text structure of the generated super-resolution image is consistent with that of the real high-resolution image by using a multi-level text structure consistency loss constraint;

[0038] A semantic consistency loss is introduced to ensure that the reconstructed image is consistent with the original image in the semantic level and forms a synergistic optimization with the multi-source text prior information guidance.

[0039] As a preferred scheme of the present application, the encoder and the decoder in the ViT-VQGAN architecture are both based on the ViT encoder and add multi-scale position encoding, which is consistent with the multi-scale code prediction maintaining architecture;

[0040] The decomposition decoding operation is realized by decomposing the feature vector into multiple sub-vectors, and a scale perception mechanism is introduced to provide a feature basis for multi-scale prediction;

[0041] The L2 normalization technology is applied to the feature vector before quantization, and a dynamic codebook updating strategy is used to optimize the utilization rate of the codebook, so that the code can be accurately predicted.

[0042] As a preferred scheme of the present application, the super-resolution backbone network is constructed based on the TATT architecture, which includes:

[0043] The shallow convolutional layer is used to extract initial features;

[0044] The multiple SRB modules are used for feature processing and enhancement, and each SRB module internally integrates a multi-head cross-attention mechanism and a scale adaptive fusion module;

[0045] The PixelShuffle up-sampling layer is used for feature dimension transformation to complete super-resolution reconstruction;

[0046] In each SRB module, high-quality code features from the multi-scale code prediction mechanism and multi-layer intermediate features from the decoder are integrated;

[0047] An incremental feature refinement mechanism is introduced to gradually refine text details and structure information at different levels of the network.

[0048] As a preferred scheme of the present application, the multi-scale attention mechanism includes:

[0049] The local detail attention branch focuses on capturing character stroke, edge local detail features;

[0050] The global structure attention branch focuses on capturing text line arrangement, overall layout global structure features;

[0051] The scale fusion attention module dynamically fuses feature information of different scales through learnable weights;

[0052] A multi-scale feature alignment module ensures the consistency of features of different scales in space and semantics.

[0053] As a preferred scheme of the present application, the acquisition and processing of the multi-source text priori information comprises:

[0054] Technical document priori: extracting domain-specific text patterns and terms from relevant technical specifications and API documents;

[0055] User manual priori: extracting common text expressions and format specifications from product usage instructions and operation manuals;

[0056] Forum Q&A priori: extracting user-generated text content and expression habits from programming forums and technical communities;

[0057] Semantically encoding various types of text priors through pre-trained BERT or GPT models;

[0058] Using a hierarchical attention weight learning mechanism to dynamically adjust the contribution weights of each source prior according to the input image features.

[0059] The present application has the beneficial effects that: the present application maps the continuous features into discrete codebook indexes by constructing a discrete feature space and a quantization autoencoder, effectively alleviating the uncertainty caused by one-to-many mapping in traditional methods. Experimental results show that, compared with traditional continuous mapping methods, the discrete feature representation method of the present application has improved PSNR and SSIM indicators, thereby improving the quality stability and prediction accuracy of the reconstructed image.

[0060] Through the multi-scale code prediction module, feature extraction branches are constructed at different scale levels, and scale fusion attention mechanisms are combined to effectively capture and utilize local detail features and global structure features. Compared with existing methods, the text recognition accuracy of the present application on the TextZoom dataset is improved, and the edge fidelity index on the RealSR dataset is improved, proving the effective extraction and utilization of prior knowledge in the discrete feature space.

[0061] By introducing CRNN text recognition models, technical documents, user manuals, programming forums and other multi-source text priori knowledge, using hierarchical attention fusion mechanisms and multi-level cross-attention mechanisms, the collaborative optimization of multi-source priori information is realized. Experimental verification shows that, compared with single text priori methods, multi-source priori fusion improves the character-level recognition accuracy and word-level recognition accuracy, thereby enhancing the generalization ability of the method to different scenarios, different fonts and different text types.

[0062] Through the synergistic effect of the multi-layer feature fusion module and the text structure consistency loss, it is ensured that the reconstructed image is consistent with the original image in the semantic level. On a comprehensive test set containing complex background, multiple fonts and different illumination conditions, the average PSNR of the method is improved compared with the existing optimal method, and the text readability score is also improved, which shows excellent robustness and generalization performance.

[0063] In summary, the technical scheme of the present application effectively solves the core technical problems in the background technology through systematic architecture design and innovative algorithm fusion, providing reliable technical support for the practical application of scene text super-resolution technology. BRIEF DESCRIPTION OF DRAWINGS

[0064] Fig. 1 is a scene image super-resolution method based on code prediction and text prior guidance provided in the embodiments of the present application, and

[0065] Fig. 2 is a block segmentation module schematic diagram of the scene image super-resolution method based on code prediction and text prior guidance in the embodiments of the present application.

[0066] Fig. 3 is a space module schematic diagram of the scene image super-resolution method based on code prediction and text prior guidance in the embodiments of the present application. DETAILED DESCRIPTION

[0067] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that discussions of these implementations are merely provided to give a more fuller understanding of the subject matter described herein and are not intended to limit the scope of the protection of this specification. The functions and arrangements of the elements discussed can be varied from those described and can be changed to provide different results. Various examples can omit, substitute, or add various procedures or components as appropriate, or change the order of the described procedures. Also, features described with respect to one example can be combined in other examples.

[0068] A scene image super-resolution method based on code prediction and text prior guidance is disclosed in at least one embodiment of the present application, as shown in Figs. 1-3 comprising the following steps:

[0069] Step 1: Construct a discrete feature space, collect high-resolution text images from a public data set, and use vector quantization technology to map text image features to a discrete codebook to form a shared basic representation of high-quality features;

[0070] Step 2: Train the quantized autoencoder based on the discrete feature space, using the ViT-VQGAN architecture, which includes an encoder, a learnable discrete codebook, and a decoder. Optimize codebook utilization and reconstruct image quality through decomposition and L2 normalization techniques.

[0071] Step 3: Based on the quantized autoencoder, construct a multi-scale code prediction mechanism. Given low-quality image features, use a non-autoregressive Transformer encoder with multi-scale attention mechanism to predict the corresponding high-quality feature indices in the codebook at different scales.

[0072] Step 4: Based on the code prediction mechanism, introduce multi-source text prior guidance. Utilize CRNN text recognition models, technical documents, user manuals, and programming forum multi-source text prior knowledge through multi-level cross-attention mechanisms to integrate multi-source text prior knowledge into the code prediction mechanism.

[0073] Step 5: Combine high-quality code features and decoders to construct a super-resolution backbone network. Perform multi-layer feature fusion processing on the high-quality code features obtained from the multi-scale code prediction mechanism and introduce SRB modules in each layer of the super-resolution backbone network to complete the final super-resolution reconstruction.

[0074] Example 1

[0075] Discrete codebook construction based on ViT-VQGAN: Map continuous high-resolution image features to discrete codebook space to lay the foundation for subsequent deterministic mapping.

[0076] Specific implementation process:

[0077] Data preprocessing: Collect high-resolution scene text images from public datasets such as Unin14M. The specific selection criteria are images with a height greater than or equal to 64 pixels. A total of about 400k high-quality text images are selected. Due to different text lengths, images are scaled to 32x128 pixels based on image height. Standardize the images by normalizing pixel values to the range [-1, 1] to ensure training stability.

[0078] Encoder design: Use the VisinTransformer architecture as the encoder backbone. Divide the input image into a set of non-overlapping image blocks with a size of 16x16, resulting in a total of 8 image blocks. Stretch each image block to 768 pixels and input it into a linear layer to obtain 512-dimensional features. Add learnable position encoding and input it into a multi-layer Transformer encoder to obtain the encoder output features.

[0079] Vector Quantization Process: A learnable discrete codebook is established, containing 1024 512-dimensional codebook vectors. For each feature vector output by the encoder, the nearest neighbor search is used to find the most matching codebook vector.

[0080] Decoding Optimization: To address the issue of low codebook utilization in VQVAE and VQGAN, ViT-VQGAN proposes a decoding technique. First, the encoder output features are mapped to a low-dimensional latent space through a linear layer, reducing the feature dimension from 512 to 32, and the codebook is also learned in this low-dimensional space. Then, another linear layer is used to map back to the original high-dimensional feature space. L2 normalization is further introduced to map the latent variables to the spherical space, where each vector finds the discrete vector in the codebook with the smallest Euclidean distance through nearest neighbor search.

[0081] Decoder Reconstruction: The quantized features are mapped to the original high-dimensional space through a linear layer, and then input into the ViT decoder for image reconstruction. The decoder is also based on the ViT encoder and needs to add position encoding. Finally, the reconstructed image is obtained.

[0082] Loss Function Optimization: The loss function in the first stage contains multiple components. First, the reconstruction loss measures the difference between the original image and the reconstructed image. Since the argmin operation is used, the gradient cannot be passed from the decoder to the encoder, so the straight-through estimator technique is used to copy the gradient from the decoder to the encoder. To avoid the reconstructed image being too blurred, the adversarial training mechanism of the generative adversarial network is introduced. At the same time, perceptual loss is introduced to improve the perceptual quality of the reconstructed image, using the VGG-16 network pre-trained on the ImageNet dataset to extract features. In addition, the codebook space needs to be optimized, and the codebook loss mainly consists of codebook learning and encoder optimization.

[0083] Autoencoder Network Training: Further optimize the performance of the encoder and decoder to ensure high-quality feature extraction and image reconstruction capabilities.

[0084] Specific Implementation Process:

[0085] Multi-scale Position Encoding: Multi-scale position encoding is added to the ViT encoder to adapt to different resolution feature processing needs, supporting different scale factors such as 1x, 2x, 4x, and 8x.

[0086] L2 Normalization Optimization: Perform L2 normalization on the feature vectors before quantization to make the feature distribution more uniform and improve the utilization efficiency of the codebook.

[0087] Dynamic Codebook Update: Exponential moving average strategy is used to update the codebook vectors, with a dynamic update factor controlling the fusion ratio of new and old features.

[0088] Perceptual loss enhancement: Introduce perceptual loss of pre-trained VGG network to improve the visual quality of reconstructed images.

[0089] Multi-scale code prediction module is to learn the mapping relationship from low-resolution image features to high-resolution discrete codebook index, realizing deterministic feature prediction. In the second stage, to prevent disturbing the codebook learned in the first stage, the parameters of the codebook and the decoder are fixed.

[0090] Specific implementation process:

[0091] Dataset and preprocessing: The second stage uses the TextZm dataset for training. TextZm is a dataset focused on text image super-resolution in real scenes, with low-resolution images of 16x32 and high-resolution images of 32x128, achieving 2x super-resolution. The training set contains 17367 images, and the test set is divided into easy (1619 images), medium (1411 images), and hard (1343 images) three subsets according to the difficulty.

[0092] Input processing: Given a 2x low-resolution image and its corresponding mask binary image synthesized by a simple clustering algorithm, the image is first corrected through the STN (Spatial Transformation Network) module, because there is a pixel misalignment problem between the low-resolution image and the high-resolution image.

[0093] Feature encoding: Take the RGB channel of the corrected image and input it into the encoder. To keep the length of the encoder output latent features consistent, the block size is set to 2x8. Similarly, the encoder output features are decomposed and L2 normalized to obtain the low-resolution image encoding features.

[0094] Non-autoregressive Transfrmer code prediction: Use a non-autoregressive Transfrmer encoder as the code prediction module. The low-resolution encoding features are input into the code prediction module (plus position encoding) to obtain code prediction features. Then, the code prediction features are mapped to the dimension of the codebook size through the classification head to obtain the codebook index prediction result.

[0095] Multi-scale code prediction attention mechanism: To overcome the limitations of traditional single-scale code prediction, a multi-scale code prediction attention module is designed to extract and focus on code features at different scale levels, achieving comprehensive feature capture from fine-grained strokes to macro-text structure.

[0096] Fine-grained stroke attention branch (Scale-1): Focus on capturing character strokes, edges, and other microscopic detail features:

[0097]

[0098]

[0099] Capture pixel-level detail changes via small-scale convolution kernels (1x1 and 3x3).

[0100] where:represents the fine-grained stroke attention output; represents the query, key, and value vectors, respectively; represents the low-resolution image features; represents the 1x1 convolution operation; represents the 3x3 convolution operation; represents the multi-head attention mechanism.

[0101] Character-level structure attention branch (Scale-2): focus on the complete structure and morphological features of individual characters:

[0102]

[0103]

[0104] Model character-level features using medium-scale convolution kernels (5x5 and 7x7).

[0105] where:represents the character-level structure attention output; represents the character-level query, key, and value vectors, respectively; represents the 5x5 convolution operation; represents the 7x7 convolution operation. Text line-level global attention branch (Scale-3): capture macro layout features such as text line arrangement, character spacing, etc.

[0106]

[0107]

[0108]

[0109]

[0110]

[0111] Combine global average pooling and large-scale convolution kernels (11x11) to obtain global context.

[0112] where:represents the text line-level global attention output; represents the global-level query, key, and value vectors, respectively; represents the global average pooling operation; represents the 11x11 convolution operation.

[0113] Cross-scale feature alignment module: ensures the consistency of different scale features in spatial and semantic dimensions:

[0114]

[0115]

[0116] wherein: represents the aligned feature of the th scale; represents the scale index; represents the concatenation of three attention outputs; represents the spatial alignment operation; represents the layer normalization operation; represents the linear transformation operation.

[0117] Adaptive scale fusion attention: dynamically fuses the code prediction results of different scales through learnable weights:

[0118]

[0119]

[0120]

[0121] wherein: represents the scale weight vector; respectively represents the weight of the three scale branches; represents the fused attention feature; represents the final multi-scale feature output; represents the normalization function; represents the multi-layer perception; represents the feature concatenation operation; represents the residual connection operation.

[0122] Multi-scale code classification prediction: for the fused multi-scale feature, a hierarchical classification strategy is adopted for codebook index prediction:

[0123]

[0124]

[0125]

[0126] The final code prediction probability is obtained by weighted fusion:

[0127]

[0128] wherein: Code prediction probability of three scale branches respectively; Linear classification layer corresponding to three scales respectively; Final code prediction probability; Fusion weight coefficients of three scales respectively, satisfying .

[0129] Multi-scale code prediction loss function: for multi-scale code prediction results, a hierarchical loss function is designed.

[0130] Scale-specific loss: each scale branch has a corresponding cross-entropy loss, which calculates the prediction loss of stroke level, character level and global level respectively.

[0131] Fusion code prediction loss: calculate the cross-entropy loss between the fusion prediction result and the true value code index.

[0132] Multi-scale feature consistency constraint: ensure the consistency of the encoder output features of different scales, and calculate the L2 distance loss between the features of each scale after global averaging.

[0133] Encoder feature constraint loss: constraint the consistency of multi-scale fusion features and high-resolution image encoding features.

[0134] Multi-source text prior guidance module: build a multi-source heterogeneous text prior fusion framework, comprehensively utilize multi-dimensional text resources such as technical documents, user manuals, programming forums, etc., and realize the deep integration of text prior knowledge through hierarchical attention fusion mechanism, to provide more comprehensive and accurate semantic guidance for code prediction process.

[0135] Multi-source heterogeneous text prior construction:

[0136] Technical document prior library: extract domain-specific text patterns and professional terms from related technical specifications, API documents, and developer guides. A hybrid method based on TF-IDF and Wrd2Vec is used to build a technical vocabulary library containing about 50,000 technical terms and common code patterns. Each entry contains term content, importance weight and context semantic vector.

[0137] User manual prior library: extract user's common text expressions and format specifications from product instructions, operation manuals, and frequently asked questions to establish a user-oriented text pattern library. Each pattern contains user text pattern, frequency of occurrence and semantic features.

[0138] Forum Q&A prior library: extract user-generated text content and expression habits from programming forums, technical communities, Stackverflw and other platforms to build a socialized text prior library. Each entry contains Q&A content, voting weight and label features.

[0139] Code Comment Prior Library: This library adds a code comment text prior, extracting high-quality code comments from open-source code repositories and establishing a code-comment correspondence. Each entry includes code content, comment content, and a similarity score.

[0140] Enhanced CRNN text recognition and multi-source prior fusion:

[0141] Basic CRNN text recognition: Input a learnable CRNN into a scaled low-resolution image to obtain basic text priors;

[0142] High-resolution reference generation: Input a high-resolution image into a pre-trained CRNN to obtain a reference check;

[0143] Multi-source text prior retrieval: Based on the prediction results of the basic text prior, relevant text patterns are retrieved from the multi-source prior library, and the top 100 most similar entries for each source are returned.

[0144] Multi-source heterogeneous text prior semantic encoding: A multimodal encoder is used to differentially encode different types of text priors.

[0145] Technical Document Encoder: Specifically designed to encode technical terminology and API documentation using the BERT-Tech model;

[0146] User content encoder: Encodes user-generated content using the GPT-User model;

[0147] Community Question Answering Encoder: Specifically designed to encode RBERTa-QA models that handle question-answer pairs;

[0148] Code comment encoder: Encodes the CdeBERT model to handle code-comment relationships.

[0149] Multi-level heterogeneous text prior fusion mechanism:

[0150] First layer: In-source adaptive attention aggregation;

[0151] Multi-head self-attention aggregation is performed within each source, and then fused features are obtained through layer normalization and residual connections;

[0152] The second layer: cross-source attention calculation calculates the cross-attention between technical documents and user content, technical documents and forum Q&A, user content and forum Q&A, code comments and the other three sources;

[0153] The third layer: Image feature-guided dynamic weight learning. Based on the input image features, the importance weights of each source are dynamically learned. The weight allocation of the four sources is calculated through an MLP network to ensure that the sum of the weights is 1.

[0154] The fourth layer: multi-modal feature alignment and fusion aligns features of different dimensions to a unified space through projection, then performs weighted fusion and adds residual connection and layer normalization.

[0155] Multi-source text prior loss function: for multi-source heterogeneous text prior, a hierarchical loss constraint is designed:

[0156] Basic text prior loss: including L1 loss and KL divergence loss, where KL divergence loss measures the difference between low-resolution prediction distribution and high-resolution reference distribution;

[0157] Multi-source consistency loss: ensures consistency of text priors from different sources in semantic space, calculates L2 distance loss of average-pooled features between all source pairs;

[0158] Inter-source complementarity loss: encourages different sources to provide complementary information rather than redundant information, achieved by negative cosine similarity;

[0159] Fusion quality loss: constraints the quality of fused features, achieved by L2 loss between CRNN predicted enhanced features and high-resolution reference.

[0160] Multi-source heterogeneous cross-attention fusion: integrates enhanced multi-source text prior knowledge into code prediction module:

[0161] Multi-source query generation: generates corresponding query, key and value matrices for each source from fused text prior;

[0162] Multi-scale code feature query: uses multi-scale code features in step 3 as query vectors;

[0163] Multi-source cross-attention calculation: calculates attention scores and outputs between code query and key-value pairs of four sources respectively;

[0164] Multi-source attention aggregation: integrates multi-source attention outputs through dynamically learned fusion weights and adds them to original multi-scale features.

[0165] Multi-layer feature fusion super-resolution backbone network: deeply fuses high-quality features obtained from code prediction module with super-resolution network to achieve high-quality image reconstruction.

[0166] Specific implementation process:

[0167] High-quality code feature decoding: converts predicted code index to corresponding codebook features, inputs decoder trained in step 1, and extracts 5 different levels of intermediate features.

[0168] TATT super-resolution backbone network: the baseline text super-resolution network TATT is used. TATT introduces a text prior module and a text structure consistency loss on the basis of TSRN.

[0169] Shallow feature extraction: the input image is processed by a shallow convolutional layer to obtain shallow features.

[0170] SRB module processing: input into a 5-layer SRB module. The SRB module is proposed in TSRN and is mainly used to learn the sequence information of text images. It introduces a recurrent neural network.

[0171] Up-sampling reconstruction: the output features of the last SRB module are processed by a PixelShuffle up-sampling layer to obtain an image with the same size as the high-resolution image.

[0172] Text prior fusion: TATT proposes a TPInterpreter module to interpret the previously obtained text prior features as image features, so that the prior features can be aligned and fused into the corresponding spatial positions of the image features. The TPInterpreter module inputs the shallow features into the Transfrmer decoder, and after the text prior is transformed into a new dimension, the output features are fused into the decoder through the Transfrmer encoder. The text prior feature map extracted by TPInterpreter is added to the intermediate features of each SRB to integrate the prior knowledge into the super-resolution network.

[0173] Multi-layer feature fusion module: the corresponding deterministic high-quality code features obtained by the code prediction module can be fused into the super-resolution backbone network to better perform super-resolution. The effective high-quality code features learned by the code prediction module are input into the decoder learned in the first stage to extract five intermediate features. These intermediate features are mapped to each SRB module, and the multi-head cross-attention is introduced to obtain the super-resolution network. At the same time, in order to let the high-quality code features also participate in this process, they are concatenated with each intermediate feature and introduced into the multi-head cross-attention. Specifically, the tken output by self-attention is used as the query vector, and the concatenated high-quality code features and intermediate features are used as the value and key vectors for attention calculation. The participation of high-quality code features in this feature fusion process is to enable the intermediate features of the super-resolution network to also select high-quality code features, without being affected by the intermediate features in the decoder.

[0174] PixelShuffle up-sampling: the final SRB features are up-sampled by PixelShuffle to obtain up-sampled features, and then a 3x3 convolution is performed to obtain the super-resolution image.

[0175] Progressive feature refinement: gradually refining text details at different levels of the network:

[0176] Low level: focus on edge and stroke recovery;

[0177] Mid-level: focus on character structure and arrangement;

[0178] High level: optimize overall semantic and visual quality.

[0179] Text structure consistency constraint: through the multi-source text prior module and text structure consistency loss, ensure the accuracy of the generated super-resolution image in structure and semantics.

[0180] Specific implementation process:

[0181] Multi-source TPInterpreter module: fuse the integrated text prior features and SRB features through TPInterpreter, TPInterpreter integrates text prior features into image features through attention mechanism, and adds residual connection.

[0182] Multi-level text structure consistency loss:

[0183] Character-level consistency: use CRNN to predict the text content of the super-resolution image and the high-resolution image respectively, and calculate the L2 loss;

[0184] Structural consistency: use edge detector to extract edge features of both, and calculate L2 loss;

[0185] Semantic consistency: use CLIP model to extract semantic features, and calculate cosine similarity loss.

[0186] Comprehensive loss function: the loss function of the super-resolution backbone network TATT consists of three losses. The first is the super-resolution loss, which measures the L1 difference between the super-resolution image and the corresponding high-resolution label image. The second is the gradient contour loss proposed by Wang et al. for text images, which calculates the L1 difference of the gradient fields of the two. The third is the text structure consistency loss.

[0187] Enhanced second stage overall loss function: combine multi-source text prior loss, text structure consistency loss, super-resolution loss, gradient contour loss, multi-scale code prediction loss and encoder feature constraint loss, and balance through different weight coefficients.

[0188] The weight settings of each loss term are as follows: the weight of multi-source text prior is 2.0, the weight of text structure consistency is 10.0, the weight of basic super-resolution is 1.0, the weight of gradient contour is 20.0, the weight of multi-scale code prediction is 1.5, and the weight of encoder feature constraint is 1.0.

[0189] Progressive training strategy: The second phase adopts a progressive training strategy, first training the basic code prediction module, then gradually introducing multi-scale mechanisms and multi-source text priors, and finally performing end-to-end joint optimization to ensure training stability and convergence.

[0190] Embodiment 2

[0191] Data preprocessing: According to the characteristics of license plate images, the input size is adjusted to 144x48 pixels to adapt to the length-width ratio of license plates. Character segmentation and background removal preprocessing are performed on the license plate images.

[0192] Enhanced multi-source prior knowledge customization:

[0193] 1. License plate rule prior library: According to the license plate coding rules of different regions, a license plate character constraint dictionary containing technical specifications, legal provisions, and standard documents is constructed, covering the license plate format specifications in China.

[0194] 2. Multi-scale font feature prior: Fine-grained level: Extract the stroke thickness, corner features, and character edges of license plate-specific fonts; character level: analyze the complete structure, height-width ratio, and symmetry features of individual characters; global level: model the overall layout, character spacing, and separator position rules of the license plate.

[0195] 3. Multi-source environment adaptation prior: Technical document source: license plate technical standard documents from traffic management departments; user manual source: license plate descriptions in user operation guides such as vehicle registration and annual inspection; forum Q&A source: user Q&A content about license plate recognition in automobile forums; code annotation source: code annotations and documents in open-source license plate recognition projects.

[0196] 4. Multi-modal environment adaptation: For complex environments such as license plate reflection, contamination, night, rain, and snow, an environment-text prior mapping mechanism is established.

[0197] Enhanced performance verification:

[0198] 1. License plate recognition performance improvement: Basic method: CRNN recognition accuracy benchmark level; single-scale code prediction method: improved; multi-scale code prediction method: further improved; multi-source text prior fusion method: improved; multi-scale + multi-source fusion method: significantly improved, with some improvement relative to the basic method.

[0199] 2. TextZm dataset performance improvement: basic TATT method: CRNN recognition accuracy benchmark level, PSNR and SSIM benchmark value; multi-scale code prediction enhancement: recognition accuracy improved, PSNR and SSIM improved; multi-source text prior enhancement: recognition accuracy further improved, PSNR and SSIM improved; comprehensive enhancement method: recognition accuracy significantly improved, PSNR and SSIM significantly improved.

[0200] 3. Performance analysis by scale: Easy test set: recognition accuracy improved; Medium test set: recognition accuracy improved; Hard test set: recognition accuracy improved.

[0201] 4. Multi-source text prior contribution analysis: only technical document prior improved; only user manual prior improved;

[0202] only forum Q&A prior improved; only code comment prior improved;

[0203] Four-source fusion: the effectiveness of multi-source fusion is verified.

[0204] Example 3: Enhanced fusion of TSRN framework.

[0205] Multi-scale code prediction module fusion:

[0206] 1. Insert multi-scale code prediction module after feature extraction stage of TSRN;

[0207] 2. Fuse prediction features of three scales (stroke level, character level, and global level) with different levels of TSRN features respectively;

[0208] 3. Realize effective integration of multi-level features through cross-scale attention mechanism.

[0209] Multi-source text prior enhancement:

[0210] 1. Construct special technical document and user manual prior library for TSRN;

[0211] 2. Enhance text understanding ability of TSRN through multi-source heterogeneous cross-attention mechanism;

[0212] 3. Introduce inter-source complementarity loss to avoid information redundancy.

[0213] Enhanced results:

[0214] Basic TSRN: CRNN recognition accuracy benchmark level;

[0215] Single-scale code prediction fusion: improved;

[0216] Multi-scale code prediction fusion: improved;

[0217] Multi-source text prior fusion: further improved;

[0218] Comprehensive enhancement fusion: significantly improved, relative to the basic method.

[0219] Enhanced fusion of TPGSR framework.

[0220] Multi-source text prior module replacement:

[0221] 1. Replace the original single text prior extraction method of TPGSR with a four-source heterogeneous text prior module;

[0222] 2. Keep the language prior and stroke prior of TPGSR, and add multi-source text prior as the third type of prior;

[0223] 3. Design a collaborative fusion mechanism for the three types of priors to achieve complementary enhancement.

[0224] Multi-scale deterministic feature fusion:

[0225] 1. Integrate the deterministic features obtained by multi-scale code prediction into the reconstruction process of TPGSR in a hierarchical manner;

[0226] 2. Introduce code features of corresponding scales at different reconstruction stages of TPGSR;

[0227] 3. Balance the contributions of the three types of priors through an adaptive weight learning mechanism.

[0228] Enhanced results:

[0229] Basic TPGSR: CRNN recognition accuracy benchmark level;

[0230] Multi-source text prior enhancement: improved;

[0231] Multi-scale code prediction enhancement: improved;

[0232] Comprehensive enhancement: significantly improved, relative to the basic method.

[0233] General applicability verification: through successful application in two different architectures of TSRN and TPGSR, the good universality and scalability of multi-scale code prediction and multi-source text prior module are verified, providing technical support for their application in more text super-resolution methods.

[0234] Example 4

[0235] Experimental environment configuration.

[0236] Hardware environment: The experiment was conducted on a server equipped with NVIDIA RTX3090 GPU, with 32GB of memory to ensure sufficient computing resources.

[0237] Software environment: PyTrch1.9.0 deep learning framework, CUDA11.1 version, Pythn3.8 environment were used.

[0238] Key parameter settings.

[0239] First stage parameters:

[0240] Codebook size K = 1024;

[0241] Feature dimension d = 512, dimension after dimension reduction 32;

[0242] Learning rate set to 1e-4, using Adam optimizer;

[0243] Batch size batch_size = 16;

[0244] Training rounds epchs = 100;

[0245] Balancing parameter set to 0.25, 1.0, 0.1, 1.0, 1.0.

[0246] Second stage enhanced parameters:

[0247] Learning rate set to 1e-4, using Adam optimizer;

[0248] Batch size batch_size = 8;

[0249] Training rounds epchs = 300 (increase training rounds to adapt to more complex model);

[0250] Enhanced balancing parameter set to 2.0, 10.0, 1.0, 20.0, 1.5, 1.0;

[0251] Multi-scale loss weight set to 0.3, 0.2 (consistency constraint weight);

[0252] Multi-source prior loss weight set to 0.1, 0.05, 0.15 (consistency, diversity, fusion quality weight).

[0253] Enhanced network architecture parameters:

[0254] Multi-scale code prediction module:

[0255] Transfrmer encoder layers: 6 layers;

[0256] Number of multi-head attention heads: 8 (for each scale branch);

[0257] Three-scale convolution kernel size: [1x1, 3x3], [5x5, 7x7], [11x11, Glbal];

[0258] Scale fusion weight learning network: 3-layer MLP, hidden layer dimension 256.

[0259] Multi-source text prior module:

[0260] Four special encoders: BERT-Tech, GPT-User, RBERTa-QA, CdeBERT;

[0261] Each encoder output dimension: 768, 1024, 768, 768;

[0262] Feature alignment projection layer: Linear mapping to unified 512 dimensions;

[0263] Number of layers of cross-attention between sources: 4 layers.

[0264] Fusion network parameters:

[0265] ViT patch size: 16x16 (first stage), 2x8 (second stage);

[0266] SRB module number: 5 layers (each layer integrates multi-scale feature fusion);

[0267] Multi-source text prior library size: technical document 50K, user manual 30K, forum question and answer 80K, code comment 40K.

[0268] Training strategy.

[0269] Data augmentation: random rotation (±5°), random scaling (0.9-1.1 times), random brightness adjustment, etc. Data augmentation strategy is used to improve the generalization ability of the model.

[0270] Learning rate scheduling: cosine annealing learning rate scheduling strategy is used, and the initial learning rate is gradually decayed during the training process.

[0271] Early stopping mechanism: monitor the PSNR index on the validation set, and when there is no improvement for 10 consecutive epochs, stop training in advance.

[0272] The above describes embodiments of the present application, but the embodiments are not limited to the specific implementation described above, which is only illustrative and not limiting. Those skilled in the art can make more forms of equivalent embodiments under the inspiration of the embodiments, which are all within the protection scope of the embodiments.

Claims

1. A scene image super-resolution method based on code prediction and text prior guidance, characterized in that, include: Construct a discrete feature space, collect high-resolution text images from public datasets, and use vector quantization techniques to map text image features onto a discrete codebook to form a shared basic representation of high-quality features; Based on the discrete feature space, a quantized autoencoder is trained and the ViT-VQGAN architecture is adopted, which includes an encoder, a learnable discrete codebook and a decoder. The codebook utilization is optimized and the image quality is reconstructed through split decoding and L2 normalization techniques. Based on a quantized autoencoder, a multi-scale code prediction mechanism is constructed. Given low-quality image features, a non-autoregressive Transformer encoder with a multi-scale attention mechanism predicts the index of the corresponding high-quality features in the codebook at different scales. Based on the code prediction mechanism, multi-source text prior guidance is introduced. The CRNN text recognition model, technical documents, user manuals, and programming forums are comprehensively utilized to integrate multi-source text prior knowledge into the code prediction mechanism through a multi-level cross-attention mechanism. By combining high-quality code features and decoders, a super-resolution backbone network is constructed. The high-quality code features obtained by the multi-scale code prediction mechanism are processed by multi-layer feature fusion and introduced into each layer of the super-resolution backbone network's SRB module to complete the final super-resolution reconstruction.

2. The method of scene image super-resolution based on code prediction and text prior guidance according to claim 1, characterized in that, The trained quantization autoencoder includes: Based on the discrete codebook, the encoder divides the input image into blocks and performs linear projection to obtain feature vectors. After adding position encoding, the input image is fed into the encoder to obtain feature vectors. The features are mapped to a low-dimensional latent space through decoding and normalization operations, the nearest discrete vector in the codebook is found, and after quantization, it is mapped back to a high-dimensional space. The quantized features are input into the decoder to reconstruct the image, providing an encoding-decoding reference for the code prediction mechanism; The encoder outputs high-quality text image feature representations through loss functions, including reconstruction loss, adversarial training mechanism, perceptual loss, and codebook space optimization loss.

3. The method of scene image super-resolution based on code prediction and text prior guidance according to claim 1, characterized in that, The multi-scale code prediction includes: Based on a quantized autoencoder, a non-autoregressive Transformer encoder with a multi-scale attention mechanism is used as the code prediction mechanism. Feature extraction branches are constructed at different scale levels to capture local detail features and global structural features respectively; Integrate feature information at different scales through a scale fusion attention mechanism; Input features are mapped to the codebook quantity dimension via a multi-scale classification head and trained using weighted cross-entropy loss. By introducing multi-scale L2 loss to constrain the encoder output, making it close to the output of a high-resolution image encoder at each scale, we obtain multi-scale code prediction results guided by text priors and fused with features.

4. The method of scene image super-resolution based on code prediction and text prior guidance according to claim 1, characterized in that, The multi-source text prior guidance includes: Based on the multi-scale code prediction results, the scaled low-resolution image is input into a learnable CRNN to obtain logits values, and then the basic text prior knowledge is obtained through softmax operation. Collect multi-source text resources related to the target code, such as technical documents, user manuals, and programming forum Q&A, and extract semantic features through a pre-trained language model; A hierarchical attention fusion mechanism is adopted, including intra-source attention and inter-source attention, to weight and fuse the multi-source text resources; High-resolution image input pre-trained CRNN obtains reference prior knowledge; A multi-source KL divergence loss is used to make the source prior distribution close to the reference distribution; In the code prediction mechanism, the fused multi-source text prior knowledge is used as the value and key vector, and the feature token output by the self-attention is used as the query vector, and a multi-level cross-attention mechanism is used to fuse the text prior knowledge to obtain high-quality code features to realize super-resolution reconstruction.

5. The method of scene image super-resolution based on code prediction and text prior guidance according to claim 1, characterized in that, The multi-layer feature fusion processing includes: The high-quality code features of the multi-scale code prediction mechanism are input into the decoder to extract multi-layer intermediate features; The intermediate features are introduced into each SRB module of the super-resolution backbone network through multi-head cross-attention; The high-quality code features and the intermediate features are concatenated and introduced into the multi-head cross-attention, so that the intermediate features of the super-resolution network use the high-quality code features for feature extraction; A scale adaptive mechanism is introduced at different levels to dynamically adjust the fusion weight according to the current layer feature resolution and the multi-scale prediction result to complete the final super-resolution reconstruction.

6. The method of scene image super-resolution based on code prediction and text prior guidance according to claim 1, characterized in that, It also includes a text structure consistency loss, including: The multi-source text prior mechanism and the text structure consistency loss are introduced into the super-resolution backbone network; The multi-source text prior feature map is extracted and fused into the image feature through the multi-source TP-Interpreter; A multi-level text structure consistency loss is used to constrain the text structure of the generated super-resolution image to be consistent with the real high-resolution image; The semantic consistency loss is introduced to make the reconstructed image consistent with the original image at the semantic level and form a collaborative optimization with the multi-source text prior information guidance.

7. The method of scene image super-resolution based on code prediction and text prior guidance according to claim 4, characterized in that, The encoder and the decoder in the ViT-VQGAN architecture are both based on the ViT encoder and add multi-scale position encoding to maintain consistency with the multi-scale code prediction architecture; The decomposition decoding operation is realized by decomposing the feature vector into multiple sub-vectors, and a scale perception mechanism is introduced to obtain the feature basis of multi-scale prediction; The L2 normalization technique is applied to the feature vector before quantization, and a dynamic codebook update strategy is used to optimize the utilization rate of the codebook, so that the code is accurately predicted.

8. The method of scene image super-resolution based on code prediction and text prior guidance according to claim 1, characterized in that, The super-resolution backbone network is constructed based on the TATT architecture, including: A shallow convolutional layer is used to extract initial features; A plurality of SRB modules are used for feature processing and enhancement, and each SRB module internally integrates a multi-head cross-attention mechanism and a scale adaptive fusion module; A PixelShuffle up-sampling layer is used for feature dimension transformation to complete super-resolution reconstruction; High-quality code features from the multi-scale code prediction mechanism and multi-layer intermediate features from the decoder are introduced into each SRB module; An incremental feature refinement mechanism is introduced to gradually refine text details and structure information at different levels of the network.

9. The method of scene image super-resolution based on code prediction and text prior guidance according to claim 1, characterized in that, The multi-scale attention mechanism includes: A local detail attention branch that focuses on capturing character stroke, edge local detail features; A global structure attention branch that focuses on capturing text line arrangement, overall layout global structure features; Scale fusion attention module dynamically fuses feature information of different scales through learnable weights; Multi-scale feature alignment module ensures the consistency of features of different scales in space and semantics.

10. The method of scene image super-resolution based on code prediction and text prior guidance according to claim 6, characterized in that, The acquisition and processing of the multi-source text prior information include: Technical document prior: extracting domain-specific text patterns and terms from relevant technical specifications and API documents; User manual prior: extracting common text expressions and format specifications from product usage instructions and operation manuals; Forum Q&A prior: extracting user-generated text content and expression habits from programming forums and technical communities; Semantically encoding various types of text priors through pre-trained BERT or GPT models; Using a hierarchical attention weight learning mechanism to dynamically adjust the contribution weights of each source prior according to the input image features.

Images