Text image super-resolution enhancement method, electronic device and storage medium

Abstract

The application discloses a text image super-resolution enhancement method fusing text edge, segmentation and semantic features, an electronic device and a storage medium. The text image super-resolution enhancement method comprises the following steps: acquiring edge features, segmentation features, semantic features and text image features of a low-resolution input text image; and performing fusion processing on the edge features, the segmentation features, the semantic features and the text image features to obtain a high-resolution text image. By introducing a visual-semantic prompt generator, the edge features and the segmentation features of the text image are fused with the semantic features respectively to generate two complementary text visual-semantic prompts to guide the reconstruction of the text image. By introducing a multi-prompt reconstruction module and a cross-modal attention mechanism, the information in the two text visual-semantic prompts is fused and enhanced with the text image features. After adaptive fusion and pixel up-sampling processing, the enhanced text image features obtain a super-resolution enhanced text image.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to a method for super-resolution enhancement of text images. Background Technology

[0002] Text appearing in images is an important source of information for people in their daily lives. However, a considerable number of text images have low resolution, which makes it very difficult for people to read the text information in the images, and on the other hand, it also seriously affects the accuracy of various text recognition methods in recognizing text images.

[0003] Recent text image super-resolution enhancement methods have effectively improved the resolution and clarity of text images, enhancing their visual quality. These methods can be broadly categorized into two types: methods for arbitrary images and methods specific to text images. Methods for arbitrary images treat text images as general images, using various forms of deep neural network models such as convolutional neural networks and generative adversarial networks to learn the mapping from low-resolution to high-resolution images, thereby reconstructing and generating high-resolution images based on the input low-resolution image. Methods specific to text images, building upon methods for arbitrary images, introduce various cues reflecting different aspects of text characteristics, such as visual or semantic cues, and utilize these cues to improve the quality of the generated text images.

[0004] Although existing text image super-resolution enhancement methods have achieved some good results, due to the limited variety of text cues used and the deficiencies in text cue modeling and utilization, the generated text images still often suffer from the loss or distortion of text details, such as blurred text edges or irregular text shapes. Therefore, they often fail to meet the needs of accurate text recognition in images. Summary of the Invention

[0005] The purpose of this invention is to provide a text image super-resolution enhancement method, electronic device, and storage medium that integrates text edge, segmentation, and semantic features.

[0006] This method differs from most existing text image super-resolution enhancement methods that primarily utilize the features of the text image itself. It introduces edge and segmentation features of the text image as supplementary visual cues, and effectively fuses them with the semantic features of the text image to generate two complementary visual-semantic cues to guide the reconstruction of the text image. Correspondingly, this method proposes a multi-cue reconstruction module and a cross-modal attention mechanism, using information from the visual-semantic cues to enhance and fuse the text image features, effectively improving the clarity and quality of the generated text image, thereby contributing to improved accuracy in text recognition within the image.

[0007] The present invention specifically adopts the following technical solution:

[0008] This invention first provides a text image super-resolution enhancement method that integrates text edge, segmentation, and semantic features, comprising the following steps:

[0009] Obtain edge features, segmentation features, semantic features, and text image features from low-resolution input text images;

[0010] By fusing edge features, segmentation features, semantic features, and text image features, a high-resolution text image is obtained.

[0011] The steps for fusing edge features, segmentation features, semantic features, and text image features to obtain a high-resolution text image include:

[0012] Edge features and segmentation features are input together with semantic features into a visual-semantic cue generator to obtain two types of text visual-semantic cues: one combining edge and semantic information, and the other combining segmentation and semantic information.

[0013] Two types of text visual-semantic cues and text image features are input into a multi-cue reconstruction module with multiple cascaded integrated cross-modal attention mechanisms for fusion and enhancement;

[0014] Finally, a high-resolution text image is obtained through the text image feature adaptive fusion module and pixel upsampling processing.

[0015] The multi-cue reconstruction module contains two parallel processing branches. Each branch uses one of two text visual-semantic cues to enhance the text image features, and then uses a cross-modal attention mechanism to further fuse and enhance the text image features obtained from the two branches.

[0016] The adaptive fusion module combines the two sets of text image features output by the final multi-cue reconstruction module, and then performs pixel upsampling processing to obtain the final high-resolution text image.

[0017] The visual-semantic cue generator takes the edge features, segmentation features, and semantic features of the text image as input. First, it applies deconvolutional and batch normalization layers to the semantic features to obtain a two-dimensional semantic feature map of the same size as the image's edge and segmentation features. Then, it applies deformable convolutional layers to improve the cue generator's ability to handle text deformation. Next, the visual-semantic cue generator introduces a sparse cross-attention module, which effectively combines the semantic features and edge or segmentation features of the text image and characterizes the correlation between them, while adaptively mitigating the impact of errors that may exist in the edge / segmentation map on feature matching and fusion, thereby generating accurate visual-semantic cuees. The processing of the sparse cross-attention module can be described as follows:

[0018]

[0019]

[0020] Where Top-K is the operation of sorting the input values ​​in descending order and taking the K highest values; S ij Φ is the value of the element in the i-th row and j-th column of matrix S; Top-K (S) is a matrix storing the results of Top-K operations on the values ​​in matrix S, with the values ​​at positions corresponding to non-Top-K operation results set to 0; [Φ Top-K (S)] ij For matrix Φ Top-K The value of the element in the i-th row and j-th column of (S); SPCA(Q,K,V) is the sparse cross-attention for the query feature matrix Q, the key feature matrix K, and the value feature matrix V; Softmax represents the softmax function; K T d represents the transpose of the key feature matrix K; K is the dimension of the feature vectors in the key feature matrix K. In the visual-semantic cue generator, the features in Q are derived from a two-dimensional semantic feature map after processing with deformable convolution, layer normalization, convolution, and channel-wise convolution. The features in K and V are derived from edge feature maps or segmentation feature maps after processing with layer normalization, convolution, and channel-wise convolution. The output of the sparse cross-attention module is further processed through convolutional layers, residual connections, and a feedforward neural network to obtain a text visual-semantic cue that combines edge or segmentation information and semantic information of the text image.

[0021] Each processing branch of the multi-cue reconstruction module consists of a sequential loop block enhanced with visual-semantic cues and a mutual-learning dynamic convolution block;

[0022] The sequential cyclic block for visual-semantic cue enhancement first adaptively fuses text image features and text visual-semantic cue features. The fusion process is described as follows:

[0023]

[0024] Among them, F out Indicates the characteristics after fusion; F represents the Hadamard product, Sigmoid represents the sigmoid layer, W is a learnable linear transformation matrix, and F... 1 in F 2 in F 3 inIt is the result of projecting the text image features and text visual-semantic cue features, which are concatenated along the channel dimension, into three different feature spaces using convolution operations respectively; the fused feature F out Then, after passing through two convolutional layers and a bidirectional long short-term memory network, the text image features are obtained after text visual-semantic cue enhancement.

[0025] The mutual learning dynamic convolutional block further uses the text visual cues characterized by the text image features obtained from another processing branch to supplement and enhance the text image features generated by the sequential loop block that enhances visual-semantic cues in this branch. In response to the shape characteristics of the text, the mutual learning dynamic convolutional block uses dynamic snake convolution to implement a cross-modal attention mechanism. First, based on the features obtained from another processing branch, a convolutional network is used to predict the sampling grid displacement parameters of the dynamic snake convolution. Then, the dynamic snake convolution is applied to the features of this processing branch, thereby effectively combining the information from the two processing branches.

[0026] The adaptive fusion module takes two sets of text image features output from the two processing branches of the last module in a cascaded multi-cue reconstruction module as input, and combines them using adaptive weights. The process is described as follows:

[0027] W F =Sigmoid(W1[F s ;F e ])

[0028] F = W F ⊙(W2[F s ;F e ])

[0029] Among them, W F F is the fusion weight matrix; F is the enhanced text image features; ⊙ represents element-wise multiplication, [] represents concatenation, Sigmoid represents the sigmoid layer, W1 and W2 are learnable linear transformation matrices, and F s and F e These are the features obtained from the two processing branches; the text image features F after fusion and enhancement are then processed by subsequent pixel upsampling to obtain a high-resolution text image.

[0030] The loss function for the super-resolution enhancement model is:

[0031] L=λ1L SR +λ2L TP

[0032] Where L is the total loss of the model; L SRFor super-resolution loss, the Euclidean norm of the difference between the high-resolution text image output by the model and the real high-resolution text image is calculated; L TP The text prior loss is calculated as the cross-entropy between the text recognition probability distribution obtained on a low-resolution text image using a pre-trained text recognizer and the true text category distribution; λ1 and λ2 are predefined weights.

[0033] The present invention also provides an electronic device, including a processor and a memory; the memory stores a program or instructions, which are loaded and executed by the processor to implement the above-described text image super-resolution enhancement method that integrates text edges, segmentation, and semantic features.

[0034] The present invention also provides a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the above-described text image super-resolution enhancement method that integrates text edges, segmentation, and semantic features.

[0035] The text image super-resolution enhancement method that integrates text edge, segmentation, and semantic features disclosed in this invention has the following advantages:

[0036] 1. The method introduces edge and segmentation features of text images as a supplement to image features, which more fully depicts the visual characteristics and appearance details of text. By combining edge and segmentation features with semantic features, it provides richer and more effective text clues for the reconstruction of text images.

[0037] 2. The method introduces a multi-cue reconstruction module for text images, which uses two parallel processing branches to fuse text visual-semantic cues that combine edge or segmentation information with semantic information with text image features. By cascading multiple multi-cue reconstruction modules, the feature representation of the text image is enhanced step by step and iteratively using multi-faceted cue information, thereby improving the details of the reconstructed text image.

[0038] 3. The method introduces a cross-modal attention mechanism between different processing branches of the multi-cue reconstruction module, which utilizes the complementarity between different text cue information to further enhance the text image features and improve the quality of the output high-resolution text image. Attached Figure Description

[0039] Figure 1 The overall structure diagram of the text image super-resolution enhancement model that integrates text edge, segmentation and semantic features;

[0040] Figure 2 A structural diagram of a visual-semantic cue generator;

[0041] Figure 3A structural diagram of sequential loop blocks for enhancing visual-semantic cues;

[0042] Figure 4 The structure diagram of the dynamic convolutional block for mutual learning. Detailed Implementation

[0043] The following describes in detail the text image super-resolution enhancement method of the present invention, which integrates text edge, segmentation, and semantic features, with reference to the accompanying drawings and specific embodiments, including the following steps:

[0044] (1) Construct a text image super-resolution enhancement model that integrates text edge, segmentation, and semantic features. The overall process of the model is as follows: Figure 1 The diagram shows the following sub-steps:

[0045] (1.1) The edge map and segmentation map of the input text image are obtained by using the Canny edge extraction operator and the pre-trained image segmentation network respectively. The corresponding edge features and segmentation features are obtained by using the convolutional network layer. On the other hand, the semantic features of the input text image are obtained by using the text recognizer.

[0046] (1.2) Construct a visual-semantic cue generator, the components of which are as follows: Figure 2 As shown in the diagram, the visual-semantic cue generator takes the edge features, segmentation features, and semantic features of the text image as input. First, it applies a deconvolutional layer and a batch normalization layer to the semantic features to obtain a two-dimensional semantic feature map of the same size as the image's edge and segmentation features. Then, it applies a deformable convolutional layer to improve the cue generator's ability to handle text deformation. Next, the visual-semantic cue generator introduces a sparse cross-attention module. This module effectively combines the semantic features and edge or segmentation features of the text image and characterizes the correlation between them, while adaptively mitigating the impact of errors that may exist in the edge / segmentation map on feature matching and fusion, thereby generating accurate visual-semantic cues. The processing of the sparse cross-attention module can be described as follows:

[0047]

[0048]

[0049] Top-K represents the operation of sorting the input values ​​in descending order and taking the K highest values; S ij Φ is the value of the element in the i-th row and j-th column of matrix S; Top-K (S) is a matrix storing the results of Top-K operations on the values ​​in matrix S, with the values ​​at positions corresponding to non-Top-K operation results set to 0; [Φ Top-K (S)] ij For matrix Φ Top-KThe value of the element in the i-th row and j-th column of (S); SPCA(Q,K,V) is the sparse cross-attention for the query feature matrix Q, the key feature matrix K, and the value feature matrix V; Softmax represents the softmax function; K T d represents the transpose of the key feature matrix K; K is the dimension of the feature vectors in the key feature matrix K. In the visual-semantic cue generator, the features in Q are derived from a two-dimensional semantic feature map processed by deformable convolution, layer normalization, convolution, and channel-wise convolution, while the features in K and V are derived from edge feature maps or segmentation feature maps processed by layer normalization, convolution, and channel-wise convolution. The output of the sparse cross-attention module is further processed through convolutional layers, residual connections, and a feedforward neural network to obtain a text visual-semantic cue that combines edge or segmentation information and semantic information from the text image.

[0050] (1.3) Construct a multi-prompt reconstruction module, the components of which are as follows: Figure 1 As shown in the diagram, the multi-cue reconstruction module comprises two parallel processing branches, each taking the features of the text image and one of two text visual-semantic cues as input. Each processing branch consists of a sequential recurrent block for visual-semantic cue enhancement and a mutual-learning dynamic convolutional block.

[0051] The composition of visual-semantic cue-enhanced sequential loop blocks is as follows: Figure 3 As shown in the diagram. This block first adaptively fuses text image features and text visual-semantic cue features. The fusion process is described below:

[0052]

[0053] in, F represents the Hadamard product, Sigmoid represents the sigmoid layer, W is a learnable linear transformation matrix, and F... 1 in F 2 in F 3 in This is the result of projecting the text image features and text visual-semantic cue features, which are concatenated along the channel dimension, into three different feature spaces using convolution operations. The fused feature F out Then, after passing through two convolutional layers and a bidirectional long short-term memory network, the text image features are obtained after text visual-semantic cues enhancement.

[0054] The composition of mutual learning dynamic convolutional blocks is as follows: Figure 4As shown in the diagram, this block uses textual visual cues characterized by textual image features obtained from another processing branch to supplement and enhance the textual image features generated by the sequential loop block of this branch, which enhances visual-semantic cues. For the shape characteristics of the text, the mutual learning dynamic convolution block implements a cross-modal attention mechanism using dynamic snake convolution. First, based on the features obtained from another processing branch, a convolutional network is used to predict the sampling grid displacement parameters of the dynamic snake convolution. Then, the dynamic snake convolution is applied to the features of this processing branch, thus effectively combining the information from both processing branches.

[0055] In this embodiment, the text image super-resolution enhancement model cascades five multi-cue reconstruction modules. Each module uses the two sets of text image features output by the previous module as one of the inputs to two processing branches. For the first multi-cue reconstruction module, the features extracted from the input text image are used as one of the inputs to the two processing branches.

[0056] (1.4) Constructing the Adaptive Fusion Module. The adaptive fusion module takes the two sets of text image features output from the two processing branches of the last module in a cascaded multi-cue reconstruction module as input, and combines them using adaptive weights. The process is described below:

[0057] W F =Sigmoid(W1[F s ;F e ])

[0058] F = W F ⊙(W2[F s ;F e ])

[0059] Where ⊙ denotes element-wise multiplication, [] denotes concatenation, Sigmoid represents the sigmoid layer, W1 and W2 are learnable linear transformation matrices, and F s Fe and W are features obtained from the two processing branches, respectively. F It is the fusion weight matrix. F is the text image feature after fusion enhancement, which is then processed by subsequent pixel upsampling to obtain a high-resolution text image.

[0060] (1.5) The loss function of the text image super-resolution enhancement model is calculated as follows:

[0061] L=λ1L SR +λ2L TP

[0062] Where L is the total loss of the model; L SRFor super-resolution loss, the Euclidean norm of the difference between the high-resolution text image output by the model and the real high-resolution text image is calculated. TP The text prior loss is calculated as the cross-entropy between the text recognition probability distribution obtained on a low-resolution text image using a pre-trained text recognizer and the true text category distribution. λ1 and λ2 are predefined weights, which can be set to 1.0 and 0.01, respectively.

[0063] (2) Training a text image super-resolution enhancement model that integrates text edge, segmentation, and semantic features includes the following sub-steps:

[0064] (2.1) For a low-resolution text image as an input model, the edge map and segmentation map of the input text image are first obtained by using the Canny edge extraction operator and a pre-trained image segmentation network, respectively. Then, convolutional network layers are used to obtain the corresponding edge features and segmentation features. On the other hand, a text recognizer is used to obtain the semantic features of the input text image. Then, the edge features and segmentation features are input together with the semantic features into a visual-semantic cue generator to obtain two types of text visual-semantic cues: one combining edge and semantic information, and the other combining segmentation and semantic information. These cues are then input into a multi-cue reconstruction module with multiple cascaded integrated cross-modal attention mechanisms to fuse and enhance the text image features. Finally, a high-resolution text image is obtained after adaptive fusion of text image features and pixel upsampling. Then, the backpropagation algorithm is used to train the text image super-resolution enhancement model end-to-end.

[0065] (2.2) Repeat step (2.1) so that the text image super-resolution enhancement model gradually learns the mapping relationship from the input low-resolution text image to the output high-resolution text image during the training process;

[0066] (3) For a low-resolution text image, input it into the trained text image super-resolution enhancement model, and the output of the model is a high-resolution text image.

[0067] This embodiment utilizes a 64-bit Ubuntu 18.04.3LTS operating system platform. Based on the PyTorch 1.8 deep learning framework and written in Python 3.6.9, a text-image super-resolution enhancement model is built, and training is performed using two NVIDIA Tesla V100 GPUs. The Adam optimizer is used during training, with the learning rate set to 0.001.

Claims

1. A text image super-resolution enhancement method that integrates text edge, segmentation, and semantic features, characterized in that, Includes the following steps: Obtain edge features, segmentation features, semantic features, and text image features from low-resolution input text images; By fusing edge features, segmentation features, semantic features, and text image features, a high-resolution text image is obtained. The visual-semantic cue generator takes the edge features, segmentation features, and semantic features of a text image as input. First, it applies deconvolutional and batch normalization layers to the semantic features to obtain a two-dimensional semantic feature map of the same size as the image's edge and segmentation features. Then, it applies deformable convolutional layers to improve the generator's ability to handle text deformation. Next, the visual-semantic cue generator introduces a sparse cross-attention module. This module effectively combines the semantic features and edge or segmentation features of the text image and characterizes the correlation between them, while adaptively mitigating the impact of errors that may exist in the edge / segmentation map on feature matching and fusion, thereby generating accurate visual-semantic cues. The processing of the sparse cross-attention module can be described as follows: Where Top-K is the operation of sorting the input values ​​in descending order and taking the K highest values; S ij Φ is the value of the element in the i-th row and j-th column of matrix S; T op- K (S) is a matrix storing the results of Top-K operations on the values ​​in matrix S, with the values ​​at positions corresponding to non-Top-K operation results set to 0; [Φ Top-K (S)] ij For matrix Φ Top-K The value of the element in the i-th row and j-th column of (S); SPCA(Q,K,V) is the sparse cross-attention for the query feature matrix Q, the key feature matrix K, and the value feature matrix V; Softmax represents the softmax function; K T d represents the transpose of the key feature matrix K; K is the dimension of the feature vectors in the key feature matrix K; in the visual-semantic cue generator, the features in Q are derived from a two-dimensional semantic feature map after processing with deformable convolution, layer normalization, convolution, and channel-wise convolution, while the features in K and V are derived from edge feature maps or segmentation feature maps after processing with layer normalization, convolution, and channel-wise convolution; the output of the sparse cross-attention module is further processed through convolutional layers, residual connections, and feedforward neural networks to obtain a text visual-semantic cue that combines edge or segmentation information and semantic information of the text image; Each processing branch of the multi-cue reconstruction module consists of a sequential recurrent block enhanced with visual-semantic cues and a mutually learning dynamic convolutional block; The sequential cyclic block for visual-semantic cue enhancement first adaptively fuses text image features and text visual-semantic cue features. The fusion process is described as follows: Among them, F out Indicates the characteristics after fusion; F represents the Hadamard product, Sigmoid represents the sigmoid layer, W is a learnable linear transformation matrix, and F... 1 in F 2 in F 3 in It is the result of projecting the text image features and text visual-semantic cue features, which are concatenated along the channel dimension, into three different feature spaces using convolution operations respectively; the fused feature F out Then, after passing through two convolutional layers and a bidirectional long short-term memory network, the text image features are obtained after text visual-semantic cues enhancement. The mutual learning dynamic convolutional block further uses the text visual cues characterized by the text image features obtained from another processing branch to supplement and enhance the text image features generated by the sequential loop block of the visual-semantic cue enhancement in this branch. In view of the shape characteristics of the text, the mutual learning dynamic convolutional block uses dynamic serpentine convolution to implement a cross-modal attention mechanism. First, based on the features obtained from another processing branch, a convolutional network is used to predict the sampling grid displacement parameters of the dynamic serpentine convolution. Then, the dynamic serpentine convolution is applied to the features of this processing branch, thereby effectively combining the information of the two processing branches. The steps for fusing edge features, segmentation features, semantic features, and text image features to obtain a high-resolution text image include: Edge features and segmentation features are input together with semantic features into a visual-semantic cue generator to obtain two types of text visual-semantic cues: one combining edge and semantic information, and the other combining segmentation and semantic information. Two types of text visual-semantic cues and text image features are input into a multi-cue reconstruction module with multiple cascaded integrated cross-modal attention mechanisms for fusion and enhancement; Finally, a high-resolution text image is obtained through the text image feature adaptive fusion module and pixel upsampling processing.

2. The text image super-resolution enhancement method fusing text edge, segmentation, and semantic features as described in claim 1, characterized in that, The multi-cue reconstruction module contains two parallel processing branches. Each branch uses one of two text visual-semantic cues to enhance the text image features, and then uses a cross-modal attention mechanism to further fuse and enhance the text image features obtained from the two branches.

3. The text image super-resolution enhancement method that integrates text edge, segmentation, and semantic features as described in claim 2, characterized in that, For the two sets of text image features output by the final multi-cue reconstruction module, the adaptive fusion module is used to fuse them together, and then pixel upsampling is performed to obtain the final high-resolution text image.

4. The text image super-resolution enhancement method fusing text edge, segmentation, and semantic features as described in claim 1, characterized in that, The adaptive fusion module takes two sets of text image features output from the two processing branches of the last module in a cascaded multi-cue reconstruction module as input, and combines them using adaptive weights. The process is described as follows: W F =Sigmoid(W1[F s ;F e ]) F=W F ⊙(W2[F s ;F e ]) Among them, W F F is the fusion weight matrix; F is the enhanced text image features; ⊙ represents element-wise multiplication, [] represents concatenation, Sigmoid represents the sigmoid layer, W1 and W2 are learnable linear transformation matrices, and F s and F e These are the features obtained from the two processing branches; the text image features F after fusion and enhancement are then processed by subsequent pixel upsampling to obtain a high-resolution text image.

5. The text image super-resolution enhancement method fusing text edge, segmentation, and semantic features as described in claim 4, characterized in that, The loss function for the super-resolution enhancement model is: L=λ1L SR +λ2L TP Where L is the total loss of the model; L SR For super-resolution loss, the Euclidean norm of the difference between the high-resolution text image output by the model and the real high-resolution text image is calculated; L TP The text prior loss is calculated as the cross-entropy between the text recognition probability distribution obtained on a low-resolution text image using a pre-trained text recognizer and the true text category distribution; λ1 and λ2 are predefined weights.

6. An electronic device, characterized in that, It includes a processor and a memory; the memory stores a program or instructions, which are loaded and executed by the processor to implement the text image super-resolution enhancement method according to any one of claims 1 to 5, which integrates text edges, segmentation, and semantic features.

7. A computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the text image super-resolution enhancement method according to any one of claims 1 to 5, which integrates text edges, segmentation, and semantic features.

Images