Text image super-resolution enhancement method, electronic equipment and storage medium

Abstract

The invention discloses a text image super-resolution enhancement method fusing text edge, segmentation and semantic features, electronic equipment and a storage medium. The text image super-resolution enhancement method comprises the following steps: acquiring edge features, segmentation features, semantic features and text image features of a low-resolution input text image; and carrying out fusion processing on the edge features, the segmentation features, the semantic features and the text image features to obtain a high-resolution text image. According to the method, a vision-semantic prompt generator is introduced, edge features and segmentation features of a text image are fused with semantic features respectively, and two complementary text vision-semantic prompts are generated to guide reconstruction of the text image. A multi-prompt reconstruction module and a cross-modal attention mechanism are introduced, and information in two text vision-semantic prompts and text image features are fused and enhanced. And the enhanced text image features are processed by the adaptive fusion module and pixel up-sampling processing to obtain a super-resolution enhanced text image.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and relates to a super-resolution enhancement method for text images. Background Art

[0002] Text that appears in images is an important source of information for people in their daily lives. However, many text images have low resolution, which makes it difficult for people to read the text information in the image. On the other hand, it also seriously affects the recognition accuracy of various text recognition methods for text images.

[0003] The text image super-resolution enhancement methods proposed in recent years can effectively improve the resolution and clarity of text images and improve their visual quality. These methods can basically be divided into two categories: methods for arbitrary images and methods specific to text images. The super-resolution enhancement methods for arbitrary images treat text images as general images, and use various forms of deep neural network models such as convolutional neural networks and generative adversarial networks to learn the mapping from low-resolution images to high-resolution images, so as to reconstruct and generate high-resolution images based on the input low-resolution images. Based on the methods for arbitrary images, the methods specific to text images introduce various clues that reflect different aspects of the text, such as visual or semantic clues of the text, and use these text clues to improve the quality of the generated text images.

[0004] Although existing text image super-resolution enhancement methods have achieved some good results, due to the singleness of the text clues used in the methods and the deficiencies in text clue modeling and utilization, the generated text images still often suffer from loss or distortion of text details, such as blurred text edges or irregular text shapes. Therefore, it is often difficult to meet the needs of accurate text recognition in images. Summary of the invention

[0005] The purpose of the present invention is to provide a text image super-resolution enhancement method, electronic device and storage medium that integrate text edge, segmentation and semantic features.

[0006] This method is different from most existing text image super-resolution enhancement methods that mainly use the characteristics of text images themselves. It introduces the edge and segmentation features of text images as supplements to text visual cues, and effectively fuses them with the semantic features of text images, generating two complementary text visual-semantic cues to guide the reconstruction of text images. Correspondingly, this method proposes a multi-cue reconstruction module and a cross-modal attention mechanism, which uses the information in the text visual-semantic cues to enhance and fuse the text image features, effectively improving the clarity and quality of the generated text images, thereby helping to improve the recognition accuracy of text in images.

[0007] The present invention specifically adopts the following technical solutions:

[0008] The present invention first provides a text image super-resolution enhancement method integrating text edge, segmentation and semantic features, comprising the following steps:

[0009] Obtain edge features, segmentation features, semantic features and text image features of low-resolution input text images;

[0010] The edge features, segmentation features, semantic features and text image features are fused to obtain a high-resolution text image.

[0011] The steps of fusing edge features, segmentation features, semantic features and text image features to obtain a high-resolution text image include:

[0012] The edge features and segmentation features are input into the visual-semantic cue generator together with the semantic features, and two types of text visual-semantic cues, which combine edge and semantic information and segmentation and semantic information, are obtained;

[0013] The two types of text visual-semantic cues and text image features are input into multiple cascaded multi-cue reconstruction modules with integrated cross-modal attention mechanisms for fusion and enhancement;

[0014] Finally, a high-resolution text image is obtained after a text-image feature adaptive fusion module and pixel upsampling processing.

[0015] The multi-cue reconstruction module contains two parallel processing branches, each of which uses one of the two text visual-semantic cues to enhance the text image features, and then uses a cross-modal attention mechanism to further fuse and enhance the text image features obtained from the two branches.

[0016] The adaptive fusion module fuses the two sets of text image features output by the final multi-prompt reconstruction module, and then obtains the final high-resolution text image through pixel upsampling.

[0017] The visual-semantic cue generator takes the edge features, segmentation features and semantic features of the text image as input, first applies a deconvolution layer and a batch normalization layer to the semantic features to obtain a two-dimensional semantic feature map of the same size as the edge and segmentation features of the image, and then applies a deformable convolution layer to it to improve the ability of the cue generator to cope with text deformation; then, the visual-semantic cue generator introduces a sparse cross-attention module, which effectively combines the semantic features and edge or segmentation features of the text image and characterizes the correlation between the two, while adaptively reducing the impact of errors that may exist in the edge / segmentation map on feature matching and fusion, thereby generating accurate text visual-semantic cues; the processing process of the sparse cross-attention module can be described as:

[0018]

[0019]

[0020] Among them, Top-K is the operation of sorting the input values ​​in descending order and taking the top K highest values; S ij is the value of the element in the i-th row and j-th column of the matrix S; Φ Top-K (S) is a matrix that stores the result values ​​of the Top-K operation on the values ​​in the matrix S. The values ​​of the other positions in the matrix corresponding to the results of non-Top-K operations are set to 0; [Φ Top-K (S)] ij is the matrix Φ Top-K The value of the element in the i-th row and j-th column in (S); SPCA(Q,K,V) is the sparse cross attention for the query feature matrix Q, the key feature matrix K and the value feature matrix V; Softmax represents the softmax function; K T represents the transpose of the key feature matrix K; d K is the dimension of the feature vector in the key feature matrix K. In the visual-semantic cue generator, the features in Q come from the two-dimensional semantic feature map after deformable convolution, layer normalization, convolution and channel-by-channel convolution, and the features in K and V come from the edge feature map or segmentation feature map after layer normalization, convolution and channel-by-channel convolution; the output of the sparse cross attention module is further processed through the convolution layer, residual connection and feedforward neural network to obtain the text visual-semantic cue combining the edge or segmentation information of the text image with the semantic information.

[0021] Each processing branch of the multi-cue reconstruction module consists of a sequential recurrent block for visual-semantic cue enhancement and a mutual learning dynamic convolution block;

[0022] The sequential recurrent block for visual-semantic cue enhancement first adaptively fuses the text image features and the text visual-semantic cue features. The fusion process is described as follows:

[0023]

[0024] Among them, F out Represents the fused features; represents the Hadamard product, Sigmoid represents the sigmoid layer, W is the learnable linear transformation matrix, and F 1 in 、F 2 in 、F 3 inIt is the result of projecting the text image features and text visual-semantic cue features cascaded along the channel dimension into three different feature spaces using convolution operations respectively; the fused feature F out Then, after two convolutional layers and a bidirectional long short-term memory network, the text image features enhanced by text visual-semantic cues are obtained;

[0025] The mutual learning dynamic convolution block further uses the text visual clues characterized by the text image features obtained by another processing branch to supplement and enhance the text image features generated by the sequential cycle block of the visual-semantic cue enhancement of this branch; in view of the shape characteristics of the text, the mutual learning dynamic convolution block uses dynamic snake convolution (Dynamic SnakeConvolution) to implement a cross-modal attention mechanism, in which a convolutional network is first used to predict the sampling grid displacement parameters of the dynamic snake convolution based on the features obtained by another processing branch, and then the dynamic snake convolution is applied to the features of this processing branch, thereby effectively combining the information of the two processing branches.

[0026] The adaptive fusion module takes two sets of text image features output by two processing branches of the last module in the cascade of multiple multi-prompt reconstruction modules as input, and combines the two using adaptive weights. The process is described as follows:

[0027] W F =Sigmoid(W1[F s ; F e ])

[0028] F=W F ⊙(W2[F s ; F e ])

[0029] Among them, W F is the fusion weight matrix; F is the text image feature after fusion enhancement; ⊙ represents the element-by-element multiplication operation, [] represents the concatenation operation, Sigmoid represents the sigmoid layer, W1 and W2 are learnable linear transformation matrices, and F s and F e They are the features obtained by the two processing branches respectively; the text image feature F after fusion enhancement is processed by subsequent pixel upsampling to obtain a high-resolution text image.

[0030] The loss function of the super-resolution enhancement model is:

[0031] L=λ1L SR +λ2L TP

[0032] Among them, L is the total loss of the model; L SRis the super-resolution loss, calculated as the Euclidean norm of the difference between the high-resolution text image output by the model and the true high-resolution text image; L TP is the text prior loss, calculated as the cross entropy between the text recognition probability distribution obtained on the low-resolution text image using the pre-trained text recognizer and the true text category distribution; λ1 and λ2 are predefined weights.

[0033] The present invention also provides an electronic device, comprising a processor and a memory; the memory stores a program or instruction, and the program or instruction is loaded and executed by the processor to implement the text image super-resolution enhancement method that integrates text edge, segmentation and semantic features provided above.

[0034] The present invention also provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the text image super-resolution enhancement method that integrates text edge, segmentation and semantic features provided above is implemented.

[0035] The text image super-resolution enhancement method integrating text edge, segmentation and semantic features disclosed in the present invention has the following advantages:

[0036] 1. The method introduces edge and segmentation features of text images as a supplement to image features, which more fully characterizes the visual characteristics and appearance details of the text. By combining edge and segmentation features with semantic features, it provides richer and more effective text clues for the reconstruction of text images.

[0037] 2. The method introduces a multi-cue reconstruction module for text images, in which two parallel processing branches are used to fuse the text visual-semantic cues that combine edge or segmentation information and semantic information with the text image features, and by cascading multiple multi-cue reconstruction modules, the feature representation of the text image is enhanced step by step and iteratively using multi-faceted cue information, thereby improving the details of the reconstructed text image.

[0038] 3. The method introduces a cross-modal attention mechanism between different processing branches of the multi-cue reconstruction module, utilizes the complementarity between different text cue information to further enhance the text image features, and improves the quality of the output high-resolution text images. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The overall structure diagram of the text image super-resolution enhancement model that integrates text edge, segmentation and semantic features;

[0040] Figure 2 The structure diagram of the visual-semantic cue generator;

[0041] Figure 3Structural diagram of sequential recurrent blocks enhanced for visual-semantic cues;

[0042] Figure 4 The structural diagram of the dynamic convolutional block for mutual learning. DETAILED DESCRIPTION

[0043] The text image super-resolution enhancement method integrating text edge, segmentation and semantic features of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments, including the following steps:

[0044] (1) Build a text image super-resolution enhancement model that integrates text edge, segmentation and semantic features. The overall process of the model is as follows: Figure 1 As shown in , it includes the following sub-steps:

[0045] (1.1) Use the Canny edge extraction operator and the pre-trained image segmentation network to obtain the edge map and segmentation map of the input text image, and further use the convolutional network layer to obtain the corresponding edge features and segmentation features. On the other hand, use the text recognizer to obtain the semantic features of the input text image;

[0046] (1.2) Construct a visual-semantic cue generator, which consists of Figure 2 As shown in . The visual-semantic cue generator takes the edge features, segmentation features, and semantic features of the text image as input. It first applies deconvolution layers and batch normalization layers to the semantic features to obtain a two-dimensional semantic feature map of the same size as the edge and segmentation features of the image, and then applies a deformable convolution layer to it to improve the cue generator's ability to cope with text deformation. Then, the visual-semantic cue generator introduces a sparse cross-attention module, which effectively combines the semantic features and edge or segmentation features of the text image and characterizes the correlation between the two, while adaptively reducing the impact of errors that may exist in the edge / segmentation map on feature matching and fusion, thereby generating accurate text visual-semantic cues. The processing process of the sparse cross-attention module can be described as:

[0047]

[0048]

[0049] Among them, Top-K is the operation of sorting the input values ​​in descending order and taking the top K highest values; S ij is the value of the element in the i-th row and j-th column of the matrix S; Φ Top-K (S) is a matrix that stores the result values ​​of the Top-K operation on the values ​​in the matrix S. The values ​​of the other positions in the matrix corresponding to the results of non-Top-K operations are set to 0; [Φ Top-K (S)] ij is the matrix Φ Top-KThe value of the element in the i-th row and j-th column in (S); SPCA(Q,K,V) is the sparse cross attention for the query feature matrix Q, the key feature matrix K and the value feature matrix V; Softmax represents the softmax function; K T represents the transpose of the key feature matrix K; d K is the dimension of the feature vector in the key feature matrix K. In the visual-semantic cue generator, the features in Q come from the two-dimensional semantic feature map after deformable convolution, layer normalization, convolution and channel-by-channel convolution, and the features in K and V come from the edge feature map or segmentation feature map after layer normalization, convolution and channel-by-channel convolution. The output of the sparse cross attention module is further processed through the convolution layer, residual connection and feedforward neural network to obtain the text visual-semantic cue that combines the edge or segmentation information of the text image with the semantic information.

[0050] (1.3) Construct a multi-prompt reconstruction module, which consists of Figure 1 As shown in . The multi-cue reconstruction module contains two parallel processing branches, each of which takes the features of the text image and one of the two text visual-semantic cues as input. Each processing branch consists of a sequential recurrent block enhanced with visual-semantic cues and a mutual learning dynamic convolution block.

[0051] The composition of the sequential recurrent block enhanced by visual-semantic cues is as follows Figure 3 As shown in . This block first adaptively fuses the text image features and the text visual-semantic cue features. The fusion process is described as follows:

[0052]

[0053] in, represents the Hadamard product, Sigmoid represents the sigmoid layer, W is the learnable linear transformation matrix, and F 1 in 、F 2 in 、F 3 in It is the result of projecting the text image features and text visual-semantic cue features cascaded along the channel dimension into three different feature spaces using convolution operations. The fused feature F out Subsequently, after two convolutional layers and a bidirectional long short-term memory network, the text image features enhanced by text visual-semantic cues are obtained.

[0054] The composition of the mutual learning dynamic convolution block is as follows Figure 4As shown in . This block uses the text visual clues characterized by the text image features obtained by another processing branch to supplement and enhance the text image features generated by the sequential cycle block of visual-semantic cue enhancement in this branch. In view of the shape characteristics of the text, the mutual learning dynamic convolution block uses dynamic snake convolution to implement a cross-modal attention mechanism, in which a convolutional network is first used to predict the sampling grid displacement parameters of the dynamic snake convolution based on the features obtained by another processing branch, and then the dynamic snake convolution is applied to the features of this processing branch, thereby effectively combining the information of the two processing branches.

[0055] In this embodiment, the text image super-resolution enhancement model cascades five multi-prompt reconstruction modules, each of which uses two sets of text image features output by the previous module as one of the inputs of the two processing branches. For the first multi-prompt reconstruction module, the features extracted from the input text image are used as one of the inputs of the two processing branches.

[0056] (1.4) Construct an adaptive fusion module. The adaptive fusion module takes the two sets of text image features output by the two processing branches of the last module in the cascade of multiple multi-prompt reconstruction modules as input and combines the two using adaptive weights. The process is described as follows:

[0057] W F =Sigmoid(W1[F s ; F e ])

[0058] F=W F ⊙(W2[F s ; F e ])

[0059] Among them, ⊙ represents the element-by-element multiplication operation, [] represents the concatenation operation, Sigmoid represents the sigmoid layer, W1 and W2 are learnable linear transformation matrices, and F s and Fe are the features obtained from the two processing branches, W F is the fusion weight matrix. F is the text image feature after fusion enhancement, and a high-resolution text image is obtained after subsequent pixel upsampling.

[0060] (1.5) The loss function of the text image super-resolution enhancement model is calculated as:

[0061] L=λ1L SR +λ2L TP

[0062] Among them, L is the total loss of the model; L SRL is the super-resolution loss, calculated as the Euclidean norm of the difference between the high-resolution text image output by the model and the true high-resolution text image. TP is the text prior loss, calculated as the cross entropy between the text recognition probability distribution obtained on the low-resolution text image using the pre-trained text recognizer and the true text category distribution. λ1 and λ2 are predefined weights that can be set to 1.0 and 0.01, respectively.

[0063] (2) Training a text image super-resolution enhancement model that integrates text edge, segmentation, and semantic features, including the following sub-steps:

[0064] (2.1) For a low-resolution text image of an input model, first use the Canny edge extraction operator and the pre-trained image segmentation network to obtain the edge map and segmentation map of the input text image, and then use the convolutional network layer to obtain the corresponding edge features and segmentation features. On the other hand, use the text recognizer to obtain the semantic features of the input text image. Then, the edge features and segmentation features are respectively input into the visual-semantic cue generator together with the semantic features to obtain two types of text visual-semantic cues that combine edge and semantic information and segmentation and semantic information. Then, they are input into multiple cascaded multi-cue reconstruction modules that integrate cross-modal attention mechanisms to fuse and enhance the text image features. Finally, a high-resolution text image is obtained after the text image feature adaptive fusion module and pixel upsampling processing. Subsequently, the back-propagation algorithm is used to train the text image super-resolution enhancement model end-to-end.

[0065] (2.2) Repeat step (2.1) so that the text image super-resolution enhancement model gradually learns the mapping relationship from the input low-resolution text image to the output high-resolution text image during the training process;

[0066] (3) For a low-resolution text image, it is input into the trained text image super-resolution enhancement model, and the output of the model is a high-resolution text image.

[0067] In this example, a text image super-resolution enhancement model was built on a 64-bit Ubuntu 18.04.3LTS operating system platform based on the PyTorch 1.8 deep learning framework and using Python 3.6.9 language, and two NVIDIA Tesla V100 GPUs were used to complete the model training. The training process used the Adam optimizer and set the learning rate to 0.001.

Claims

1. A text image super-resolution enhancement method integrating text edge, segmentation and semantic features, characterized in that: The following steps are involved: Obtain edge features, segmentation features, semantic features and text image features of low-resolution input text images; The edge features, segmentation features, semantic features and text image features are fused to obtain a high-resolution text image.

2. The text image super-resolution enhancement method integrating text edge, segmentation and semantic features as claimed in claim 1, characterized in that: The steps of fusing edge features, segmentation features, semantic features and text image features to obtain a high-resolution text image include: The edge features and segmentation features are input into the visual-semantic cue generator together with the semantic features, and two types of text visual-semantic cues, which combine edge and semantic information and segmentation and semantic information, are obtained; The two types of text visual-semantic cues and text image features are input into multiple cascaded multi-cue reconstruction modules with integrated cross-modal attention mechanisms for fusion and enhancement; Finally, a high-resolution text image is obtained after a text-image feature adaptive fusion module and pixel upsampling processing.

3. The text image super-resolution enhancement method integrating text edge, segmentation and semantic features as claimed in claim 2, characterized in that: The multi-cue reconstruction module contains two parallel processing branches, each of which uses one of the two text visual-semantic cues to enhance the text image features, and then uses a cross-modal attention mechanism to further fuse and enhance the text image features obtained from the two branches.

4. The text image super-resolution enhancement method integrating text edge, segmentation and semantic features as claimed in claim 3, characterized in that: For the two sets of text image features output by the final multi-prompt reconstruction module, the adaptive fusion module is used to fuse the two, and then the pixel upsampling process is performed to obtain the final high-resolution text image.

5. The text image super-resolution enhancement method integrating text edge, segmentation and semantic features as claimed in claim 2, characterized in that: The visual-semantic cue generator takes the edge features, segmentation features and semantic features of the text image as input, first applies a deconvolution layer and a batch normalization layer to the semantic features to obtain a two-dimensional semantic feature map of the same size as the edge and segmentation features of the image, and then applies a deformable convolution layer to it to improve the ability of the cue generator to cope with text deformation; then, the visual-semantic cue generator introduces a sparse cross-attention module, which effectively combines the semantic features and edge or segmentation features of the text image and characterizes the correlation between the two, while adaptively reducing the impact of errors that may exist in the edge / segmentation map on feature matching and fusion, thereby generating accurate text visual-semantic cues; the processing process of the sparse cross-attention module can be described as: Among them, Top-K is the operation of sorting the input values ​​in descending order and taking the top K highest values; S ij is the value of the element in the i-th row and j-th column of the matrix S; Φ Top-K (S) is a matrix that stores the result values ​​of the Top-K operation on the values ​​in the matrix S. The values ​​of the other positions in the matrix corresponding to the results of non-Top-K operations are set to 0; [Φ Top-K (S)] ij is the matrix Φ Top-K The value of the element in the i-th row and j-th column in (S); SPCA(Q,K,V) is the sparse cross attention for the query feature matrix Q, the key feature matrix K and the value feature matrix V; Softmax represents the softmax function; K T represents the transpose of the key feature matrix K; d K is the dimension of the feature vector in the key feature matrix K. In the visual-semantic cue generator, the features in Q come from the two-dimensional semantic feature map after deformable convolution, layer normalization, convolution and channel-by-channel convolution, and the features in K and V come from the edge feature map or segmentation feature map after layer normalization, convolution and channel-by-channel convolution; the output of the sparse cross attention module is further processed through the convolution layer, residual connection and feedforward neural network to obtain the text visual-semantic cue combining the edge or segmentation information of the text image with the semantic information.

6. The text image super-resolution enhancement method integrating text edge, segmentation and semantic features as claimed in claim 3, characterized in that: Each processing branch of the multi-cue reconstruction module consists of a sequential recurrent block for visual-semantic cue enhancement and a mutual learning dynamic convolution block; The sequential recurrent block for visual-semantic cue enhancement first adaptively fuses the text image features and the text visual-semantic cue features. The fusion process is described as follows: Among them, F out Represents the fused features; represents the Hadamard product, Sigmoid represents the sigmoid layer, W is the learnable linear transformation matrix, and F 1 in 、F 2 in 、F 3 in It is the result of projecting the text image features and text visual-semantic cue features cascaded along the channel dimension into three different feature spaces using convolution operations respectively; the fused feature F out Then, after two convolutional layers and a bidirectional long short-term memory network, the text image features enhanced by text visual-semantic cues are obtained; The mutual learning dynamic convolution block further uses the text visual clues characterized by the text image features obtained by another processing branch to supplement and enhance the text image features generated by the sequential cycle block of the visual-semantic cue enhancement of this branch; in view of the shape characteristics of the text, the mutual learning dynamic convolution block uses dynamic snake convolution to implement a cross-modal attention mechanism, in which a convolutional network is first used to predict the sampling grid displacement parameters of the dynamic snake convolution based on the features obtained by another processing branch, and then the dynamic snake convolution is applied to the features of this processing branch, thereby effectively combining the information of the two processing branches.

7. The text image super-resolution enhancement method integrating text edge, segmentation and semantic features as claimed in claim 6, characterized in that: The adaptive fusion module takes two sets of text image features output by two processing branches of the last module in the cascade of multiple multi-prompt reconstruction modules as input, and combines the two using adaptive weights. The process is described as follows: W F =Sigmoid(W1[F s ;F e ]) F=W F ⊙(W2[F s ;F e ]) Among them, W F is the fusion weight matrix; F is the text image feature after fusion enhancement; ⊙ represents the element-by-element multiplication operation, [] represents the concatenation operation, Sigmoid represents the sigmoid layer, W1 and W2 are learnable linear transformation matrices, and F s and F e They are the features obtained by the two processing branches respectively; the text image feature F after fusion enhancement is processed by subsequent pixel upsampling to obtain a high-resolution text image.

8. The text image super-resolution enhancement method integrating text edge, segmentation and semantic features as claimed in claim 7, characterized in that: The loss function of the super-resolution enhancement model is: L=λ1L SR +λ2L TP Among them, L is the total loss of the model; L SR is the super-resolution loss, calculated as the Euclidean norm of the difference between the high-resolution text image output by the model and the true high-resolution text image; L TP is the text prior loss, calculated as the cross entropy between the text recognition probability distribution obtained on the low-resolution text image using the pre-trained text recognizer and the true text category distribution; λ1 and λ2 are predefined weights.

9. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores a program or instruction, and the program or instruction is loaded and executed by the processor to implement the text image super-resolution enhancement method that integrates text edge, segmentation and semantic features as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the text image super-resolution enhancement method integrating text edge, segmentation and semantic features as described in any one of claims 1 to 8.

Images