A text image super-resolution method

Abstract

The present invention discloses a text image super-resolution method, comprising the steps of: 1. correcting the text image by using a spatial transformation network; 2. extracting text features in the corrected image by using a cyclic residual block; 3. fusing and up-sampling the text features; 4. generating clear and sharp text by using a discriminator-supervised super-resolution network. The method of the present invention has simple steps, reasonable design, and convenient implementation. It can be effectively applied to the super-resolution of text images, and has significant effects. The edges of the characters in the reconstructed text images are clearer and sharper. It has broad application prospects and is easy to promote.

Description

Technical Field

[0001] The invention belongs to the technical field of text recognition, and in particular relates to a text image super-resolution method. Background Art

[0002] Real-world text recognition has a wide range of practical applications, such as license plate recognition, document recognition, and text recognition. However, image acquisition devices have problems such as low resolution, poor illumination, and blurred focus, which reduce the recognition rate or even make it impossible to recognize. Therefore, it is still challenging to accurately recognize text in low-quality text images in real scenes.

[0003] Image super-resolution technology can improve image quality and is a feasible method to solve this problem. Image super-resolution refers to the process of restoring a high-resolution image from a low-resolution image or image sequence. In recent years, single image super-resolution technology (SISR) has made rapid progress. Most methods use high-definition images to obtain low-resolution image synthetic datasets after degradation processing to evaluate the performance of SISR models. Although the SISR method improves image quality, in real scenes, the low-resolution images obtained are much more complex, such as images taken with cameras of different focal lengths, and the main content of the image is complex and changeable. The previous SISR method is used to improve the quality of low-resolution text images, but it is difficult to extend it to the super-resolution of text images in real scenes because it only focuses on the reconstruction of texture details, while ignoring specific problems such as semantic information, background interference, and text shape in text images.

[0004] In the prior art, patent CN112991168A proposes a text super-resolution method, which extracts the features of the input image through training with a self-built data set, fuses the feature map with a feature pyramid, and finally upsamples to obtain a super-resolution result. This method applies the general super-resolution method to text image super-resolution, effectively improving the recognition rate of text images, but the reconstructed character edges are not clear and sharp enough, and there is no modeling for the contextual information of the text image, and the text sequence information cannot be restored well. Patent CN113256494A proposes a text super-resolution method based on deep learning, which uses a parallel dual-branch structure to capture visual contextual information and reconstructs text high-frequency information through adaptive weighted features. This method associates contextual semantic information, and the reconstructed text image is more realistic, but text images with variable characters are easily interfered by the background on the reconstructed characters, and the quality of the super-resolution image is low. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a text image super-resolution method in view of the deficiencies in the above-mentioned prior art. The method has simple steps, reasonable design, and easy implementation. It can be effectively applied to the super-resolution of text images, with significant effects. The edges of the reconstructed text image characters are clearer and sharper, and the method has broad application prospects and is easy to promote.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a text image super-resolution method, comprising the following steps:

[0007] Step 1: Use the spatial transformation network to correct the text image;

[0008] Step 2: Using a cyclic residual block to extract text features in the corrected image;

[0009] Step 3: Fusion up-sample text features;

[0010] Step 4: Use the discriminator to supervise the super-resolution network to generate clear and sharp text.

[0011] In the above-mentioned text image super-resolution method, the specific process of correcting the text image using the spatial transformation network in step 1 includes:

[0012] The spatial transformer network STN is used as the center alignment module for alignment. The alignment process is as follows:

[0013] V=T stn (U), V∈R H×W×C ,U∈R H′×W′×C′

[0014] Among them, U represents the input low-resolution image LR, V represents the aligned image, and T stn (U) represents the STN operation on U, R H×W×C Represents the number field vector of the input low-resolution image, R H′×W′×C′ A number field vector representing the rectified low-resolution image.

[0015] In the above-mentioned text image super-resolution method, the cyclic residual block in step 2 includes a convolutional layer, a multi-scale feature extraction module and a multi-dimensional attention module.

[0016] The above-mentioned text image super-resolution method extracts shallow features through the convolution layer, and the specific process includes:

[0017] The image corrected by the STN network is sent to two convolution layers with a convolution kernel of 3X3 to extract shallow features. The formula is expressed as:

[0018] F=T conv (V), V∈R H×W×C

[0019] Among them, F represents the feature map after two layers of convolution, T conv (V) represents two convolution operations on V and the normalization and prelu activation function after the convolution.

[0020] The above-mentioned text image super-resolution method uses five residual block basic units to extract text image features. The specific process includes:

[0021] Input F into five residual block basic units to extract text image features. The formula is expressed as:

[0022] R=T rsb (F)

[0023] Among them, T rsb (F) represents the cyclic residual block operation on F, and R represents the feature map obtained after five identical residual block structures.

[0024] The above-mentioned text image super-resolution method extracts more detailed text features through the multi-scale feature extraction module, and the specific process includes:

[0025] Input R into the multi-scale feature extraction module. The multi-scale feature extraction module structure simulates the receptive field of human vision and uses convolution kernels of different scales of 5X5, 3X3, and 1X1 and convolution layers of multiple branches to enhance the feature extraction capability. The formula is expressed as:

[0026] R RFB =T rfb (R)

[0027] Among them, R RFB It represents the feature map after the feature fusion of the multi-scale feature extraction module, T rfb (R) represents the convolution operation under the multi-scale feature extraction module of R.

[0028] The above-mentioned text image super-resolution method enables the network to pay more attention to the text area of ​​the image and reduce background interference through the multi-dimensional attention module.

[0029] In the above-mentioned text image super-resolution method, the specific process of fusing and up-sampling text features in step 3 includes:

[0030] The extracted shallow features F and the output R of the cyclic residual structure and multi-scale feature extraction module structure are RFB The fusion is used as the input of CARAFE upsampling, and the formula is expressed as:

[0031] O deep =T carafe (R RFB)

[0032] Among them, O deep represents the image obtained by upsampling after the cyclic residual block, T carafe (R RFB ) indicates R RFB CARAFE upsampling operation;

[0033] Then input the first convolution layer F of the network into CARAFE upsampling, and compare the result with O deep After fusion, the image of the super-resolution network is finally output after dimension transformation. The formula is expressed as:

[0034] O shallow =T carafe (F)

[0035] O sr =T conv (O deep +O shallow )

[0036] Among them, O shallow Represents the result of the first layer of convolution after CARAFE upsampling, O sr represents the SR image output after fusion upsampling and convolution dimension transformation, T carafe (F) represents the CARAFE upsampling operation on F, T conv (O deep +O shallow ) represents the fusion of deep upsampling and shallow upsampling.

[0037] In the above-mentioned text image super-resolution method, the specific process of using the discriminator to supervise the super-resolution network to generate clear and sharp text in step 4 includes:

[0038] Taking the LR and HR images as the input of the discriminator, by alternately training the generator and the discriminator, a SR image with more realistic details is generated. The formula is expressed as:

[0039]

[0040] Where x represents the HR image, z represents the noise input to the generator G, G(z) represents the image generated by the generator, D(x) represents the probability that the discriminator judges whether the HR image is real, and D(G(z)) represents the probability that the discriminator judges whether the image generated by the generator is real. represents the expected value of the true distribution x~Pdata, Represents the random noise distribution z~p z The expected value of (z).

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1. The method of the present invention has simple steps, reasonable design and is easy to implement.

[0043] 2. The present invention adds a multi-scale feature fusion module to the residual block of the super-resolution network to extract more detailed image information.

[0044] 3. The present invention adds a multi-dimensional attention module to the residual block of the super-resolution network. The self-attention is used to maintain the semantic information dependency of the text in the image. The ECA-Net captures cross-channel interactions. The SA spatial attention focuses on the meaningful areas in the text image. By fusing channel and spatial attention, the importance of pixels in different channels and the importance of pixels at different positions in the same channel are taken into account at the same time, thereby reducing the interference of the text background area on the text reconstruction.

[0045] 4. The present invention adds a lightweight CARAFE upsampling method to the network. Its larger receptive field makes better use of the surrounding information, thereby obtaining a reconstructed SR image with higher recognition accuracy than the PixelShuffle upsampling. On this basis, in order to reduce the loss of text image information, the features that have been sent to the network and have undergone the first convolution are upsampled and fused with the results of upsampling on the super-resolution generation network, and finally the reconstructed SR image is output.

[0046] 5. The present invention can be effectively applied to the super-resolution of text images with remarkable effects. The edges of characters in the reconstructed text images are clearer and sharper. The invention has broad application prospects and is easy to promote.

[0047] In summary, the method of the present invention has simple steps, reasonable design, and easy implementation. It can be effectively applied to the super-resolution of text images with remarkable effects. The edges of the characters in the reconstructed text images are clearer and sharper. It has broad application prospects and is easy to promote.

[0048] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the method of the present invention;

[0050] Figure 2 It is a network model diagram of the present invention;

[0051] Figure 3 A detailed diagram of the super-resolution network in the network model of the present invention;

[0052] Figure 4 It is a comparison diagram of the effects of the present invention. DETAILED DESCRIPTION

[0053] like Figure 1 to Figure 3As shown, the text image super-resolution method of the present invention comprises the following steps:

[0054] Step 1: Use the spatial transformation network to correct the text image;

[0055] Step 2: Using a cyclic residual block to extract text features in the corrected image;

[0056] Step 3: Fusion up-sample text features;

[0057] Step 4: Use the discriminator to supervise the super-resolution network to generate clear and sharp text.

[0058] In this embodiment, the specific process of correcting the text image using the spatial transformation network in step 1 includes:

[0059] The spatial transformer network STN is used as the center alignment module for alignment. The alignment process is as follows:

[0060] V=T stn (U), V∈R H×W×C ,U∈R H′×W′×C′

[0061] Among them, U represents the input low-resolution image LR, V represents the aligned image, and T stn (U) represents the STN operation on U, R H×W×C Represents the number field vector of the input low-resolution image, R H′×W′×C′ A number field vector representing the rectified low-resolution image.

[0062] In this embodiment, the cyclic residual block in step 2 includes a convolutional layer, a multi-scale feature extraction module and a multi-dimensional attention module.

[0063] In this embodiment, shallow features are extracted through the convolution layer, and the specific process includes:

[0064] The image corrected by the STN network is sent to two convolution layers with a convolution kernel of 3X3 to extract shallow features. The formula is expressed as:

[0065] F=T conv (V), V∈R H×W×C

[0066] Among them, F represents the feature map after two layers of convolution, T conv (V) represents two convolution operations on V and the normalization and prelu activation function after the convolution.

[0067] In this embodiment, five residual block basic units are used to extract text image features, and the specific process includes:

[0068] Input F into five residual block basic units to extract text image features. The formula is expressed as:

[0069] R=T rsb (F)

[0070] Among them, T rsb (F) represents the cyclic residual block operation on F, and R represents the feature map obtained after five identical residual block structures.

[0071] In this embodiment, more detailed text features are extracted by the multi-scale feature extraction module, and the specific process includes:

[0072] Input R into the multi-scale feature extraction module. The multi-scale feature extraction module structure simulates the receptive field of human vision and uses convolution kernels of different scales of 5X5, 3X3, and 1X1 and convolution layers of multiple branches to enhance the feature extraction capability. The formula is expressed as:

[0073] R RFB =T rfb (R)

[0074] Among them, R RFB It represents the feature map after the feature fusion of the multi-scale feature extraction module, T rfb (R) represents the convolution operation under the multi-scale feature extraction module of R.

[0075] In this embodiment, the multi-dimensional attention module enables the network to pay more attention to the text area of ​​the image and reduce background interference.

[0076] In the specific implementation, a multi-dimensional attention module is added to the residual block of the super-resolution network. The semantic information dependency of the text in the image is maintained through self-attention, ECA-Net captures cross-channel interactions, and SA spatial attention focuses on meaningful areas in the text image. By fusing channel and spatial attention, the importance of pixels in different channels and the importance of pixels at different positions in the same channel are taken into account at the same time, thereby reducing the interference of the text background area on text reconstruction.

[0077] In this embodiment, the specific process of fusing and upsampling the text features in step 3 includes:

[0078] The extracted shallow features F and the output R of the cyclic residual structure and multi-scale feature extraction module structure are RFB The fusion is used as the input of CARAFE upsampling, and the formula is expressed as:

[0079] O deep =T carafe (R RFB )

[0080] Among them, O deep represents the image obtained by upsampling after the cyclic residual block, T carafe (R RFB ) indicates R RFB CARAFE upsampling operation;

[0081] Then input the first convolution layer F of the network into CARAFE upsampling, and compare the result with O deep After fusion, the image of the super-resolution network is finally output after dimension transformation. The formula is expressed as:

[0082] O shallow =T carafe (F)

[0083] O sr =T conv (O deep +O shallow )

[0084] Among them, O shallow Represents the result of the first layer of convolution after CARAFE upsampling, O sr represents the SR image output after fusion upsampling and convolution dimension transformation, T carafe (F) represents the CARAFE upsampling operation on F, T conv (O deep +O shallow ) represents the fusion of deep upsampling and shallow upsampling.

[0085] In the specific implementation, a lightweight CARAFE upsampling method is added to the network. Its larger receptive field makes better use of the surrounding information, thereby obtaining a reconstructed SR image with higher recognition accuracy than the PixelShuffle upsampling. On this basis, in order to reduce the loss of text image information, the features that have been sent to the network after the first convolution are upsampled and fused with the results of upsampling on the super-resolution generation network, and finally the reconstructed SR image is output.

[0086] In this embodiment, the specific process of using the discriminator to supervise the super-resolution network to generate clear and sharp text in step 4 includes:

[0087] Taking the LR and HR images as the input of the discriminator, by alternately training the generator and the discriminator, a SR image with more realistic details is generated. The formula is expressed as:

[0088]

[0089] Where x represents the HR image, z represents the noise input to the generator G, G(z) represents the image generated by the generator, D(x) represents the probability that the discriminator judges whether the HR image is real, and D(G(z)) represents the probability that the discriminator judges whether the image generated by the generator is real. represents the expected value of the true distribution x~Pdata, Represents the random noise distribution z~p z The expected value of (z).

[0090] In specific implementation, the discriminator consists of feature extraction and classification. The feature extraction consists of four blocks, each of which has two convolutional layers. The convolutional layers in the four blocks increase the number of channels to 64, 128, 256, and 512 respectively. Each convolutional layer is followed by a ReLu layer and a Batch Normalization layer. The output of the feature extraction part is the classification feature f e The classification part includes a global average pooling layer and two fully connected layers. The classification part inputs f e , output prediction score z, the output of the discriminator is normalized by Sigmoid function and converted into probability form.

[0091] In order to verify the effect of the method of the present invention, the previous method and the method of the present invention were experimentally compared in a real data set. The results are as follows: Figure 4 As shown. Figure 4 It can be seen that the character edges of the text image reconstructed by the method of the present invention are clearer, and the characters and glyphs are closer to the HR image, achieving a better visual effect.

[0092] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A text image super-resolution method, characterized in that: The following steps are involved: Step 1: Use the spatial transformation network to correct the text image; The specific process of correcting the text image using the spatial transformation network in step 1 includes: The spatial transformer network STN is used as the center alignment module for alignment. The alignment process is as follows: V=T stn (U),V∈R H×W×C ,U∈R H′×W′×C′ Among them, U represents the input low-resolution image LR, V represents the aligned image, and T stn (U) represents the STN operation on U, R H ×W×C Represents the number field vector of the input low-resolution image, R H′×W′×C′ A number field vector representing the rectified low-resolution image; Step 2: Using a cyclic residual block to extract text features in the corrected image; The cyclic residual block in step 2 includes a convolutional layer, a multi-scale feature extraction module and a multi-dimensional attention module; Step 3: Fusion up-sample text features; The specific process of fusing and upsampling text features described in step 3 includes: The extracted shallow features F and the output R of the cyclic residual structure and multi-scale feature extraction module structure are RFB The fusion is used as the input of CARAFE upsampling, and the formula is expressed as: O deep =T carafe (R RFB ) Among them, O deep represents the image obtained by upsampling after the cyclic residual block, T carafe (R RFB ) indicates R RFB CARAFE upsampling operation; Then input the first convolution layer F of the network into CARAFE upsampling, and compare the result with O deep After fusion, the image of the super-resolution network is finally output after dimension transformation. The formula is expressed as: O shallow =T carafe (F) O sr =T conv (O deep +O shallow ) Among them, O shallow Represents the result of the first layer of convolution after CARAFE upsampling, O sr represents the SR image output after fusion upsampling and convolution dimension transformation, T carafe (F) represents the CARAFE upsampling operation on F, T conv (O deep +O shallow ) represents the fusion of deep upsampling and shallow upsampling; Step 4: Use the discriminator to supervise the super-resolution network to generate clear and sharp text; The specific process of using the discriminator to supervise the super-resolution network to generate clear and sharp text in step 4 includes: Taking the LR and HR images as the input of the discriminator, by alternately training the generator and the discriminator, a SR image with more realistic details is generated. The formula is expressed as: m G inm D axV(D,G)=E x～pdata [logD(x)]+E z～pz(z) [log(1-D(G(z)))] Where x represents the HR image, z represents the noise input to the generator G, G(z) represents the image generated by the generator, D(x) represents the probability that the discriminator judges whether the HR image is real, D(G(z)) represents the probability that the discriminator judges whether the image generated by the generator is real, and E x～pdata represents the expected value of the true distribution x~Pdata, E z～pz(z) Represents the random noise distribution z~p z The expected value of (z).

2. A text image super-resolution method according to claim 1, characterized in that: The shallow features are extracted through the convolution layer. The specific process includes: The image corrected by the STN network is sent to two convolution layers with a convolution kernel of 3X3 to extract shallow features. The formula is expressed as: F=T conv (V),V∈R H×W×C Among them, F represents the feature map after two layers of convolution, T conv (V) represents two convolution operations on V and the normalization and prelu activation function after the convolution.

3. A text image super-resolution method according to claim 2, characterized in that: Five residual block basic units are used to extract text image features. The specific process includes: Input F into five residual block basic units to extract text image features. The formula is expressed as: R=T rsb (F) Among them, T rsb (F) represents the cyclic residual block operation on F, and R represents the feature map obtained after five identical residual block structures.

4. A text image super-resolution method according to claim 3, characterized in that: The multi-scale feature extraction module is used to extract more detailed text features, and the specific process includes: Input R into the multi-scale feature extraction module. The multi-scale feature extraction module structure simulates the receptive field of human vision and uses convolution kernels of different scales of 5X5, 3X3, and 1X1 and convolution layers of multiple branches to enhance the feature extraction capability. The formula is expressed as: R RFB =T rfb (R) Among them, R RFB It represents the feature map after the feature fusion of the multi-scale feature extraction module, T rfb (R) represents the convolution operation under the multi-scale feature extraction module of R.

5. A text image super-resolution method according to claim 4, characterized in that: The multi-dimensional attention module enables the network to pay more attention to the text area of ​​the image and reduce background interference.

Images