Super-resolution reconstruction method for real-world Chinese and English mixed text image based on double-window attention

Abstract

The invention relates to a method for realizing super-resolution reconstruction for a real-world Chinese and English mixed text image based on double-window attention, and the method comprises the steps: collecting a data set of the super-resolution of the real-world Chinese and English mixed text image, and carrying out the corresponding preprocessing operation; alternately adopting dense window attention and sparse window attention, extracting local features and modeling global dependence; a deep convolution branch parallel to a window attention branch is introduced, and interaction is carried out among different windows; designing a space-channel interaction module, and carrying out bidirectional fusion and interaction on the two branch features from a space dimension and a channel dimension; a front feedback network based on fast Fourier transform is constructed, and the learning ability of important frequency information is enhanced by learning a quantization matrix in a frequency domain space; fusing the output characteristics of each residual group; pixel loss and edge perception loss training is adopted, and the network is optimized. Compared with an existing frontier method, the super-resolution reconstruction method has a better super-resolution reconstruction effect.

Description

Technical Field

[0001] The present invention relates to the field of digital image technology, in particular to the field of computer vision technology, and specifically refers to a super-resolution reconstruction method for real-world Chinese-English mixed text images based on dual window attention. Background Art

[0002] Image super-resolution aims to reconstruct a high-resolution HR image with rich detail information from an input low-resolution LR image, and has a wide range of applications in various fields. Text image super-resolution focuses on text images and aims to reconstruct a blurred LR text image into a clear HR text image, thereby providing support for downstream tasks such as text recognition.

[0003] The datasets used for early text image super-resolution research are usually synthetic datasets, where the LR images are obtained by subjecting the corresponding HR images to specific degradation operations (such as blurring, noise, and downsampling). However, the degradation forms in the real world are more complex and changeable, and artificially predefined degradation methods cannot simulate the real degradation conditions well. Therefore, the models trained on synthetic datasets have poor generalization performance and cannot solve the complex degradation problems in actual application scenarios.

[0004] To address this problem, Wang et al. proposed the first paired real-world text image super-resolution dataset TextZoom, whose LR-HR image pairs are taken by cameras with different focal lengths. However, the TextZoom dataset only contains English text images of fixed resolution, while the number of English character categories is small and the stroke structure is simple. Therefore, the super-resolution model trained on the TextZoom dataset has poor reconstruction effect when processing Chinese characters with complex structures, and cannot be generalized to text images of various resolutions. Ma et al. proposed a real-world mixed Chinese and English text image super-resolution dataset Real-CE. This dataset contains a large number of Chinese text images with more diverse image sizes. Compared with English characters, Chinese characters have more categories and complex stroke structures. In addition, the Real-CE dataset can support 2× and 4× super-resolution tasks.

[0005] Most of the existing real-world text image super-resolution methods are aimed at English text images, using text recognition priors to assist in the SR reconstruction of characters. However, when the text image encounters severe degradation, the high-level semantic prior does not work, especially for characters with complex glyph structures, such as Chinese characters. In addition, most mainstream super-resolution methods are based on CNN or Transformer. CNN has a limited receptive field, and although Transformer can model global dependencies, it has a quadratic computational complexity. In summary, the existing text image super-resolution methods cannot effectively balance the global receptive field and computational complexity, nor can they reconstruct Chinese text images with complex character structures. Summary of the invention

[0006] The purpose of the present invention is to address the shortcomings of existing methods and propose a super-resolution reconstruction method for real-world Chinese and English mixed text images based on dual window attention, which effectively models local features and global dependencies by alternating between dense window attention and sparse window attention. In addition, the present invention can effectively enhance the text area by introducing Canny edge map and edge-aware loss.

[0007] In order to achieve the above objectives, the present invention implements a super-resolution reconstruction method of real-world Chinese and English mixed text images based on dual window attention as follows:

[0008] The method for super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention is mainly characterized in that the method comprises the following steps:

[0009] (1) Collect a real-world Chinese-English mixed text image super-resolution dataset and perform corresponding preprocessing operations on the text images;

[0010] (2) Extract shallow features from the input four-channel low-resolution image, then gradually extract deep features from the shallow features through multiple residual groups, then fuse the output features of each residual group and the shallow features, and finally build an image reconstruction module to reconstruct a four-channel super-resolution image;

[0011] (3) Construct a dense window attention module to extract local features;

[0012] (4) Construct a sparse window attention module to establish global dependencies;

[0013] (5) Using pixel loss and edge-aware loss to train and optimize the super-resolution network.

[0014] Preferably, the step (1) specifically comprises the following steps:

[0015] (1.1) Perform data enhancement on training images through geometric transformation and brightness and contrast adjustment;

[0016] (1.2) Divide the training image into 64×64 image blocks as the input of the super-resolution network.

[0017] Preferably, the step (2) specifically comprises the following steps:

[0018] (2.1) Given a low-resolution LR image Use the Canny edge detector to extract the Canny edge map of the LR image And concatenate it with the LR image to get a 4-channel input image Where H and W are the height and width of the input image respectively;

[0019] (2.2) A 3×3 convolutional layer is used to input the 4-channel image I in Extract shallow features from Where C represents the number of feature channels;

[0020] (2.3) Through 6 residual groups, deep features are gradually extracted from the shallow features F0. The output feature of the i-th residual group is recorded as

[0021] (2.4) The output features of the six residual groups are concatenated to obtain Then use 1×1 convolution to transform F concat The number of channels is reduced to C, and then the features are further refined through 3×3 convolution, and finally added pixel by pixel with the shallow features to obtain the final fusion feature

[0022] (2.5) Construct an image reconstruction module from the input fusion feature F fused A four-channel super-resolution image is reconstructed by: a 3×3 convolution layer is used to enhance the expressiveness of the features, and then a sub-pixel convolution layer is used to upsample the features to the target size, and finally a 3×3 convolution layer is used to obtain a four-channel output image. Among them I out is a 3-channel RGB super-resolution image And the corresponding Canny edge map splicing, r represents the upsampling multiple.

[0023] Preferably, the step (3) specifically comprises the following steps:

[0024] (3.1) Perform layer normalization on input features to stabilize the training process;

[0025] (3.2) Construct a dense window attention branch, the output of which is recorded as

[0026] (3.3) Construct a deep convolution branch and use 3×3 depthwise separable convolution to interact between different windows. The output of this branch is recorded as

[0027] (3.4) Construct the space-channel interaction module to DWA and Y DWC For bidirectional fusion and interaction, this module contains parallel channel attention branches and spatial attention branches;

[0028] (3.5) Construct a fast Fourier transform-based feedforward network to enhance the ability to learn important frequency information.

[0029] More preferably, the step (3.2) specifically comprises the following steps:

[0030] (3.2.1) Window division: Sample tokens at adjacent positions of the input feature map and divide the feature map into windows of size S×S. The number of windows is

[0031] (3.2.2) Attention calculation: Multi-head self-attention calculation is performed in each window. For the feature of the i-th window Flatten its dimensions, and we get Three linear mapping layers are then used to generate the query matrix key matrix and value matrix Then Q i , K i 、V i The three matrices are input into the multi-head self-attention mechanism MSA, as shown in the following formula:

[0032]

[0033] Where T represents the matrix transpose operation, Y i represents the attention result of the i-th window, and d represents the number of feature channels;

[0034] (3.2.3) Window recovery: The attention calculation results of N windows are recombined according to the window division method to obtain the final attention output

[0035] More preferably, the step (3.4) specifically comprises the following steps:

[0036] (3.4.1) Input to the spatial attention branch to generate a spatial attention map with pixel values ​​between 0 and 1

[0037] (3.4.2) Input to the channel attention branch to generate a channel attention map

[0038] (3.4.3) The output of each branch is multiplied by the input of the other branch: Y Channel With feature map Y DWA Perform element-wise multiplication to weight the channel attention to the feature map Y DWA ; Y Spatial With feature map Y DWC Perform element-wise multiplication to weight the spatial attention to the feature map Y DWC ;

[0039] (3.4.4) Add the two weighted feature maps element by element to get the final output of the spatial-interaction module

[0040] Y A =Y Channel ⊙Y DWA +Y Spatial ⊙Y DWC

[0041] More preferably, the step (3.5) specifically comprises the following steps:

[0042] (3.5.1) Input features through 1×1 convolution Perform channel expansion;

[0043] (3.5.2) Divide the feature map into windows with a window size of 8×8;

[0044] (3.5.3) Perform a two-dimensional fast Fourier transform in each window to obtain the spectrum feature F f , as shown in the following formula:

[0045]

[0046] Among them, H and W represent the height and width of the window respectively;

[0047] (3.5.4) Initialize a learnable frequency domain matrix W and the spectrum feature F f Perform element-by-element multiplication and update the parameters of W during network training to adaptively enhance important frequency domain information;

[0048] (3.5.5) Perform a two-dimensional inverse fast Fourier transform in each window to transform the feature map from the frequency domain to the spatial domain;

[0049] (3.5.6) Reassemble the windows in the manner of window division to restore the size of the input feature F;

[0050] (3.5.7) After 3×3 convolution, GELU activation function, 1×1 convolution, and then added to the input feature F, the final output result is obtained.

[0051] Preferably, the step (4) specifically comprises the following steps:

[0052] (4.1) Perform layer normalization on input features to stabilize the training process;

[0053] (4.2) Construct a sparse window attention branch, the output of which is recorded as

[0054] (4.3) Construct a deep convolution branch and use 3×3 depthwise separable convolution to interact between different windows. The output of this branch is recorded as

[0055] (4.4) Construct the space-channel interaction module to SWA and Y D ' WC Carry out two-way integration and interaction;

[0056] (4.5) Construct a fast Fourier transform-based feedforward network to adaptively enhance the learning ability of important frequency information by learning the quantization matrix in the frequency domain.

[0057] More preferably, the step (4.2) specifically comprises the following steps:

[0058] (4.2.1) Window division: Sample tokens at intervals I on the input feature map and divide the feature map into windows of size The number of windows N = I 2 ;

[0059] (4.2.2) Perform multi-head self-attention calculation in each window: for the feature of the i-th window Flatten its dimensions, and we get Three linear mapping layers are then used to generate the query matrix key matrix and value matrix Then Q i , K i 、V i The three matrices are input into the multi-head self-attention mechanism MSA, as shown in the following formula:

[0060]

[0061] Where T represents the matrix transpose operation, Y i represents the attention result of the i-th window, and d represents the number of feature channels;

[0062] (4.2.3) Window recovery: The attention calculation results of N windows are recombined according to the window division method to obtain the final attention output

[0063] More preferably, the step (4.4) specifically comprises the following steps:

[0064] (4.4.1) Output features Input to the spatial attention branch to generate a spatial attention map with pixel values ​​between 0 and 1

[0065] (4.4.2) Output features Input to the channel attention branch to generate a channel attention map

[0066] (4.4.3) Multiply the output of each branch by the input of the other branch: Y C ' hannel With feature map Y SWA Perform element-wise multiplication to weight the channel attention to the feature map Y SWA ; Y S ' pacial With feature map Y D ' WC Perform element-wise multiplication to weight the spatial attention to the feature map Y D ' WC ;

[0067] (4.4.4) Add the two weighted feature maps element by element to obtain the final output of the space-channel interaction module

[0068] Y A ′=Y C ' hannel ⊙Y SWA +Y S ' patial ⊙Y D ' WC .

[0069] Preferably, the step (4.5) specifically comprises the following steps:

[0070] (4.5.1) Input features through 1×1 convolution Perform channel expansion;

[0071] (4.5.2) Divide the feature map into windows with a window size of 8×8;

[0072] (4.5.3) Perform a two-dimensional fast Fourier transform in each window to obtain the spectrum feature F f , as shown in the following formula:

[0073]

[0074] Among them, H and W represent the height and width of the window respectively;

[0075] (4.5.4) Initialize a learnable frequency domain matrix W and the spectrum feature F f Perform element-by-element multiplication and update the parameters of W during network training to adaptively enhance important frequency domain information;

[0076] (4.5.5) Perform a two-dimensional inverse fast Fourier transform in each window to transform the feature map from the frequency domain to the spatial domain;

[0077] (4.5.6) Reassemble the windows in the same way as the windows are divided to restore the input features Size;

[0078] (4.5.7) After 3×3 convolution, GELU activation function, 1×1 convolution, and then with the input features Add together to get the final output result

[0079] Preferably, the step (5) specifically comprises the following steps:

[0080] (5.1) Calculate the pixel loss L between the super-resolution SR image and the high-resolution HR image pixel :

[0081] L pixel =||I Hr -I SR ||1

[0082] (5.2) Calculate the edge perception loss L edge : Use the pre-trained VGG19 network to extract image features and edge features of HR images and SR images respectively, and then use the edge features to weight the image features to enhance the text area. edge The specific calculation formula is as follows:

[0083] L edge =||F(I HR )·F(M HR )-F(I SR )·F(M SR )||1

[0084] Among them, F(·) is the pre-trained feature extractor, namely the VGG19 network, I HR , I SR Represent HR image and SR image respectively, M HR , M SR Represents the corresponding Canny edge map;

[0085] (5.3) The balance parameter λ is introduced to balance the weights of the two loss terms. The training loss calculation formula is as follows:

[0086] L=L pixel +λ·L edge

[0087] The present invention is used to implement a super-resolution reconstruction method for real-world Chinese and English mixed text images based on dual window attention. In the deep feature extraction stage, dense window attention and sparse window attention are alternately applied to capture local context and model global dependencies, respectively. A deep convolution branch parallel to the window attention branch is introduced to interact between different windows. A space-channel interaction module is designed to adaptively fuse and interact the two branches in both spatial and channel dimensions. A fast Fourier transform-based feedforward network is constructed, and the learning ability of important frequency domain information is adaptively enhanced by learning the quantization matrix in the frequency domain space. This technical solution is experimentally verified on the Real-CE dataset, and compared with the cutting-edge methods, a better super-resolution reconstruction effect is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a schematic diagram of the overall structure of the present invention.

[0089] Figure 2 It is a structural schematic diagram of the window attention branch of the present invention.

[0090] Figure 3 It is a structural schematic diagram of the space-channel interaction module of the present invention.

[0091] Figure 4 It is a structural schematic diagram of the fast Fourier transform-based feedforward network of the present invention.

[0092] Figure 5 In one specific embodiment of the present invention, DA 2 Visualization results of super-resolution reconstruction of FE model and comparison model. DETAILED DESCRIPTION

[0093] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0094] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.

[0095] See also Figure 1 The method for super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention comprises the following steps:

[0096] (1) Collect a real-world Chinese-English mixed text image super-resolution dataset and perform corresponding preprocessing operations on the text images;

[0097] (2) Extract shallow features from the input four-channel low-resolution image, then gradually extract deep features from the shallow features through multiple residual groups, then fuse the output features of each residual group and the shallow features, and finally build an image reconstruction module to reconstruct a four-channel super-resolution image;

[0098] (3) Construct a dense window attention module to extract local features;

[0099] (4) Construct a sparse window attention module to establish global dependencies;

[0100] (5) Using pixel loss and edge-aware loss to train and optimize the super-resolution network.

[0101] As a preferred embodiment of the present invention, the step (1) specifically comprises the following steps:

[0102] (1.1) Perform data enhancement on training images through geometric transformation and brightness and contrast adjustment;

[0103] (1.2) Divide the training image into 64×64 image blocks as the input of the super-resolution network.

[0104] In practical applications, the above step (1) is specifically as follows:

[0105] Download the Real-CE dataset from the dataset website to obtain high-resolution and low-resolution image pairs. Then, perform data augmentation on the training images through geometric transformation (including image flipping and rotation) and brightness and contrast adjustment, and divide them into 64×64 image blocks as the input of the super-resolution network.

[0106] As a preferred embodiment of the present invention, the step (2) specifically comprises the following steps:

[0107] (2.1) Given a low-resolution LR image Use the Canny edge detector to extract the Canny edge map of the LR image And concatenate it with the LR image to get a 4-channel input image Where H and W are the height and width of the input image respectively;

[0108] (2.2) A 3×3 convolutional layer is used to input the 4-channel image I in Extract shallow features from Where C represents the number of feature channels;

[0109] (2.3) Through 6 residual groups, deep features are gradually extracted from the shallow features F0. The output feature of the i-th residual group is recorded as

[0110] (2.4) The output features of the six residual groups are concatenated to obtain Then use 1×1 convolution to transform F comcat The number of channels is reduced to C, and then the features are further refined through 3×3 convolution, and finally added pixel by pixel with the shallow feature F0 to obtain the final fusion feature

[0111] (2.5) Construct an image reconstruction module from the input fusion feature F fused A four-channel super-resolution image is reconstructed by using a 3×3 convolution layer to enhance the expressiveness of features, and then a sub-pixel convolution layer is used to upsample the features to the target size, and finally a 3×3 convolution layer is used to output a four-channel image. Among them I out is a 3-channel RGB super-resolution image And the corresponding Canny edge map splicing, r represents the upsampling multiple.

[0112] As a preferred embodiment of the present invention, the step (3) specifically comprises the following steps:

[0113] (3.1) Perform layer normalization on input features to stabilize the training process;

[0114] (3.2) Construct a dense window attention branch, the output of which is recorded as

[0115] (3.3) Construct a deep convolution branch and use 3×3 depthwise separable convolution to interact between different windows. The output of this branch is recorded as

[0116] (3.4) Construct the space-channel interaction module to DWAand Y DWC For bidirectional fusion and interaction, this module contains parallel channel attention branches and spatial attention branches;

[0117] (3.5) Construct a fast Fourier transform-based feedforward network to adaptively enhance the learning ability of important frequency information by learning the quantization matrix in the frequency domain.

[0118] As a preferred embodiment of the present invention, the following step (3.2) specifically includes the following steps:

[0119] (3.2.1) Window division: Sample tokens at adjacent positions of the input feature map and divide the feature map into windows of size S×S. The number of windows is

[0120] (3.2.2) Perform multi-head self-attention calculation in each window: for the feature of the i-th window Flatten its dimensions, and we get Three linear mapping layers are then used to generate the query matrix key matrix and value matrix Then Q i , K i 、V i The three matrices are input into the multi-head self-attention mechanism MSA, as shown in the following formula:

[0121]

[0122] Where T represents the matrix transpose operation, Y i Represents the attention result of the i-th window;

[0123] (3.2.3) Window recovery: The attention calculation results of N windows are recombined according to the window division method to obtain the final attention output

[0124] As a preferred embodiment of the present invention, the step (3.4) specifically includes the following steps:

[0125] (3.4.1) Input to the spatial attention branch to generate a spatial attention map with pixel values ​​between 0 and 1

[0126] (3.4.2) Input to the channel attention branch to generate a channel attention map

[0127] (3.4.3) The output of each branch is multiplied by the input of the other branch: YChannel With feature map Y DWA Perform element-wise multiplication to weight the channel attention to the feature map Y DWA ; Y Spatial With feature map Y DWC Perform element-wise multiplication to weight the spatial attention to the feature map Y DWC ;

[0128] (3.4.4) Add the two weighted feature maps element by element to get the final output of the spatial-interaction module

[0129] Y A =Y Channel ⊙Y DWA +Y Spatial ⊙Y DWC

[0130] As a preferred embodiment of the present invention, the following step (3.5) specifically includes the following steps:

[0131] (3.5.1) Input features through 1×1 convolution Perform channel expansion;

[0132] (3.5.2) Divide the feature map into windows with a window size of 8×8;

[0133] (3.5.3) Perform a two-dimensional fast Fourier transform in each window to obtain the spectrum feature F f , as shown in the following formula:

[0134]

[0135] Among them, H and W represent the height and width of the window respectively;

[0136] (3.5.4) Initialize a learnable frequency domain matrix W and the spectrum feature F f Perform element-by-element multiplication and update the parameters of W during network training to adaptively enhance important frequency domain information;

[0137] (3.5.5) Perform a two-dimensional inverse fast Fourier transform in each window to transform the feature map from the frequency domain to the spatial domain;

[0138] (3.5.6) Reassemble the windows in the manner of window division to restore the size of the input feature F;

[0139] (3.5.7) After 3×3 convolution, GELU activation function, 1×1 convolution, and then added to the input feature F, the final output result is obtained.

[0140] As a preferred embodiment of the present invention, the step (4) specifically comprises the following steps:

[0141] (4.1) Perform layer normalization on input features to stabilize the training process;

[0142] (4.2) Construct a sparse window attention branch, the output of which is recorded as

[0143] (4.3) Construct a deep convolution branch and use 3×3 depthwise separable convolution to interact between different windows. The output of this branch is recorded as

[0144] (4.4) Construct the space-channel interaction module to SWA and Y D ' WC Carry out two-way integration and interaction;

[0145] (4.5) Construct a fast Fourier transform-based feedforward network to adaptively enhance the learning ability of important frequency information by learning the quantization matrix in the frequency domain.

[0146] As a preferred embodiment of the present invention, the step (4.2) specifically includes the following steps:

[0147] (4.2.1) Window division: Sample tokens at intervals I on the input feature map and divide the feature map into windows of size The number of windows N = I 2 ;

[0148] (4.2.2) Attention calculation: Multi-head self-attention calculation is performed in each window. For the feature of the i-th window Flatten its dimensions, and we get Three linear mapping layers are then used to generate the query matrix key matrix and value matrix Then Q i , K i 、V i The three matrices are input into the multi-head self-attention mechanism MSA, as shown in the following formula:

[0149]

[0150] Where T represents the matrix transpose operation, Y i represents the attention result of the i-th window, and d represents the number of feature channels;

[0151] (4.2.3) Window recovery: The attention calculation results of N windows are recombined according to the window division method to obtain the final attention output

[0152] As a preferred embodiment of the present invention, the step (4.4) specifically comprises the following steps:

[0153] (4.4.1) Output features Input to the spatial attention branch to generate a spatial attention map with pixel values ​​between 0 and 1

[0154] (4.4.2) Output features Input to the channel attention branch to generate a channel attention map

[0155] (4.4.3) Multiply the output of each branch by the input of the other branch: Y C ' hannel With feature map Y SWA Perform element-wise multiplication to weight the channel attention to the feature map Y SWA ; Y S ' pacial With feature map Y D ' WC Perform element-wise multiplication to weight the spatial attention to the feature map Y D ' WC ;

[0156] (4.4.4) Add the two weighted feature maps element by element to obtain the final output of the space-channel interaction module

[0157] Y A ′=Y C ' hannel ⊙Y SWA +Y S ' patial ⊙Y D ' WC .

[0158] As a preferred embodiment of the present invention, the step (4.5) specifically comprises the following steps:

[0159] (4.5.1) Input features through 1×1 convolution Perform channel expansion;

[0160] (4.5.2) Divide the feature map into windows with a window size of 8×8;

[0161] (4.5.3) Perform a two-dimensional fast Fourier transform in each window to obtain the spectrum feature F f , as shown in the following formula:

[0162]

[0163] Among them, H and W represent the height and width of the window respectively;

[0164] (4.5.4) Initialize a learnable frequency domain matrix W and the spectrum feature F f Perform element-by-element multiplication and update the parameters of W during network training to adaptively enhance important frequency domain information;

[0165] (4.5.5) Perform a two-dimensional inverse fast Fourier transform in each window to transform the feature map from the frequency domain to the spatial domain;

[0166] (4.5.6) Reassemble the windows in the same way as the windows are divided to restore the input features Size;

[0167] (4.5.7) After 3×3 convolution, GELU activation function, 1×1 convolution, and then with the input features Add together to get the final output result

[0168] As a preferred embodiment of the present invention, the step (5) specifically comprises the following steps:

[0169] (5.1) Calculate the pixel loss L between the super-resolution SR image and the high-resolution HR image pixel :

[0170] L pixel =||I HR -I SR ||1

[0171] (5.2) Calculate the edge perception loss L edge : Use the pre-trained VGG19 network to extract image features and edge features of HR images and SR images respectively, and then use the edge features to weight the image features to enhance the text area. edge The specific calculation formula is as follows:

[0172] L edge =||F(I HR )·F(M HR )-F(I SR )·F(M SR )||1

[0173] Among them, F(·) is the pre-trained feature extractor, namely the VGG19 network, IHR , I SR Represent HR image and SR image respectively, M HR , M SR Represents the corresponding Canny edge map;

[0174] (5.3) The balance parameter λ is introduced to balance the weights of the two loss terms. The training loss calculation formula is as follows:

[0175] L=L pixel +λ·L edge

[0176] In a specific embodiment of the present invention, the super-resolution reconstruction method based on dual window attention for real-world Chinese and English mixed text images is specifically tested as follows:

[0177] (1) Dataset

[0178] The present invention uses the Real-CE dataset proposed by Ma et al. in ICCV 2023, which contains a large number of Chinese and English text images. When making the Real-CE dataset, iPhone11pro and iPhone12pro with lens focal lengths of 13mm, 26mm and 52mm were used to shoot text images. In the same shooting scene, image pairs shot with 13mm and 26mm, image pairs shot with 26mm and 52mm are used for 2x super-resolution tasks, and image pairs shot with 13mm and 52mm are used for 4x super-resolution tasks. The Real-CE training set contains 1,935 LR-HR image pairs, and the test set contains 783 LR-HR image pairs, of which 522 image pairs are used for 2x super-resolution tasks and 261 image pairs are used for 4x super-resolution tasks. At the same time, the Real-CE dataset provides annotations such as detection boxes and text labels.

[0179] (2) Training process

[0180] The input image is divided into image blocks of size 64×64, and the training images are augmented by geometric transformation and brightness and contrast adjustment. The batch size (BatchSize) of the training is 32, and the learning rate is set to 0.0002. The Adam optimizer is used for back propagation, and the parameters β1=0.9 and β2=0.999 are set. A total of 500,000 iterations are trained.

[0181] (3) Test results

[0182] Table 1 is the DA of the method of the present invention 2After super-resolution reconstruction of the Real-CE test set by FE and existing cutting-edge methods, CRNN is used to identify the reconstructed SR images using the recognition accuracy (ACC) and normalized edit distance (NED), as well as the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0183] like Figure 5 As shown, this is the DA proposed in this scheme 2 Visualization results of super-resolution reconstruction of FE model and comparison model. The first column shows the result of LR image after bicubic upsampling. The second and third columns show the super-resolution results of comparison models RRDB and ELAN. The fourth column shows the DA proposed in this scheme. 2 The result of FE model super-resolution. The last column shows the high-resolution HR image. In order to highlight the contrast effect, a rectangular box is used to mark the local area on each image and enlarge it for display. At the same time, CRNN is used to recognize the text in the rectangular box. The experimental results show that the DA proposed in this paper 2 Compared with the comparison model, the FE model can restore clear text areas with richer details, thus effectively improving the text recognition performance.

[0184] Table 1 Experimental results

[0185]

[0186]

[0187] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0188] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0189] A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0190] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0191] In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0192] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0193] The super-resolution reconstruction method of real-world Chinese-English mixed text images based on dual window attention of the present invention is adopted. In the deep feature extraction stage, dense window attention and sparse window attention are alternately applied to capture local context and model global dependencies, respectively. A deep convolution branch parallel to the window attention branch is introduced to interact between different windows. A space-channel interaction module is designed to adaptively fuse and interact the two branch features in both spatial and channel dimensions. A fast Fourier transform-based feedforward network is constructed, and the learning ability of important frequency domain information is adaptively enhanced by learning the quantization matrix in the frequency domain space. This technical solution is experimentally verified on the Real-CE dataset, and compared with the cutting-edge methods, a better super-resolution reconstruction effect is achieved.

[0194] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A super-resolution reconstruction method for real-world Chinese and English mixed text images based on dual window attention, characterized in that: The method comprises the following steps: (1) Collect a real-world Chinese-English mixed text image super-resolution dataset and perform corresponding preprocessing operations on the text images; (2) Extract shallow features from the input four-channel low-resolution image, then gradually extract deep features from the shallow features through multiple residual groups, and fuse the output features of each residual group and the shallow features to construct an image reconstruction module to reconstruct a four-channel super-resolution image; (3) Construct a dense window attention module to extract local features; (4) Construct a sparse window attention module to establish global dependencies on input features; (5) Using pixel loss and edge-aware loss to train and optimize the super-resolution network.

2. The method for super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 1 is characterized in that: The step (1) is specifically as follows: (1.1) Perform data enhancement on training images through geometric transformation and brightness and contrast adjustment; (1.2) Divide the training image into 64×64 image blocks as the input of the super-resolution network.

3. The method for super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 2 is characterized in that: The step (2) specifically comprises the following steps: (2.1) Given a low-resolution LR image Among them, H, W and 3 represent the height, width and number of channels of the input image respectively. The Canny edge detector is used to extract the Canny edge map of the LR image. And concatenate it with the LR image to get a 4-channel input image (2.2) A 3×3 convolutional layer is used to input the 4-channel image I in Extract shallow features from Where C represents the number of feature channels; (2.3) Through 6 residual groups, deep features are gradually extracted from the shallow features F0, where the output feature of the i-th residual group is recorded as (2.4) Perform channel concatenation on the output features of the six residual groups to obtain the concatenated features Then use 1×1 convolution to concatenate the features F concat The number of channels is reduced to C, and then the features are further refined through 3×3 convolution, and finally added pixel by pixel with the shallow feature F0 to obtain the final fusion feature (2.5) Construct an image reconstruction module from the input fusion feature F fused Reconstruct a four-channel super-resolution image: Use a 3×3 convolution layer to enhance the expressiveness of features, then use a sub-pixel convolution layer to upsample the features to the target size, and finally output a four-channel image through a 3×3 convolution layer Among them I out is a 3-channel RGB super-resolution image And the corresponding Canny edge map The spliced ​​image is , r represents the upsampling factor.

4. The method for realizing super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 3 is characterized in that: The step (3) specifically comprises the following steps: (3.1) Input features Perform layer normalization to stabilize the training process; (3.2) Construct a dense window attention branch and record the output of this branch as (3.3) Construct a deep convolution branch and use 3×3 depthwise separable convolution to interact between different windows. The output of this branch is recorded as (3.4) Construct the space-channel interaction module to DWA and Y DWC For bidirectional fusion and interaction, this module contains parallel channel attention branches and spatial attention branches; (3.5) Construct a fast Fourier transform-based feedforward network to adaptively enhance the learning ability of important frequency information by learning the quantization matrix in the frequency domain.

5. The method for super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 4 is characterized in that: The step (3.2) specifically comprises the following steps: (3.2.1) Window division: Sample tokens at adjacent positions of the input feature map and divide the feature map into windows of size S×S. The number of windows is (3.2.2) Attention calculation: Multi-head self-attention calculation is performed in each window. For the feature of the i-th window Flatten its dimensions, and we get Three linear mapping layers are then used to generate the query matrix key matrix and value matrix Then Q i , K i 、V i The three matrices are input into the multi-head self-attention mechanism MSA, as shown in the following formula: Where T represents the matrix transpose operation, Y i represents the attention result of the i-th window, and d represents the number of feature channels; (3.2.3) Window recovery: The attention calculation results of N windows are recombined according to the window division method to obtain the final attention output 6. The method for super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 4 is characterized in that: The step (3.4) specifically comprises the following steps: (3.4.1) Input to the spatial attention branch to generate a spatial attention map with pixel values ​​between 0 and 1 (3.4.2) Input to the channel attention branch to generate a channel attention map (3.4.3) Multiply the output of each branch by the input of the other branch: Y Channel With feature map Y DWA Perform element-wise multiplication to weight the channel attention to the feature map Y DWA ; Y Spatial With feature map Y DWC Perform element-wise multiplication to weight the spatial attention to the feature map Y DWC ; (3.4.4) Add the two weighted feature maps element by element to obtain the final output of the space-channel interaction module AND A =And Channel ⊙And DWA +Y Spatial ⊙And DWC 。 7. The method for super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 4, characterized in that: The step (3.5) specifically comprises the following steps: (3.5.1) Input features through 1×1 convolution Perform channel expansion; (3.5.2) Divide the feature map into windows with a window size of 8×8; (3.5.3) Perform a two-dimensional fast Fourier transform in each window to obtain the spectrum feature F f , as shown in the following formula: Among them, H and W represent the height and width of the window respectively; (3.5.4) Initialize a learnable frequency domain matrix W and the spectrum feature F f Perform element-by-element multiplication and update the parameters of W during network training to adaptively enhance important frequency domain information; (3.5.5) Perform a two-dimensional inverse fast Fourier transform in each window to transform the feature map from the frequency domain to the spatial domain; (3.5.6) Reassemble the windows in the same way as the windows are divided to restore the input features Size; (3.5.7) After 3×3 convolution, GELU activation function, 1×1 convolution, and then with the input features Add together to get the final output result 8. The method for realizing super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 4, characterized in that: The step (4) specifically comprises the following steps: (4.1) Input features Perform layer normalization to stabilize the training process; (4.2) Construct a sparse window attention branch, the output feature of which is recorded as (4.3) Construct a deep convolution branch and use 3×3 depthwise separable convolution to interact between different windows. The output feature of this branch is recorded as (4.4) Construct the space-channel interaction module to SWA and Y D ′ WC Carry out two-way integration and interaction; (4.5) Construct a fast Fourier transform-based feedforward network to adaptively enhance the learning ability of important frequency information by learning the quantization matrix in the frequency domain.

9. The method for realizing super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 8, characterized in that: The step (4.2) specifically comprises the following steps: (4.2.1) Window division: Sample tokens at intervals I on the input feature map and divide the feature map into windows of size The number of windows N = I 2 ; (4.2.2) Perform multi-head self-attention calculation in each window: for the feature of the i-th window Flatten its dimensions, and we get Three linear mapping layers are then used to generate the query matrix key matrix and value matrix Then Q i , K i 、V i The three matrices are input into the multi-head self-attention mechanism MSA, as shown in the following formula: Where T represents the matrix transpose operation, Y i represents the attention result of the i-th window, and d represents the number of feature channels; (4.2.3) Window recovery: The attention calculation results of N windows are recombined according to the window division method to obtain the final attention output 10. The method for super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 8, characterized in that: The step (4.4) specifically comprises the following steps: (4.4.1) Output features Input to the spatial attention branch to generate a spatial attention map with pixel values ​​between 0 and 1 (4.4.2) Output features Input to the channel attention branch to generate a channel attention map (4.4.3) Multiply the output of each branch by the input of the other branch: Y′ Channel With feature map Y SWA Perform element-wise multiplication to weight the channel attention to the feature map Y SWA ; Y′ Spacial With the feature map Y′ DWC Perform element-wise multiplication to weight the spatial attention to the feature map Y′ DWC ; (4.4.4) Add the two weighted feature maps element by element to get the final output of the space-channel interaction module AND' A =Y′ Channel ⊙And SWA +Y′ Spatial ⊙Y′ DWC 。 11. The method for realizing super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 8, characterized in that: The step (4.5) specifically comprises the following steps: (4.5.1) Input features through 1×1 convolution Perform channel expansion; (4.5.2) Divide the feature map into windows with a window size of 8×8; (4.5.3) Perform a two-dimensional fast Fourier transform in each window to obtain the spectrum feature F f , as shown in the following formula: Among them, H and W represent the height and width of the window respectively; (4.5.4) Initialize a learnable frequency domain matrix W and the spectrum feature F f Perform element-by-element multiplication and update the parameters of W during network training to adaptively enhance important frequency domain information; (4.5.5) Perform a two-dimensional inverse fast Fourier transform in each window to transform the feature map from the frequency domain to the spatial domain; (4.5.6) Reassemble the windows in the same way as the windows are divided to restore the input features Size; (4.5.7) After 3×3 convolution, GELU activation function, 1×1 convolution, and then with the input features Add together to get the final output result 12. The method for super-resolution reconstruction of real-world Chinese-English mixed text images based on dual window attention according to claim 8, characterized in that: The step (5) specifically comprises the following steps: (5.1) Calculate super-resolution SR image I SR and high-resolution HR image I SR The pixel loss between pixel : L pixel =||I HR -I SR ||1 (5.2) Calculate the edge perception loss L edge : Use the pre-trained VGG19 network to extract image features and edge features of HR images and SR images respectively, and then use the edge features to weight the image features to enhance the text area. edge The specific calculation formula is as follows: L edge =||F(I HR )·F(M HR )-F(I SR )·F(M SR )||1 Among them, F(·) is the pre-trained feature extractor, namely the VGG19 network, M HR , M SR Represents the Canny edge map corresponding to the HR image and the SR image; (5.3) The balance parameter λ is introduced to balance the weights of the two loss terms. The training loss calculation formula is as follows: L=L pixel +λ·L edge .

Images