Image super-resolution method based on convolutional neural network and Transform hybrid network structure

Abstract

The invention discloses an image super-resolution method based on a convolutional neural network and Transform hybrid network structure, and belongs to the field of computer vision. The method comprises the following steps: firstly, constructing an image super-resolution network of a convolutional neural network and Transform mixed structure, and realizing extraction of shallow layer features by using simple convolution; then, a plurality of residual groups are combined to realize extraction of depth features, each residual group is composed of a plurality of proposed double-path cooperation modules, and each module comprises a double-attention Transform module, a multi-scale convolution module and a selective fusion module; a double-attention Transform module is utilized to combine channel attention with space attention, so that the features can capture wider global information; the multi-scale convolution module is used for enhancing local representation of the features; the selection fusion module effectively retains a global contour and local details according to similarity self-adaptive fusion or interaction between global and local features; and finally, spatial feature conversion of the deep features is realized by using an up-sampling module, and a super-resolution result of the low-resolution image is obtained.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image super-resolution method based on a convolutional neural network and a Transformer hybrid network structure. Background Art

[0002] Image super-resolution is an important research topic in the field of computer vision and is a classic inverse problem in image reconstruction tasks. The main goal of image super-resolution is to restore the original high-resolution image from the degraded low-resolution image. Due to its wide application in computer vision fields such as medical imaging, video surveillance, and historical photo restoration, this technology has attracted more and more attention.

[0003] Most traditional image super-resolution technologies reconstruct high-resolution images by extracting low-level image features that have limited ability to express texture details such as image contours, which greatly limits the image reconstruction effect. In recent years, deep learning-based methods can extract more expressive image features from image data and adaptively learn the mapping relationship between low-resolution and high-resolution images. In particular, convolutional neural network-based and Transformer-based methods have been widely used in the field of image super-resolution, achieving better reconstruction effects. Related studies have shown that convolutional neural network models can obtain local features of images, and Transformer models can obtain global features of images. However, in the process of image super-resolution, there are still huge challenges to ensure that the global content of the image does not change and accurately restore the details of the image. Summary of the invention

[0004] In view of this, the purpose of this application is to provide an image super-resolution method based on a convolutional neural network and a Transformer hybrid network structure, which ensures that the local details and global contours of the image can be well restored while reconstructing a high-resolution image. The method is divided into three stages. The first stage is the shallow feature extraction stage, in which the image features are preliminarily extracted. The second stage is the deep feature extraction stage, in which the backbone network is used to achieve deep feature extraction. The third stage is the reconstruction stage, after which the image is reconstructed to obtain a high-resolution image after super-resolution.

[0005] The embodiment of the present application provides an image super-resolution method based on a convolutional neural network and a Transformer hybrid network structure, wherein the image super-resolution network mainly includes an initial convolution module, a backbone network, and an upsampling module. The backbone network is composed of a residual group, including a plurality of dual-path collaborative modules composed of a dual-attention Transformer module, a multi-scale convolution module, and a selection fusion module; the determined method includes:

[0006] Obtain the low-resolution image to be processed and perform preprocessing;

[0007] Build an image super-resolution network based on a hybrid structure of convolutional neural networks and Transformer;

[0008] Inputting the preprocessed low-resolution image into the initial convolution module for preliminary processing to obtain shallow features of the image;

[0009] Inputting the shallow features of the image into the backbone network to obtain the deep features of the image;

[0010] Inputting the deep features of the image into the upsampling module to obtain a reconstructed high-resolution image;

[0011] The image super-resolution network based on the convolutional neural network and the Transformer hybrid structure is optimized based on the target value to obtain a final image super-resolution network structure; wherein the target value includes: the original high-resolution image corresponding to the low-resolution image.

[0012] Furthermore, the initial convolution module includes only one basic convolution process; the step of inputting the pre-processed low-resolution image into the initial convolution module for preliminary processing includes:

[0013] The preprocessed low-resolution image is input into the initial convolution module to obtain four-dimensional batch image features, where the four dimensions are batch size, number of color channels, image height, and image width.

[0014] Inputting the shallow features of the image into the backbone network; the backbone network is mainly composed of a residual group; the residual group includes a plurality of dual-path collaborative modules; the dual-path collaborative modules include a dual-attention Transformer module, a multi-scale convolution module and a selection fusion module; inputting the shallow features of the image into the backbone network to obtain the image features of the last layer, and aggregating with the shallow features of the image to obtain the deep features of the image, including:

[0015] Inputting the shallow features of the image into the first residual group in the backbone network;

[0016] The shallow features of the image are input into the two-way collaborative module in the first residual group, and are aggregated with the shallow features of the image after being processed by multiple two-way collaborative modules to obtain intermediate features of the image; the intermediate features of the image are used as input features of the next residual group, and enhanced image features are obtained after being processed by the residual group; the enhanced image features continue to be processed as input features of the next residual group until all the residual groups are processed to obtain the image features of the last layer.

[0017] The dual-path collaborative module includes a dual-attention Transformer module, a multi-scale convolution module and a selective fusion module; the shallow features of the image are divided through channels to obtain divided image features; the divided image features are respectively input into two branches for processing; one of the two branches includes an initial convolution and a dual-attention Transformer module, and the other includes an initial convolution and a multi-scale convolution module to obtain global features and local features; the global features and local features are input into the selective fusion module to obtain image fusion features with global information and local information; the image fusion features are used as input features of the next dual-path collaborative module, and enhanced image fusion features are obtained after being processed by multiple dual-path collaborative modules, and are aggregated with the shallow features of the image to obtain the initial intermediate features of the image;

[0018] The divided image features are input into the branch including an initial convolution and a dual attention Transformer module; the divided image features are subjected to the initial convolution to obtain initial global features; the initial global features are input into the dual attention Transformer module; the dual attention Transformer module includes a first layer normalization unit and a window attention module, a second layer normalization unit and a feedforward network; the global features are obtained specifically through the following processing:

[0019] Inputting the initial global features into a normalization unit to obtain normalized image features;

[0020] The normalized image features are input into the window attention module and aggregated with the initial global features to obtain the first layer of image features; the window attention module comprises:

[0021] The normalized features are subjected to three convolutions and depth-separable convolutions to obtain query, key, and numerical vectors, and the matrix multiplication of the query and key vectors is calculated to obtain spatial attention, and the matrix multiplication with the numerical vector is calculated to obtain image features that focus on spatial characteristics;

[0022] The normalized features are passed through a channel attention module to obtain channel attention, including:

[0023] The normalized features are passed through a global pooling unit to obtain pooled features; the pooled features are input into a convolution unit, and then passed through a linear activation unit to obtain the channel attention.

[0024] The channel attention is multiplied by the image feature focusing on the spatial characteristics to obtain an intermediate global feature.

[0025] Further, the first layer image features are input into the normalization unit of the second layer to obtain normalized first layer image features;

[0026] Inputting the normalized first layer image features into the feedforward network includes:

[0027] The normalized first-layer image features are input into a fully connected layer, and the second-layer image features are obtained through linear units and another fully connected layer.

[0028] The second layer image features and the first layer image features are aggregated to obtain the global features.

[0029] Furthermore, the divided image features are input into the branch including an initial convolution and a multi-scale convolution module; the divided image features are subjected to the initial convolution to obtain initial local features; and the initial local features are input into the multi-scale convolution module, including:

[0030] The initial local features are subjected to three hole convolutions with different dilation rates respectively, and are aggregated with the initial local features, and then subjected to a convolution unit to obtain enhanced local features;

[0031] The enhanced local feature is aggregated with the initial local feature to obtain the local feature.

[0032] Furthermore, the global features and local features are input into the selection fusion module to obtain the image fusion features with global information and local information; the global features and local features are fused specifically in the following manner:

[0033] Calculating the cosine similarity between the global features and the local features, and obtaining a similarity weight matrix by activating the unit; multiplying the similarity weight matrix with the global features and the local features to extract global and local similar features and dissimilar features;

[0034] Aggregating the global and local similar features in a cascading manner to obtain similar features;

[0035] The global and local dissimilar features are transformed into interactive global and local features through a spatial feature transformation module, including:

[0036] The global features and local features are both modulated by learning modulation parameters through convolution units and activation units, and modulation is achieved by performing dot product and summation with corresponding features, and finally obtaining the interactive global and local features;

[0037] The global and local features of the interaction are aggregated through cascade to obtain dissimilar features;

[0038] The similar features and dissimilar features are aggregated, and the image fusion features with global information and local information are obtained through a convolution unit.

[0039] Furthermore, the deep features of the image are input into the upsampling module; the module includes two convolution units and an upsampling unit; the upsampling module obtains the reconstructed high-resolution image in the following manner:

[0040] Inputting the deep features of the image into the convolution unit to obtain the preliminarily processed deep features of the image;

[0041] The deep features of the preliminarily processed image are subjected to an upsampling unit to obtain high-resolution image features;

[0042] The high-resolution image features are passed through a convolution unit to obtain the reconstructed high-resolution image.

[0043] Furthermore, the image super-resolution network based on the hybrid structure of convolutional neural network and Transformer is optimized based on the target value to obtain the final image super-resolution network, including:

[0044] Based on the original high-resolution image corresponding to the low-resolution image, an l1 loss function of the image super-resolution network is calculated.

[0045] Furthermore, the L1 loss function calculation formula is as follows:

[0046]

[0047] Among them, f(x) represents the predicted value, y represents the true value, and ‖·‖1 represents the norm.

[0048] The present invention provides a computer system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the image super-resolution method based on a hybrid network structure of a convolutional neural network and a Transformer is implemented.

[0049] Beneficial effects: Compared with existing image super-resolution methods, this method has the following advantages:

[0050] 1. The present invention hybridizes convolutional neural networks and Transformer structures, takes advantage of the convolutional neural networks to reconstruct local details and the Transformer structures to restore global contours, and proposes a dual-path collaborative module that integrates a dual-attention Transformer module, a multi-scale convolution module, and a selective fusion module, thereby better improving the quality of the reconstructed image.

[0051] 2. The present invention uses a dual-attention Transformer module to strengthen the comprehensive attention to image features from two dimensions: space and channel. It pays more attention to the correlation between pixels in space and the importance of channels in channels, thereby comprehensively acquiring the global information of the image.

[0052] 3. The present invention obtains local features at different scales through a multi-scale convolution module, retains the diversity of local features, and thus enhances the local content of the image.

[0053] 4. The present invention realizes the process of adaptively fusion features by selecting a fusion module according to the similarity between global features and local features, thereby ensuring that the complete image contour and fine texture details can be retained during the fusion process.

[0054] 5. The experimental results in the specific embodiments below confirm the effectiveness and superiority of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a structural schematic diagram of an image super-resolution method based on a convolutional neural network and a Transformer hybrid network structure of the present invention.

[0056] Figure 2 It is a schematic diagram of the dual-path collaborative module of the present invention.

[0057] Figure 3 Schematic diagram of the dual-attention Transformer module of the present invention.

[0058] Figure 4 Schematic diagram of the multi-scale convolution module of the present invention.

[0059] Figure 5 Schematic diagram of selecting fusion module for the present invention.

[0060] Figure 6 Schematic diagram of the upsampling module of the present invention.

[0061] Figure 7 This is a super-resolution image effect diagram of the present invention. DETAILED DESCRIPTION

[0062] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] An image super-resolution method based on a hybrid network structure of a convolutional neural network and a Transformer is provided in an embodiment of the present invention. The existing network architecture based on a convolutional neural network or a Transformer is improved, and a Transformer branch is added to the structure based on the convolutional neural network. At the same time, the features of local and global information are extracted respectively according to the convolutional neural network and the Transformer, and the interaction and fusion of the two are realized through adaptive fusion. In addition, a multi-scale process is added to the convolutional neural network to extract local features with multiple contents. The Transformer structure pays attention to both spatial and channel aspects at the same time to obtain more comprehensive global features. The structural flow diagram of the network based on which the embodiment of the present invention is based is shown in the figure. Figure 1 shown.

[0064] Specifically, an image super-resolution method based on a hybrid network structure of a convolutional neural network and a Transformer in an embodiment of the present invention first extracts features from an input image to obtain shallow features of the image, that is, the dimensions are B×C×H×W, where B is the batch size of the image, which is set to 8, C is the number of channels of the image features, which is set to 64, and H and W are the height and width of the image, respectively; then the shallow image of the image is input into a deep network, that is, a network composed of multiple residual groups; wherein each residual group includes multiple dual-path collaborative modules, each module is processed by dual attention Transformer. The sformer module, multi-scale convolution module and selective fusion module realize the feature enhancement process to obtain deep features, that is, the dimension is B×C×H×W, B is 8, C is 64, H and W are the height and width of the image features, respectively, which are consistent with the shallow features; finally, the deep features are restored to the image space through the upsampling module, that is, the dimension is B×C×H'×W', at this time B is 8, C is 3, H' and W' correspond to the image height and width obtained by different super-resolution multiples, for example, the result of 2x super-resolution should be expressed as H'=2×H, so as to realize the reconstruction process from low-resolution to high-resolution images.

[0065] Among them, the dual-attention Transformer module first obtains the query, key, and numerical vectors related to the features through linear transformation, and at the same time obtains the weight matrix of the importance of different channels of the input features through the channel attention unit, and then calculates the spatial features of the numerical vector with the query and key vectors to obtain the global features that pay attention to both space and channels.

[0066] The multi-scale convolution module acquires local information of different scales through dilated convolutions with different expansion rates, and aggregates them with the original input features to obtain various local features, thereby enhancing the focus on local information and obtaining more detailed local features.

[0067] The selective fusion module calculates the implicit similarities and differences between global features and local features, converts them into corresponding weight matrices, adaptively realizes the fusion of similar features and the interaction of dissimilar features, and enhances the overall characteristics of the image.

[0068] In the upsampling module, the image is converted from the feature space to the original space by linear transformation and pixel rearrangement to obtain a reconstructed high-resolution image.

[0069] Combine the following Figure 1 Detailed steps to explain the above method in detail:

[0070] Step 1: Obtain a low-resolution image to be reconstructed, with an image resolution of 64×64;

[0071] Step 2: Extract shallow features of the input low-resolution image through the initial convolution module, that is, B×C×H×W, where B represents the batch size of the input image, C represents the number of channels of the image, H represents the height of the image, and W represents the width of the image; in this example, the batch size is 8, the number of channels is 64, and the height and width of the image are 64 and 64 respectively;

[0072] Step 3: Input the extracted shallow features (dimension is 8×64×64×64) into the backbone network to obtain the deep features of the image, and keep the feature dimension of the image unchanged, where the dimension of the deep features is still 8×64×64×64);

[0073] Step 4: Input the deep features of the image (i.e., spatial features with a dimension of 8×64×64×64) into the upsampling module, and finally obtain the reconstructed image, with the dimension converted from the original feature space to the image space. If the image is super-resolved by 2 times, the dimension is 8×3×128×128; if the image is super-resolved by 3 times, the dimension is 8×3×192×192; if the image is super-resolved by 4 times, the dimension is 8×3×256×256.

[0074] The process of obtaining the low-resolution image in step 1 in this example includes: downloading the data set from the open source website, constructing 64×64 low-resolution images and 128×128, 192×192, and 256×256 high-resolution image data pairs, corresponding to 2x, 3x, and 4x super-resolution multiples, respectively.

[0075] In this example, step 2 uses basic convolutional units to extract shallow features, and its expression is as follows:

[0076] F shallow =Conv 3×3 (I LR );

[0077] Among them, F shallow Represents shallow features, whose dimensions are B×C×H×W, Conv 3×3 (·) represents the basic convolution unit, the convolution kernel size is 3×3, the stride is 1, and I LR Represents a low-resolution image with a resolution of 64×64.

[0078] In this example, step 3 extracts the shallow features of the image F shallow Input into the backbone network, where the backbone network is mainly composed of multiple residual groups. Processing to obtain different features F n-1 As the input of the next residual group, the expression for obtaining the deep features is as follows:

[0079]

[0080] Among them, F deep Represents the obtained image depth features, with dimensions of B×C×H×W, represents the processing process of the nth residual group, F n-1 It represents the image features after processing the n-1th residual group, that is, the intermediate features of the image, with a dimension of B×C×H×W.

[0081] In the above backbone network, the input shallow features are input into the residual group, processed by multiple two-way collaborative modules, and aggregated with the initial input features to obtain the output features of the residual group, which are expressed as follows:

[0082]

[0083] Among them, F n represents the features after processing of the nth residual group, represents m groups of dual-path cooperative modules, F n-1 Represents the features of the n-1th residual group after processing.

[0084] For each two-way collaborative module processing feature, the module structure is as follows Figure 2 The specific steps are described as follows:

[0085] 3-1. The input image features are processed by channel division and evenly divided into two branches, that is, the image feature dimension obtained by each branch is Among them, one branch is mainly based on the dual attention Transformer module to extract the global features of the image, and the other branch is mainly based on the multi-scale convolution module to extract the local features of the image. The expressions for obtaining the two-way features are as follows:

[0086] F global ,F local =Split(F n-1 );

[0087] Among them, F n-1 represents the input image features, the dimension is B×C×H×W, Split(·) represents the channel division process, F global Represents the global features of the image, F local Represents the local features of the image, and the dimensions of both are

[0088] 3-2. Input the features obtained in step 3-1 into the branch mainly composed of the dual attention Transformer module. First, pass it through a basic convolution unit to obtain the initial global features, which are expressed as follows:

[0089] F′ global =Conv 3×3 (F global );

[0090] Among them, F′ global Indicates that after the basic convolution unit Conv 3×3 (·), the convolution kernel size is 3×3, the step size is 1, and the initial global feature obtained after processing, F global Represents the global features of the input image.

[0091] 3-3. The initial global features obtained in step 3-2 are passed through the dual attention Transformer module. The module structure is as follows Figure 3 As shown in Figure 1. First, the initial global features of the input are normalized and divided into two parts according to the number of heads M and the number of channels C to calculate the attention in parallel. On the one hand, we take the feature F′ global Divide it into non-overlapping H×sw axial windows and denote the feature of the i-th window as X i ,in Indicates the number of vertical windows, X i represents the image features inside the i-th vertical window, and this method is called vertical window attention, denoted as Av-WA. On the other hand, F′ global Divide into non-overlapping sh×W axial windows and denote the feature of the jth window as X j ,in Indicates the number of horizontal windows, X jrepresents the image features inside the jth horizontal window, and this method is called horizontal window attention, denoted as Ah-WA. The sw and sh mentioned here represent the width of the divided vertical window and the height of the horizontal window respectively, and X i and X j All are input features F′ global For feature X in the vertical window i and the feature X in the horizontal window j Through the corresponding projection matrix W i Q , W i K , W i V and Perform linear changes to obtain the query, key, and numerical vectors required to calculate spatial attention, that is, to calculate the vertical window feature X i The attention (Q i ,K i ,V i ) and horizontal window feature X j The attention (Q j ,K j ,V j ), which is expressed as follows:

[0092] (Q i ,K i ,V i )=(X i W i Q ,X i W i K ,X i W i V );

[0093]

[0094] in, represents the projection matrix, Represents the projection dimension.

[0095] 3-4. The vertical window features X obtained in step 3-3 are i and horizontal window feature X j The query, key, and numerical vector (Q i ,K i ,V i ) and (Q j ,K j ,V j ) to calculate the spatial attention Y i and Yj , which is expressed as follows:

[0096]

[0097]

[0098] Among them, Y i and Y j Represents the vertical window feature X i and horizontal window feature X j Attention features, B represents dynamic position encoding.

[0099] 3-5. Reshape and fuse the attention features of each part obtained in step 3-4 in order, and then combine them in a cascade manner to form the final spatial attention F A-WA , which is expressed as follows:

[0100] F A-WA =Axial-WA(F′ global )=Concat(Y v ,Y h );

[0101] Among them, Y v Represents the attention features of all vertical windows Y i The collection of Y h Represents all horizontal window attention features Y j A collection of .

[0102] 3-6. After obtaining the spatial attention in step 3-5, the average pooling, ReLU activation function, convolution The channel attention module composed of the Sigmoid activation function is used to input the initial global feature F′ global Through the channel attention module processing, the channel attention matrix F that focuses on the importance of the channel is obtained SGCA In this process, the expression is as follows:

[0103] F SGCA =SGCA(F′ global )=Sigmoid(ConvLayer(Avg(F′ global )))⊙V;

[0104] Among them, Avg(·) represents the average pooling process, ConvLayer(·) represents the structure composed of ReLU activation function and convolution unit, Sigmoid(·) represents the activation function process, ⊙ represents the dot product, and V represents the numerical vector consistent with the calculation window attention process, that is, V i and V jThey respectively represent the i-th vertical window space feature numerical vector and the j-th horizontal window space feature numerical vector of feature V.

[0105] The above steps 3-3 to 3-6 can be summarized as the following expression:

[0106] F DATM =DATM(LN(F′ global ))+F′ global ;

[0107] Among them, DATM(·) represents the dual attention Transformer module, LN(·) represents the normalization unit, and F DATM Represents the features obtained after processing by the two-way collaborative module.

[0108] 3-7. The feature F obtained in step 3-6 DATM Input into the normalization unit of the second layer, and enhance the global features through the feedforward network to obtain the overall global feature X GF The feedforward network consists of two fully connected layers and linear units. The expression of this process is as follows:

[0109] X GF =FFN(LN(F DATM ))+F DATM ;

[0110] Among them, FFN(·) represents the feed-forward network layer, X GF Represents the final global feature.

[0111] 3-8. Input the features obtained in step 3-1 into the branch mainly composed of multi-scale convolution modules. The module structure is as follows: Figure 4 As shown. First, it is passed through a basic convolution unit to obtain the initial local features, which are expressed as follows:

[0112] F′ local =Conv 3×3 (F local );

[0113] Among them, F′ local Indicates that after the basic convolution unit Conv 3×3 (·) The initial local features obtained after processing, the convolution kernel size is 3×3, the step size is 1, and F local Represents the input image features.

[0114] 3-9. Input the initial local features obtained in step 3-8 into different receptive field dilated convolutions to obtain three features focusing on different scales of content, and connect them with the original input features and aggregate them through a basic convolution unit. The expression of the whole process is as follows:

[0115] X LF =Conv 1×1 (Concat(F′ local ,Conv d＝k (F′ local )))+F′ local ;

[0116] Among them, Conv 1×1 (·) represents the basic convolution unit, the convolution kernel size is 1×1, the step size is 1, and Concat(·) represents the connection process. d＝k (·) represents the dilated convolution operation with a dilation rate of k (k = 1, 3, 5), X LF Represents the enhanced local features obtained.

[0117] 3-10. Input the global and local features obtained in steps 3-7 and 3-9 into the selection fusion module. The module structure is as follows: Figure 5 First, the cosine similarity is used to calculate the similarity matrix of the global and local features of the input from the channel and space aspects respectively. and Then, the eigenvalues ​​of the values ​​in the matrix between [0,1] and [-1,0) are used as the feature criteria for distinguishing similarities and dissimilarities, and the selection matrix is ​​calculated using the Sigmoid activation function. Its expression is as follows:

[0118]

[0119]

[0120] M = Reshape(Sigmoid(M c ) T *Sigmoid(M s ));

[0121] in, Represents the calculation process of cosine similarity. Each value of M is between -1 and 1. The closer it is to 1, the higher the similarity.

[0122] 3-11. The selection matrix obtained in step 3-10 is used to screen the features generated by the two branches to distinguish similar and dissimilar content. For features with high similarity, cascade processing is directly used to retain the common information in these features. The expression is as follows:

[0123]

[0124]

[0125]

[0126] in, t=trans or t=cnn indicates similar features, F sim Indicates similar features between global features and local features, that is, the data of its eigenvalues ​​are between [0,1], and the dimension is

[0127] 3-12. For dissimilar features, a spatial feature transformation module is used to promote interaction, in the hope that the global and local features obtained from steps 3-7 and 3-9 can complement each other while retaining their respective characteristics. The spatial transformation module needs to be learned based on different characteristics so that they can focus on the unique features of the two branches at the same time. The expression of this process is as follows:

[0128]

[0129]

[0130]

[0131] F dis =Concat(X' u );

[0132] in, and X' u , u=trans or u=cnn represent different features and modified features respectively, and SFT(·) represents the function used to obtain the modulation coefficient γ v and β v (v=trans or v=cnn) spatial feature transformation module, where γ v and β v is a learnable parameter obtained by processing 3×3 and 1×1 convolution kernels, with a step size of 1. dis Indicates the dissimilar features between global features and local features, that is, the data of its eigenvalues ​​are between [-1,0), and the dimension is

[0133] 3-13. Integrate the similar and dissimilar contents obtained in steps 3-11 and 3-12 into the same feature space, while keeping the same dimension as the initial feature space. The expression is as follows:

[0134]

[0135] Among them, F SFM Indicates the output feature of the selected fusion module, that is, the output feature obtained by the last module of the n-1th residual group, that is, the output feature F of the n-1th residual groupn-1 , represents the aggregation process, Conv(·) represents the basic convolution unit for reducing the feature dimension, the convolution kernel size is 3×3, the step size is 1, and the number of channels is set to

[0136] In this example, step 4 converts the deep features F deep Input to the upsampling module, which is mainly composed of basic convolution units and pixel rearrangement. The module structure is as follows Figure 6 As shown. After the module processes the features, the image is restored from the feature space to the original image space to obtain the final reconstructed image. The specific steps are as follows:

[0137] 4-1. Input the deep features obtained in step 3 into the basic convolution unit to obtain the initial reconstruction features, which are expressed as follows:

[0138] F mid =Conv 3×3 (F deep );

[0139] Among them, F deep represents the deep features obtained in step 3, F mid represents the initial reconstruction features, Conv 3×3 (·) represents the basic convolution unit, the convolution kernel size is 3×3, and the stride is 1.

[0140] 4-2. Input the initial reconstruction feature obtained in step 4-1 into the pixel rearrangement unit to obtain the reconstruction feature, which is expressed as follows:

[0141] F out =PixelShuffle(F mid );

[0142] Among them, F out represents the reconstruction feature, and PixelShuffle(·) represents the pixel rearrangement unit.

[0143] 4-3. The reconstructed features obtained in step 4-2 are restored to the original image space through the basic convolution unit to obtain the final reconstructed image, which is expressed as follows:

[0144] SR=Conv 3×3 (F out );

[0145] Among them, Conv 3×3 (·) represents the basic convolution unit, the convolution kernel size is 3×3, the stride is 1, and SR represents the reconstructed high-resolution image.

[0146] The following examples illustrate the implementation effect of the present invention in the actual application process:

[0147] In an experiment, the L1 loss function is used to optimize the image super-resolution network. The image super-resolution network is trained for a total of 500K iterations, and the learning rate is set to 2×10 -4 , and halved at [250K, 400K, 450K, 475K]; the input image is cropped to a size of 64×64; the batch size is set to 8; here, this application uses PyTorch as the code implementation, and uses the Adam optimizer to optimize the model.

[0148] This application is trained on the classic DF2K dataset, which includes two datasets: DIV2K and Flickr2K. In addition, the model is evaluated on five standard datasets, including Set5, Set14, B100, Urban100 and Manga109. In order to evaluate the reconstruction ability of the model, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to calculate the reconstruction results on the Y channel of the YCbCr color space.

[0149] The test results on five standard datasets are shown in Table 1. The smaller the indicators are, the better the reconstruction effect is and the higher the image quality is.

[0150]

[0151] Table 1. Experimental data of image super-resolution on various standard datasets

[0152] It can be seen that the image super-resolution network in the present invention has achieved excellent image super-resolution effect. In addition to ensuring the preservation of the overall content of the image during the reconstruction process, it also restores the characteristics of image texture details and other aspects. The specific subjective effect of super-resolution can be seen Figure 7 .

[0153] Based on the same inventive concept, an embodiment of the present invention provides an image super-resolution system based on a convolutional neural network and a Transformer hybrid network structure, comprising an input module, an image super-resolution network model and an output module; the input module is used to obtain a low-resolution image and input it into the image super-resolution network model; the image super-resolution network model is used to perform a super-resolution process on the input low-resolution image to obtain an image that meets the super-resolution multiple requirements; the output module is used to reconstruct a high-resolution image based on the image features output by the super-resolution network model; the image super-resolution network model is composed of a series of residual groups, It includes a two-way collaborative module; and the two-way collaborative module is composed of a dual-attention Transformer module, a multi-scale convolution module and a selective fusion module; the initial feature extraction is performed on the input low-resolution image, input into the residual group, and the image depth features are obtained after processing by multiple residual groups, wherein each residual group contains multiple two-way collaborative modules, which divide the input features into two parts, one part is passed through the dual-attention Transformer module to obtain the global features of the image, and the other part is passed through the multi-scale convolution module to obtain the local features of the image, and then the global features and local features are adaptively fused and interacted through the selective fusion module. The specific implementation of each module of the network model refers to the above-mentioned method embodiment and will not be repeated here. Those not described in detail in the present invention are all prior art.

[0154] Based on the same inventive concept, an embodiment of the present invention provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the above-mentioned image super-resolution method based on a hybrid network structure of a convolutional neural network and a Transformer are implemented.

Claims

1. An image super-resolution method based on a hybrid network structure of a convolutional neural network and a Transformer, characterized in that: First, feature extraction is performed on the input image to obtain the shallow features of the image, that is, the dimension is B×C×H×W, where B is the batch size of the image, set to 8, C is the number of channels of the image feature, set to 64, and H and W are the height and width of the image respectively; then the shallow image of the image is input into the deep network, that is, a network composed of multiple residual groups; wherein each residual group contains multiple two-way collaborative modules, each module realizes the feature enhancement process through a dual attention Transformer module, a multi-scale convolution module and a selective fusion module to obtain the deep features, that is, the dimension is B×C×H×W, B is 8, C is 64, H and W are the height and width of the image features respectively, which is consistent with the shallow features; finally, the deep features are restored to the image space through the upsampling module, that is, the dimension is B×C×H'×W', at this time B is 8, C is 3, H' and W' correspond to the image height and width obtained at different super-resolution multiples.

2. The method according to claim 1, characterized in that include: The dual-attention Transformer module first obtains the query, key, and numerical vectors related to the features through linear transformation, and at the same time obtains the weight matrix of the importance of different channels of the input features through the channel attention unit, and then calculates the spatial features of the query and key vectors acting on the numerical vector to obtain the global features that pay attention to both space and channels.

3. The method according to claim 1, characterized in that include: The multi-scale convolution module acquires local information of different scales through dilated convolutions with different expansion rates, and aggregates them with the original input features to obtain various local features. It expands the receptive field by changing the convolution form without changing the original feature size, thereby enhancing the focus on local information.

4. The method according to claim 1, characterized in that: include: The selective fusion module calculates the implicit similarities and differences between global features and local features, converts them into corresponding weight matrices, adaptively realizes the fusion of similar features and the interaction of dissimilar features, and enhances the overall characteristics of the image.

5. The method according to claim 1, characterized in that The following steps are involved: Step 1: Obtain a low-resolution image to be reconstructed, with an image resolution of 64×64; Step 2: Extract shallow features of the input low-resolution image through the initial convolution module, that is, B×C×H×W, where B represents the batch size of the input image, C represents the number of channels of the image, H represents the height of the image, and W represents the width of the image; in this example, the batch size is 8, the number of channels is 64, and the height and width of the image are 64 and 64 respectively; Step 3: Input the extracted shallow features (dimension is 8×64×64×64) into the backbone network to obtain the deep features of the image, and keep the feature dimension of the image unchanged, where the dimension of the deep features is still 8×64×64×64); Step 4: Input the deep features of the image (i.e., spatial features with a dimension of 8×64×64×64) into the upsampling module, and finally obtain the reconstructed image, with the dimension converted from the original feature space to the image space; when performing 2x image super-resolution, the dimension is 8×3×128×128; when performing 3x image super-resolution, the dimension is 8×3×192×192; when performing 4x super-resolution, the dimension is 8×3×256×256.

6. The method according to claim 1, characterized in that The following steps are involved: The process of obtaining low-resolution images in step 1 includes: downloading the dataset from an open source website, constructing 64×64 low-resolution images and 128×128, 192×192, and 256×256 high-resolution image data pairs, corresponding to 2x, 3x, and 4x super-resolution multiples, respectively; Step 2 extracts shallow features using basic convolutional units, which are expressed as follows: F shallow =Conv 3×3 (I LR ); Among them, F shallow Represents shallow features, whose dimensions are B×C×H×W, Conv 3×3 (·) represents the basic convolution unit, the convolution kernel size is 3×3, the stride is 1, and I LR Indicates that the input resolution is a low-resolution image of 64×64; Step 3: Extract the shallow features F of the image shallow Input into the backbone network, where the backbone network is mainly composed of multiple residual groups. Processing to obtain different features F n-1 As the input of the next residual group, the expression for obtaining the deep features is as follows: Among them, F deep Represents the obtained image depth features, with dimensions of B×C×H×W, represents the processing process of the nth residual group, F n-1 Represents the image features after processing the n-1th residual group, that is, the intermediate features of the image, with dimensions of B×C×H×W; In the above backbone network, the intermediate feature F n-1 The input is input into the residual group, processed by multiple two-way collaborative modules, and aggregated with the initial input features to obtain the output features of the residual group, which are expressed as follows: Among them, F n represents the features after processing of the nth residual group, represents m groups of dual-path cooperative modules, F n-1 Represents the features after processing of the n-1th residual group; For the process of processing features by each two-way collaborative module, the module structure is shown in Figure 2, and the specific steps are described as follows: 3-1. The input image features are processed by channel division and evenly divided into two branches, that is, the image feature dimension obtained by each branch is Among them, one branch is mainly based on the dual attention Transformer module to extract the global features of the image, and the other branch is mainly based on the multi-scale convolution module to extract the local features of the image. The expressions for obtaining the two-way features are as follows: F global ,F local =Split(F n-1 ); Among them, F n-1 represents the input image features, the dimension is B×C×H×W, Split(·) represents the channel division process, F global Represents the global features of the image, F local Represents the local features of the image, and the dimensions of both are 3-2. The global feature F obtained in step 3-1 global Input into the branch mainly composed of the dual attention Transformer module in claim 2, first pass it through a basic convolution unit to obtain the initial global feature, which is expressed as follows: F’ global =Conv 3×3 (F global ); Among them, F' global Indicates that after the basic convolution unit Conv 3×3 (·), the convolution kernel size is 3×3, the step size is 1, and the initial global feature obtained after processing, F global Represents the global features of the input image; 3-3. The initial global features obtained in step 3-2 are passed through the dual attention Transformer module. The module structure is shown in Figure 3. First, the initial global features of the input are normalized and divided into two parts according to the number of heads M and the number of channels C to calculate the attention in parallel. On the one hand, we take the feature F' global Divide it into non-overlapping H×sw axial windows and denote the feature of the i-th window as X i , where i = 1, 2, ..., Indicates the number of vertical windows, X i represents the image features inside the i-th vertical window, and this method is called vertical window attention, denoted as Av-WA; on the other hand, we global Divide into non-overlapping sh×W axial windows and denote the feature of the jth window as X j , where j = 1, 2, ..., Indicates the number of horizontal windows, X j represents the image features inside the jth horizontal window, and this method is called horizontal window attention, denoted as Ah-WA; the sw and sh mentioned here represent the width of the divided vertical window and the height of the horizontal window respectively, and X i and X j All are input features F' global Sub-part of; for feature X in the vertical window i and the feature X in the horizontal window j Through the corresponding projection matrix W i Q , W i K , W i V and Perform linear changes to obtain the query, key, and numerical vectors required to calculate spatial attention, that is, to calculate the vertical window feature X i The attention (Q i ,K i ,V i ) and horizontal window feature X j The attention (Q j ,K j ,V j ), which is expressed as follows: (Q i ,K i ,V i )=(X i W i Q ,X i W i K ,X i W i V ); in, represents the projection matrix, represents the projection dimension; 3-4. The vertical window features X obtained in step 3-3 are i and horizontal window feature X j The query, key, and numerical vector (Q i ,K i ,V i ) and (Q j ,K j ,V j ) to calculate the spatial attention Y i and Y j , which is expressed as follows: Among them, Y i and Y j Represents the vertical window feature X i and horizontal window feature X j Attention features, B represents dynamic position encoding; 3-5. Reshape and fuse the attention features of each part obtained in step 3-4 in order, and then combine them in a cascade manner to form the final spatial attention F A-WA , which is expressed as follows: F A-WA =Axial-WA(F’ global )=Concat(Y v ,Y h ); Among them, Y v Represents the attention features of all vertical windows Y i The collection of Y h Represents all horizontal window attention features Y j A collection of; 3-6. After obtaining the spatial attention in step 3-5, the initial global feature F of the input is obtained through a channel attention module consisting of average pooling, ReLU activation function, convolution and Sigmoid activation function. g ' lobal Through the channel attention module processing, the channel attention matrix F that focuses on the importance of the channel is obtained SGCA ; In this process, its expression is as follows: F SGCA =SGCA(F’ global )=Sigmoid(ConvLayer(Avg(F’ global )))⊙V; Among them, Avg(·) represents the average pooling process, ConvLayer(·) represents the structure composed of ReLU activation function and convolution unit, Sigmoid(·) represents the activation function process, ⊙ represents the dot product, and V represents the numerical vector consistent with the calculation window attention process, that is, Vi and V j Respectively represent the i-th vertical window space feature value vector and the j-th horizontal window space feature value vector of feature V; The above steps 3-3 to 3-6 can be summarized as the following expression: F DATM =DATM(LN(F g ' lobal ))+F g ' lobal ; Among them, DATM(·) represents the dual attention Transformer module, LN(·) represents the normalization unit, and F DATM It represents the features obtained after being processed by the two-way collaborative module; 3-7. The feature F obtained in step 3-6 DATM Input into the normalization unit of the second layer, and enhance the global features through the feedforward network to obtain the overall global feature X GF ; The feedforward network consists of two fully connected layers and linear units. The expression of this process is as follows: X GF =FFN(LN(F DATM ))+F DATM ; Among them, FFN(·) represents the feed-forward network layer, X GF Represents the final global feature; 3-8. The feature obtained in step 3-1 is input into the branch mainly composed of the multi-scale convolution module described in claim 3, and the module structure is shown in FIG4 ; first, the initial local feature is obtained by passing it through a basic convolution unit, and its expression is as follows: F’ local =Conv 3×3 (F local ); Among them, F' local Indicates that after the basic convolution unit Conv 3×3 (·) The initial local features obtained after processing, the convolution kernel size is 3×3, the step size is 1, F local Represents the local features of the input; 3-9. Input the initial local features obtained in step 3-8 into different receptive field dilated convolutions to obtain three features focusing on different scales of content, and connect them with the original input features and aggregate them through a basic convolution unit. The expression of the whole process is as follows: X LF =Conv 1×1 (Concat(F’ local ,Conv d＝k (F’ local )))+F’ local ; Among them, Conv 1×1 (·) represents the basic convolution unit, the convolution kernel size is 1×1, the step size is 1, and Concat(·) represents the connection process. d＝k (·) represents the dilated convolution operation with a dilation rate of k (k = 1, 3, 5), X LF Represents the enhanced local features obtained; 3-10. The global and local features obtained in step 3-7 and step 3-9 are input into the selective fusion module described in claim 4, and the module structure is shown in FIG5 ; first, the similarity matrix of the global and local features of the input is calculated from the channel and space aspects respectively using cosine similarity and Then, the eigenvalues ​​of the values ​​in the matrix between [0,1] and [-1,0) are used as the feature criteria for distinguishing similarities and dissimilarities, and the selection matrix is ​​calculated using the Sigmoid activation function. Its expression is as follows: M=Reshape(Sigmoid(M c ) T *Sigmoid(M s )); in, Represents the calculation process of cosine similarity. Each value of M is between -1 and 1. The closer it is to 1, the higher the similarity. 3-11. The selection matrix obtained in step 3-10 is used to screen the features generated by the two branches to distinguish similar and dissimilar contents; for features with high similarity, cascade processing is directly used to retain the common information in these features, and its expression is as follows: in, t=trans or t=cnn indicates similar features, F sim Indicates similar features between global features and local features, that is, the data of its eigenvalues ​​are between [0,1], and the dimension is 3-12. For features that are not similar in global features and local features, a spatial feature transformation module is used to promote interaction, in the expectation that the global and local features obtained from steps 3-7 and 3-9 can complement each other while retaining their respective features; the spatial transformation module needs to be learned based on different characteristics so that they can focus on the unique features of the two branches at the same time. The expression of this process is as follows: F dis =Concat(X u '); in, and X' u , u=trans or u=cnn represent different features and modified features respectively, and SFT(·) represents the function used to obtain the modulation coefficient γ v and β v (v=trans or v=cnn) spatial feature transformation module, where γ v and β v is a learnable parameter obtained by processing 3×3 and 1×1 convolution kernels, with a step size of 1. dis Indicates the dissimilar features between global features and local features, that is, the data of its eigenvalues ​​are between [-1,0), and the dimension is 3-13. Integrate the similar and dissimilar contents obtained in steps 3-11 and 3-12 into the same feature space, while keeping the same dimension as the initial feature space. The expression is as follows: Among them, F SFM Indicates the output feature of the selected fusion module, that is, the output feature obtained by the last module of the n-1th residual group, that is, the output feature F of the n-1th residual group n-1 , represents the aggregation process, Conv(·) represents the basic convolution unit for reducing the feature dimension, the convolution kernel size is 3×3, the step size is 1, and the number of channels is set to In this example, step 4 converts the deep features F deep The input is sent to the upsampling module, which is mainly composed of basic convolution units and pixel rearrangement. The module structure is shown in Figure 6. After the module processes the features, the image is restored from the feature space to the original image space to obtain the final reconstructed image. The specific steps are as follows: 4-1. Input the deep features obtained in step 3 into the basic convolution unit to obtain the initial reconstruction features, which are expressed as follows: F mid =Conv 3×3 (F deep ); Among them, F deep represents the deep features obtained in step 3, F mid represents the initial reconstruction features, Conv 3×3 (·) represents the basic convolution unit, the convolution kernel size is 3×3, and the stride is 1; 4-2. Input the initial reconstruction feature obtained in step 4-1 into the pixel rearrangement unit to obtain the reconstruction feature, which is expressed as follows: F out =PixelShuffle(F mid ); Among them, F out represents the reconstruction feature, PixelShuffle(·) represents the pixel rearrangement unit; 4-3. The reconstructed features obtained in step 4-2 are restored to the original image space through the basic convolution unit to obtain the final reconstructed image, which is expressed as follows: SR=Conv 3×3 (F out ); Among them, Conv 3×3 (·) represents the basic convolution unit, the convolution kernel size is 3×3, the stride is 1, and SR represents the reconstructed high-resolution image.

Images