An image super-resolution method based on a convolutional neural network and a hybrid network structure of a Transformer

Abstract

The image super-resolution method based on convolutional neural network and transformer hybrid network structure belongs to the field of computer vision. First, the image super-resolution network with convolutional neural network and transformer hybrid structure is built, and simple convolution is used to extract shallow features. Then, multiple residual groups are combined to extract deep features, where the residual group is composed of multiple proposed double-path collaborative modules, including a double-attention transformer module, a multi-scale convolution module and a selective fusion module. The double-attention transformer module combines channel attention and spatial attention to enable the feature to capture more extensive global information. The multi-scale convolution module is used to enhance the local representation of the feature. The selective fusion module adaptively fuses or interacts according to the similarity between global and local features, effectively preserving global contours and local details. Finally, the spatial feature conversion of the deep feature is realized by using the up-sampling module, and the result of the low-resolution image super-resolution is obtained.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to an image super-resolution method based on a hybrid network structure of convolutional neural networks and Transformers. Background Technology

[0002] Image super-resolution is an important research area in computer vision, belonging to a classic inverse problem in image reconstruction tasks. The main goal of image super-resolution is to recover the original high-resolution image from a degraded low-resolution image. Due to its wide application in computer vision fields such as medical imaging, video surveillance, and historical photo restoration, this technique is receiving increasing attention.

[0003] Traditional image super-resolution techniques mostly reconstruct high-resolution images by extracting low-level features that have limited expressive power for texture details such as image contours, thus significantly limiting the reconstruction results. In recent years, deep learning-based methods have been able to extract more expressive image features from image data and adaptively learn the mapping relationship between low-resolution and high-resolution images. In particular, methods based on convolutional neural networks (CNNs) and Transformers have been widely applied in the field of image super-resolution, achieving better reconstruction results. Related research has demonstrated that CNN models can capture local image features, while Transformer models can capture global image features. However, ensuring that the global content of the image remains unchanged and accurately restoring image details during image super-resolution still faces significant challenges. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an image super-resolution method based on a hybrid network structure of convolutional neural networks and Transformers, which ensures good restoration of local details and global contours while reconstructing high-resolution images. The method consists of three stages. The first stage is a shallow feature extraction stage, which performs preliminary extraction of image features. The second stage is a deep feature extraction stage, in which deep feature extraction is achieved through a backbone network. The third stage is a reconstruction stage, which reconstructs the image to obtain the super-resolution high-resolution image.

[0005] This application provides an image super-resolution method based on a hybrid network structure of convolutional neural networks and Transformers. The image super-resolution network mainly comprises an initial convolutional module, a backbone network, and an upsampling module. The backbone network consists of residual groups and includes multiple dual-path collaborative modules composed of dual-attention Transformer modules, multi-scale convolutional modules, and selection fusion modules. The determined method includes:

[0006] Acquire the low-resolution image to be processed and perform preprocessing;

[0007] Construct an image super-resolution network based on a hybrid structure of convolutional neural networks and Transformers;

[0008] The preprocessed low-resolution image is input into the initial convolution module for preliminary processing to obtain shallow features of the image.

[0009] The shallow features of the image are input into the backbone network to obtain the deep features of the image.

[0010] The deep image features are input into the upsampling module to obtain the reconstructed high-resolution image;

[0011] The image super-resolution network based on the hybrid structure of convolutional neural network and Transformer is optimized based on the target value to obtain the final image super-resolution network structure; wherein, the target value includes: the original high-resolution image corresponding to the low-resolution image.

[0012] Furthermore, the initial convolution module includes only one basic convolution process; the step of inputting the preprocessed low-resolution image into the initial convolution module for preliminary processing includes:

[0013] The preprocessed low-resolution image is input into the initial convolution module to obtain four-dimensional batch image features, which are batch size, number of color channels, image height, and image width.

[0014] The shallow image features are input into the backbone network; the backbone network is mainly composed of residual groups; the residual groups include multiple dual-path collaborative modules; the dual-path collaborative modules include a dual-attention Transformer module, a multi-scale convolution module, and a selection fusion module; the shallow image features are input into the backbone network to obtain the image features of the last layer, and then aggregated with the shallow image features to obtain the deep image features, including:

[0015] The shallow image features are input into the first residual set in the backbone network;

[0016] The shallow image features are input into the dual-path collaborative module in the first residual group, and after processing by multiple dual-path collaborative modules, they are aggregated with the shallow image features to obtain intermediate image features. The intermediate image features are used as input features for the next residual group, and after processing by the residual group, enhanced image features are obtained. The enhanced image features are then used as input features for the next residual group, and the process continues until all residual groups are processed to obtain the final layer of image features.

[0017] The dual-path collaborative module includes a dual-attention Transformer module, a multi-scale convolution module, and a selection fusion module. The shallow image features are divided into channels to obtain segmented image features. These segmented image features are then input into two branches for processing. One branch contains an initial convolution and a dual-attention Transformer module, while the other contains an initial convolution and a multi-scale convolution module to obtain global and local features. The global and local features are then input into the selection fusion module to obtain image fusion features with both global and local information. These image fusion features are used as input features for the next dual-path collaborative module. After processing by multiple dual-path collaborative modules, enhanced image fusion features are obtained and aggregated with the shallow image features to obtain the initial intermediate features of the image.

[0018] The segmented image features are input into the branch containing an initial convolution and a dual-attention Transformer module; the segmented image features are processed by the initial convolution to obtain initial global features; the initial global features are input into the dual-attention Transformer module; the dual-attention Transformer module includes a first-layer normalization unit and a window attention module, a second-layer normalization unit and a feedforward network; specifically, the global features are obtained through the following processing steps:

[0019] The initial global features are input into the normalization unit to obtain normalized image features;

[0020] The normalized image features are input into the window attention module and aggregated with the initial global features to obtain the first layer of image features; the window attention module includes:

[0021] The normalized features are processed through three convolutions and a depthwise separable convolution to obtain query, key, and numerical vectors. Matrix multiplication of the query and key vectors is calculated to obtain spatial attention, and matrix multiplication with the numerical vector is calculated to obtain image features of the attention spatial characteristics.

[0022] The normalized features are processed through a channel attention module to obtain channel attention, including:

[0023] The normalized features are passed through a global pooling unit to obtain pooled features; the pooled features are then input into a convolutional unit, and then passed through a linear activation unit to obtain the channel attention.

[0024] The intermediate global features are obtained by multiplying the channel attention with the image features of the attention space characteristics.

[0025] Furthermore, the first layer image features are input into the normalization unit of the second layer to obtain the normalized first layer image features;

[0026] The normalized first-layer image features are input into the feedforward network, including:

[0027] The normalized first-layer image features are input into a fully connected layer, and the second-layer image features are obtained through a linear unit and another fully connected layer.

[0028] The global features are obtained by aggregating the second layer image features with the first layer image features.

[0029] Furthermore, the segmented image features are input into the branch containing an initial convolution and a multi-scale convolution module; the segmented image features are processed by the initial convolution to obtain initial local features; the initial local features are input into the multi-scale convolution module, including:

[0030] The initial local features are respectively processed through three dilated convolutions with different dilation rates, and then aggregated with the initial local features, and then processed through a convolution unit to obtain enhanced local features;

[0031] The enhanced local features are aggregated with the initial local features to obtain the local features.

[0032] Furthermore, the global and local features are input into the selection and fusion module to obtain the image fusion features containing both global and local information; specifically, the global and local features are fused in the following manner:

[0033] The cosine similarity between the global features and the local features is calculated, and a similarity weight matrix is ​​obtained through activation units. The similarity weight matrix is ​​then multiplied by the global features and the local features to extract the similar and dissimilar features of the global and local features.

[0034] The aforementioned global and local similar features are aggregated through a cascading method to obtain similar features;

[0035] The aforementioned global and local dissimilarity features are transformed using a spatial feature transformation module to obtain interactive global and local features, including:

[0036] The global and local features are both learned through convolutional and activation units to learn modulation parameters, and then modulated by dot products and summation with the corresponding features, finally obtaining the interactive global and local features.

[0037] The global and local features of the aforementioned interactions are aggregated through cascading to obtain dissimilar features;

[0038] The similar and dissimilar features are aggregated and then processed through a convolutional unit to obtain the image fusion features containing both global and local information.

[0039] Furthermore, the deep image features are input into the upsampling module; this module includes two convolutional units and an upsampling unit; the upsampling module obtains the reconstructed high-resolution image in the following manner:

[0040] The aforementioned deep image features are input into a convolutional unit to obtain pre-processed deep image features;

[0041] The deep features of the image after preliminary processing are then passed through an upsampling unit to obtain high-resolution image features;

[0042] The high-resolution image features are processed through a convolutional unit to obtain the reconstructed high-resolution image.

[0043] Furthermore, the image super-resolution network based on the hybrid structure of convolutional neural networks and Transformers, which is optimized according to the target numerical value, is used to obtain the final image super-resolution network, including:

[0044] Based on the original high-resolution image corresponding to the low-resolution image, the l1 loss function of the image super-resolution network is calculated.

[0045] Furthermore, the L1 loss function is calculated as follows:

[0046]

[0047] Where f(x) represents the predicted value, y represents the true value, and ||·||1 represents the L1 norm.

[0048] The present invention provides a computer system including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the image super-resolution method based on a hybrid network structure of convolutional neural networks and Transformers.

[0049] Beneficial effects: Compared with existing image super-resolution methods, this method has the following advantages:

[0050] 1. This invention proposes a dual-path collaborative module that integrates a dual-attention Transformer module, a multi-scale convolution module, and a selection fusion module, by combining convolutional neural networks and Transformer structures. This module leverages the advantages of convolutional neural networks in reconstructing local details and Transformer structures in restoring global contours, thereby improving the quality of reconstructed images.

[0051] 2. This invention utilizes a dual-attention Transformer module to enhance the comprehensive attention to image features from both spatial and channel dimensions. Spatially, it focuses more on the correlation between pixels, while channel-wise, it focuses more on the importance of channels, thereby comprehensively acquiring global image information.

[0052] 3. This invention uses a multi-scale convolution module to obtain local features at different scales, preserving the diversity of local features and thus enhancing the local content of the image.

[0053] 4. This invention achieves adaptive feature fusion by selecting a fusion module based on the similarity between global and local features, ensuring that complete image contours and fine texture details are preserved during the fusion process.

[0054] 5. The experimental results in the specific embodiments below confirm the effectiveness and superiority of the present invention. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the image super-resolution method based on a hybrid network structure of convolutional neural network and Transformer according to the present invention.

[0056] Figure 2 This is a schematic diagram of the dual-path collaborative module of the present invention.

[0057] Figure 3 This is a schematic diagram of the dual-attention Transformer module of the present invention.

[0058] Figure 4 This is a schematic diagram of the multi-scale convolution module of the present invention.

[0059] Figure 5 This is a schematic diagram of the fusion module selected for this invention.

[0060] Figure 6 This is a schematic diagram of the upsampling module of the present invention.

[0061] Figure 7 This is a super-resolution image rendering of the present invention. Detailed Implementation

[0062] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0063] This invention provides an image super-resolution method based on a hybrid network structure of convolutional neural networks (CNNs) and Transformers. It improves upon existing CNN- or Transformer-based network architectures by adding a Transformer branch to the CNN-based structure. Simultaneously, based on the features extracted from local and global information by the CNN and Transformer respectively, an adaptive fusion method is used to achieve interaction and integration between the two. Furthermore, a multi-scale process is incorporated into the CNN to extract local features with multiple contents. The Transformer structure simultaneously focuses on both spatial and channel aspects to obtain more comprehensive global features. The network structure flow diagram based on this invention is shown below. Figure 1 As shown.

[0064] Specifically, an image super-resolution method based on a hybrid network structure of convolutional neural networks and Transformers, according to an embodiment of the present invention, first extracts features from the input image to obtain shallow features of the image, i.e., features of dimensions B×C×H×W, where B is the batch size of the image, set to 8, C is the number of channels of the image features, set to 64, and H and W are the height and width of the image, respectively; then, the shallow image is input into a deep network, i.e., a network composed of multiple residual groups; wherein, each residual group contains multiple dual-path collaborative modules, and each module passes through a dual-attention Transformer network. The sformer module, multi-scale convolution module, and selection fusion module implement the feature enhancement process to obtain deep features, i.e., dimensions B×C×H×W, where B is 8, C is 64, and H and W are the height and width of the image features, consistent with the shallow features. Finally, the upsampling module restores the deep features to the image space, i.e., dimensions B×C×H'×W', where B is 8, C is 3, and H' and W' correspond to the image height and width obtained by different super-resolution factors, respectively. For example, the result of 2x super-resolution should be expressed as H'=2×H, so as to realize the reconstruction process of low-resolution to high-resolution images.

[0065] The dual-attention Transformer module first obtains the query, key, and numerical vectors related to the features through linear transformation. At the same time, it obtains the weight matrix of the importance of different channels of the input features through the channel attention unit. Then, it calculates the spatial features that are applied to the numerical vector by the query and key vectors to obtain global features that pay attention to both space and channels.

[0066] The multi-scale convolution module acquires local information at different scales through dilated convolutions with different dilation rates. At the same time, it aggregates with the original input features to obtain multifaceted local features, thereby enhancing the attention to local information and obtaining more detailed local features.

[0067] The fusion module selects and calculates the implicit similarities and differences between global and local features, converts them into corresponding weight matrices, adaptively achieves the fusion of similar features and the interaction of dissimilar features, and enhances the overall features of the image.

[0068] In the upsampling module, the image is transformed from the feature space to the original space through linear transformation and pixel rearrangement, thereby obtaining a reconstructed high-resolution image.

[0069] The following is combined Figure 1 The detailed steps of the above method are explained in detail:

[0070] Step 1: Obtain the low-resolution image to be reconstructed, with a resolution of 64×64;

[0071] Step 2: Extract shallow features of the input low-resolution image through the initial convolution module, namely B×C×H×W, where B represents the batch size of the input image, C represents the number of channels of the image, H represents the height of the image, and W represents the width of the image; in this example, the batch size is 8, the number of channels is 64, and the height and width of the image are 64 and 64 respectively.

[0072] Step 3: Input the extracted shallow features (dimension 8×64×64×64) into the backbone network to obtain the deep features of the image, while keeping the feature dimensions of the image unchanged (the dimension of the deep features is still 8×64×64×64).

[0073] Step 4: Input the deep features of the image (i.e., spatial features with dimensions of 8×64×64×64) into the upsampling module to obtain the reconstructed image. The dimensions are transformed from the original feature space to the image space. If performing a 2x image super-resolution, the dimension is 8×3×128×128; for a 3x image super-resolution, the dimension is 8×3×192×192; and for a 4x super-resolution, the dimension is 8×3×256×256.

[0074] In this example, the process of obtaining the low-resolution image in step 1 includes: downloading the dataset from the open-source website, constructing data pairs of 64×64 low-resolution images and 128×128, 192×192, and 256×256 high-resolution images, which correspond to super-resolution ratios of 2x, 3x, and 4x, respectively.

[0075] In this example, step 2, which extracts shallow features, uses a basic convolutional unit, the expression of which is as follows:

[0076] F shallow =Conv 3×3 (I LR );

[0077] Among them, F shallow This represents shallow features with dimensions B×C×H×W, Conv 3×3 (·) represents a basic convolutional unit with a kernel size of 3×3 and a stride of 1. LR This indicates an input low-resolution image with a resolution of 64×64.

[0078] In this example, step 3 extracts the shallow image features F. shallow The input is to the backbone network, which mainly consists of multiple residual sets. The residual sets are then used... Processing yields different features F n-1 As input to the next set of residuals, the expression for obtaining deep features is as follows:

[0079]

[0080] Among them, F deep This represents the obtained image depth features, with dimensions B×C×H×W. F represents the processing procedure for the nth residual group. n-1 This represents the image features after processing the (n-1)th residual group, i.e., the intermediate features of the image, with dimensions B×C×H×W.

[0081] In the aforementioned backbone network, the shallow input features are fed into the residual group, processed by multiple dual-path collaborative modules, and then aggregated with the initial input features to obtain the output features of the residual group, the expression of which is as follows:

[0082]

[0083] Among them, F n This represents the characteristics of the nth residual group after processing. Indicates m groups of dual-path collaborative modules, F n-1 This represents the characteristics of the (n-1)th residual group after processing.

[0084] For the feature processing process of each dual-path collaborative module, the module structure is as follows: Figure 2 As shown, the specific steps are described below:

[0085] 3-1. The input image features are processed by channel partitioning and evenly distributed into two branches, meaning that the image feature dimension obtained by each branch is [dimension value missing]. One branch primarily extracts global features from the image using a dual-attention Transformer module, while the other branch primarily extracts local features using a multi-scale convolution module. The expressions for obtaining the features from these two branches are as follows:

[0086] F global ,F local =Split(F n-1 );

[0087] Among them, F n-1 The input image features are represented by the dimension B×C×H×W, Split(·) represents the channel partitioning process, and F global Represents the global features of the image, F local Representing local features of an image, both have a dimension of 1.

[0088] 3-2. Input the features obtained in step 3-1 into the branch mainly composed of the dual-attention Transformer module. First, pass it through a basic convolutional unit to obtain the initial global features, the expression of which is as follows:

[0089] F′ global =Conv 3×3 (F global );

[0090] Among them, F′ global Indicates that after passing through the basic convolutional unit Conv 3×3 (·), the convolution kernel size is 3×3, the stride is 1, and the initial global features obtained after processing, F global This represents the global features of the input image.

[0091] 3-3. The initial global features obtained in step 3-2 are processed through a dual-attention Transformer module, the module structure of which is as follows: Figure 3 As shown. First, the initial global features of the input are normalized and then split into two parts according to the number of heads M and the number of channels C to compute attention in parallel. On one hand, we compute the features F′ global Divide the window into non-overlapping H×sw axial windows, and represent the feature of the i-th window as X. i ,in X represents the number of vertical windows. i Let F′ represent the image features within the i-th vertical window. This method is called vertical window attention, denoted as Av-WA. On the other hand, we will use F′ global Divide the window into non-overlapping sh×W axial windows, and represent the feature of the j-th window as X. j ,in X represents the number of horizontal windows. jLet X represent the image features within the j-th horizontal window, and this method is called horizontal window attention, denoted as Ah-WA. Here, sw and sh represent the width of the vertical window and the height of the horizontal window, respectively, and X... i and X j Both are input features F′ global The sub-part. For feature X within the vertical window. i and the feature X within the horizontal window j respectively through the corresponding projection matrix W i Q W i K W i V and By performing a linear transformation, we obtain the query, key, and numerical vectors needed to compute spatial attention, which in turn yields the vertical window feature X. i Attention (Q) i ,K i V i ) and horizontal window feature X j Attention (Q) j ,K j V j Its expression is as follows:

[0092] (Q i ,K i V i )=(X i W i Q ,X i W i K ,X i W i V );

[0093]

[0094] in, Represents the projection matrix. Indicates the projection dimension.

[0095] 3-4. The vertical window features X obtained in step 3-3 are... i and horizontal window feature X j Query, Key, Numeric Vector (Q) i ,K i V i ) and (Q j ,K j V j To calculate spatial attention Y i and Yj Its expression is as follows:

[0096]

[0097]

[0098] Among them, Y i and Y j Representing the vertical window feature X respectively i and horizontal window feature X j The attention feature is represented by B, which indicates dynamic positional encoding.

[0099] 3-5. Reshape and fuse the attention features obtained in steps 3-4 in sequence, and then combine them in a cascaded manner to form the final spatial attention F. A-WA Its expression is as follows:

[0100] F A-WA =Axial-WA(F′) global =Concat(Y) v ,Y h );

[0101] Among them, Y v Y represents the attention features of all vertical windows. i The set of Y h Y represents the attention feature of all horizontal windows. j A set of.

[0102] 3-6. After obtaining the spatial attention from step 3-5, the process involves average pooling, ReLU activation function, and convolution.

[0103] The channel attention module, consisting of the sigmoid activation function, processes the initial global features F′ of the input. global The channel attention matrix F is obtained by processing through the channel attention module to determine the importance of the channels. SGCA In this process, the expression is as follows:

[0104] F SGCA =SGCA(F′) global )=Sigmoid(ConvLayer(Avg(F′ global )))⊙V;

[0105] Where Avg(·) represents the average pooling process, ConvLayer(·) represents the structure consisting of the ReLU activation function and convolutional units, Sigmoid(·) represents the activation function process, ⊙ represents the dot product, and V represents the numerical vector consistent with the attention process of the computation window, i.e., V i and Vj Let V represent the numerical vectors of the i-th vertical window space feature and the j-th horizontal window space feature, respectively.

[0106] Steps 3-3 to 3-6 above can be summarized as follows:

[0107] F DATM =DATM(LN(F′) global ))+F′ global ;

[0108] Where DATM(·) represents the dual-attention Transformer module, LN(·) represents the normalization unit, and F DATM This represents the features obtained after processing by the dual-path collaborative module.

[0109] 3-7. The feature F obtained in step 3-6 DATM The input is fed into the normalization unit of the second layer, and the global features are enhanced through a feedforward network to obtain the overall global features X. GF The feedforward network consists of two fully connected layers and linear units, and the expression for this process is as follows:

[0110] X GF =FFN(LN(F DATM ))+F DATM ;

[0111] Where FFN(·) represents the feedforward network layer, X GF This represents the final global feature.

[0112] 3-8. Input the features obtained in step 3-1 into the branch mainly composed of multi-scale convolutional modules, with the module structure as follows: Figure 4 As shown. First, it is passed through a basic convolutional unit to obtain initial local features, the expression of which is as follows:

[0113] F′ local =Conv 3×3 (F local );

[0114] Among them, F′ local Indicates that after passing through the basic convolutional unit Conv 3×3 (·) The initial local features obtained after processing, with a convolution kernel size of 3×3, a stride of 1, and F local This represents the features of the input image.

[0115] 3-9. Input the initial local features obtained in step 3-8 into dilated convolutions with different receptive fields to obtain three features focusing on different scales of content. These features are then concatenated with the original input features and aggregated through a basic convolutional unit. The expression for the entire process is as follows:

[0116] X LF =Conv 1×1 (Concat(F′ local Conv d＝k (F′ local )))+F′ local ;

[0117] Among them, Conv 1×1 (·) represents the basic convolutional unit with a kernel size of 1×1 and a stride of 1. Concat(·) represents the connection process. d＝k (·) represents a dilated convolution operation with an expansion rate of k (k = 1, 3, 5), X LF This represents the enhanced local features obtained.

[0118] 3-10. Input the global and local features obtained in steps 3-7 and 3-9 into the selection and fusion module. The module structure is as follows: Figure 5 As shown, firstly, cosine similarity is used to calculate the similarity matrices of the global and local features of the input from both channel and spatial perspectives. and Then, based on the feature values ​​in the matrix within the ranges [0,1] and [-1,0), which are used as the feature criteria for distinguishing between similarity and dissimilarity, a selection matrix is ​​calculated using the Sigmoid activation function. Its expression is as follows:

[0119]

[0120]

[0121] M = Reshape(Sigmoid(M) c ) T *Sigmoid(M s ));

[0122] in, This represents the calculation process of cosine similarity. Each value of M is between -1 and 1. The closer it is to 1, the higher the similarity.

[0123] 3-11. Use the selection matrix obtained in step 3-10 to filter the features generated from the two branches to distinguish between similar and dissimilar content. For features with high similarity, cascade processing is directly used to retain the common information among these features, as shown in the following expression:

[0124]

[0125]

[0126]

[0127] in, t = trans or t = cnn represents similar features, F sim This indicates similar features existing between global and local features, meaning their feature values ​​are between [0,1], and the dimension is...

[0128] 3-12. For dissimilar features, a spatial feature transformation module is used to facilitate interaction, aiming to ensure that the global and local features obtained from steps 3-7 and 3-9 complement each other while preserving their respective characteristics. This spatial transformation module needs to be learned based on different traits, enabling it to simultaneously focus on the unique features of both branches. The expression for this process is as follows:

[0129]

[0130]

[0131]

[0132] F dis =Concat(X' u );

[0133] in, and X' u u = trans or u = cnn represent different features and the modified feature, respectively, and SFT(·) represents the function used to obtain the modulation coefficient γ. v and β v The spatial feature transformation module (v = trans or v = cnn) where γ v and β v The learnable parameters are obtained through processing with 3×3 and 1×1 convolutional kernels, both with a stride of 1, F dis This represents dissimilar features that exist between global and local features, meaning their feature values ​​are between [-1, 0), and the dimension is...

[0134] 3-13. Integrate the similar and dissimilar content obtained in steps 3-11 and 3-12 into the same feature space, while maintaining the same dimension as the initial feature space. The expression is as follows:

[0135]

[0136] Among them, F SFM This represents the output feature of the selected fusion module, which is the output feature obtained after passing through the last module of the (n-1)th residual group, i.e., the output feature F of the (n-1)th residual group.n-1 , The aggregation process is represented by Conv(·), which represents the basic convolutional unit for reducing feature dimensions. The kernel size is 3×3, the stride is 1, and the number of channels is set to 1.

[0137] In this example, step 4 will extract the deep features F of the image. deep The input is fed into the upsampling module, which mainly consists of basic convolutional units and pixel rearrangement. The module structure is as follows: Figure 6 As shown. After the module processes the features, it restores the image from the feature space to the original image space, obtaining the final reconstructed image. The specific steps are as follows:

[0138] 4-1. Input the depth features obtained in step 3 into the basic convolutional unit to obtain the initial reconstructed features, the expression of which is as follows:

[0139] F mid =Conv 3×3 (F deep );

[0140] Among them, F deep F represents the depth features obtained in step 3. mid Represents the initial reconstructed features, Conv 3×3 (·) represents a basic convolutional unit with a kernel size of 3×3 and a stride of 1.

[0141] 4-2. Input the initial reconstructed features obtained in step 4-1 into the pixel rearrangement unit to obtain the reconstructed features, the expression of which is as follows:

[0142] F out =PixelShuffle(F mid );

[0143] Among them, F out Represents the reconstructed features, and PixelShuffle(·) represents the pixel rearrangement unit.

[0144] 4-3. The reconstructed features obtained in step 4-2 are restored to the original image space through basic convolutional units to obtain the final reconstructed image, the expression of which is as follows:

[0145] SR = Conv 3×3 (F out );

[0146] Among them, Conv 3×3 (·) represents the basic convolutional unit with a kernel size of 3×3 and a stride of 1. SR represents the reconstructed high-resolution image.

[0147] The following examples illustrate the implementation effects of this invention in practical applications:

[0148] In one experiment, the L1 loss function was used to optimize the image super-resolution network. The image super-resolution network was trained for a total of 500K iterations with a learning rate of 2×10⁻⁶. -4 The input image is halved at [250K, 400K, 450K, 475K]; the input image is cropped to a size of 64×64; the batch size is set to 8; here, this application uses PyTorch as the code implementation and the Adam optimizer to optimize the model.

[0149] This application trains the model on the classic DF2K dataset, which includes both DIV2K and Flickr2K datasets. Additionally, the model is evaluated on five standard datasets: Set5, Set14, B100, Urban100, and Manga109. To evaluate the model's reconstruction capabilities, two common metrics, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), are used to compute reconstruction results on the Y channel of the YCbCr color space.

[0150] Table 1 below shows the test results on five standard datasets. Lower values ​​for all metrics indicate better reconstruction and higher image quality.

[0151]

[0152] Table 1. Experimental data on image super-resolution using various standard datasets.

[0153] As can be seen, the image super-resolution network in this invention achieves superior image super-resolution results. Besides preserving the overall image content during reconstruction, it also restores features such as image texture details. The subjective effect of the super-resolution is evident. Figure 7 .

[0154] Based on the same inventive concept, this invention provides an image super-resolution system based on a hybrid network structure of convolutional neural networks and Transformers, including an input module, an image super-resolution network model, and an output module. The input module acquires a low-resolution image and inputs it into the image super-resolution network model. The image super-resolution network model performs a super-resolution process on the input low-resolution image to obtain an image that meets the super-resolution ratio requirement. The output module reconstructs a high-resolution image based on the image features output by the super-resolution network model. The image super-resolution network model consists of a series of residual sets. This includes a dual-path collaborative module, which consists of a dual-attention Transformer module, a multi-scale convolution module, and a selection fusion module. Initial feature extraction is performed on the input low-resolution image, and the extracted features are fed into a residual group. After processing through multiple residual groups, image depth features are obtained. Each residual group contains multiple dual-path collaborative modules. This module divides the input features into two parts: one part passes through the dual-attention Transformer module to obtain global image features, and the other part passes through the multi-scale convolution module to obtain local image features. The global and local features are then adaptively fused and interacted through the selection fusion module. The specific implementation of each module in the network model is described in the above method embodiments and will not be repeated here. Anything not described in detail in this invention is prior art.

[0155] Based on the same inventive concept, an embodiment of the present invention provides a computer system including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the steps of the above-described image super-resolution method based on a hybrid network structure of convolutional neural networks and Transformers.

Claims

1. An image super-resolution method based on a hybrid network structure of convolutional neural networks and Transformers, characterized in that: First, feature extraction is performed on the input image to obtain shallow features, i.e., dimensions B×C×H×W, where B is the batch size of the image, set to 8, C is the number of image feature channels, set to 64, and H and W are the height and width of the image, respectively. Then, the shallow image is input into a deep network, which consists of multiple residual groups. Each residual group contains multiple dual-path collaborative modules. Each module enhances the features through a dual-attention Transformer module, a multi-scale convolution module, and a selection fusion module, resulting in deep features, i.e., dimensions B×C×H×W. Finally, an upsampling module restores the deep features to the image space, i.e., dimensions B×C'×H'×W', where B is 8, C' is 3, and H' and W' correspond to the height and width of the image obtained at different super-resolution ratios, respectively. The details are as follows: The process of obtaining low-resolution images in step 1 includes: downloading datasets from open-source websites, constructing data pairs of 64×64 low-resolution images and 128×128, 192×192, and 256×256 high-resolution images, corresponding to super-resolution ratios of 2x, 3x, and 4x respectively; Step 2 extracts shallow features using basic convolutional units, the expression of which is as follows: ； in, This represents shallow features, with dimensions B×C×H×W. This represents a basic convolutional unit with a kernel size of 3×3 and a stride of 1. This indicates an input low-resolution image with a resolution of 64×64; Step 3 extracts shallow features from the image. The input is to the backbone network, which consists of multiple residual sets. The residual sets are then used... Different features were obtained through processing. As input to the next set of residuals, the expression for obtaining deep features is as follows: ； in, This represents the obtained image depth features, with dimensions B×C×H×W. Indicates the first The process of processing group residuals Indicates the first The image features after processing the residual groups, i.e. the intermediate features of the image, have dimensions of B×C×H×W; In the aforementioned backbone network, intermediate features are... The input is fed into the residual set, processed by multiple dual-path collaborative modules, and then aggregated with the initial input features to obtain the output features of the residual set, whose expression is as follows: ； in, Indicates the first Characteristics of the residual groups after processing. express Dual-path collaborative module, Indicates the first Characteristics of the residual groups after processing; The specific steps for processing features by each dual-path collaborative module are described below: 3-1. The input image features are processed by channel partitioning and evenly distributed into two branches, meaning that the image feature dimension obtained by each branch is [dimension value missing]. One branch extracts global features of the image using a dual-attention Transformer module, while the other branch extracts local features using a multi-scale convolution module. The expressions for obtaining the features from these two branches are as follows: ； in, The input image features are represented by dimensions B×C×H×W. This indicates the channel partitioning process. Represents the global features of an image, Representing local features of an image, both have a dimension of 1. ; 3-2. The global features obtained in step 3-1 The input to the dual-attention Transformer module branch first passes it through a basic convolutional unit to obtain initial global features, the expression of which is as follows: ； in, Indicates passing through the basic convolutional unit The convolution kernel size is 3×3, the stride is 1, and the initial global features obtained after processing are... Represents the global features of the input image; 3-3. The initial global features obtained in step 3-2 are processed through a dual-attention Transformer module; firstly, the initial global features are normalized, and then divided into two parts according to the number of heads M1 and the number of channels C, so as to compute attention in parallel; on the one hand, the features are processed... Divide into non-overlapping Axial window, and the features of the i-th window are represented as ,in Indicates the number of vertical windows. Representing the image features within the i-th vertical window, this method is called vertical window attention, denoted as Av-WA; on the other hand, Divide into non-overlapping Axial window, and the features of the j-th window are represented as ,in Indicates the number of horizontal windows. Let represent the image features within the j-th horizontal window, and this method is called horizontal window attention, denoted as Ah-WA; where sw and sh represent the width of the vertical window and the height of the horizontal window, respectively. and All are input features The sub-part; for features within the vertical window Features within the horizontal window respectively through the corresponding projection matrix , , and , , A linear transformation is performed to obtain the query, key, and numerical vectors needed to compute spatial attention, which in turn yields the features required for computing the vertical window. Attention and horizontal window features Attention Its expression is as follows: ； ； in, Represents the projection matrix. Indicates the projection dimension; 3-4. The vertical window features obtained in step 3-3 are... and horizontal window features Query, Keyword, Numeric Vector and To calculate spatial attention and Its expression is as follows: ； ； in, and Representing vertical window features and horizontal window features The attention feature, where P represents dynamic position encoding; 3-5. Reshape and fuse the attention features obtained in steps 3-4 in sequence, and then combine them in a cascaded manner to form the final spatial attention. Its expression is as follows: ； in, Represents the attention features of all vertical windows The set, Represents attention features of all horizontal windows A set; 3-6. After obtaining the spatial attention from step 3-5, the initial global features of the input are processed through a channel attention module consisting of average pooling, ReLU activation function, convolution, and sigmoid activation function. The channel attention matrix is ​​obtained by processing through the channel attention module to determine the importance of each channel. In this process, the expression is as follows: ； in, This represents the average pooling process. Indicates by The structure consisting of activation functions and convolutional units This represents the activation function process. Represents the dot product. This represents a numerical vector consistent with the computational window attention process, i.e. and Representing features respectively The i-th vertical window spatial feature value vector and the j-th horizontal window spatial feature value vector; Steps 3-3 to 3-6 above can be summarized into the following expression: ； in, This refers to the dual-attention Transformer module. Represents the normalized unit. This represents the features obtained after processing by the dual-path collaborative module; 3-7. The features obtained in step 3-6 The input is fed into the normalization unit of the second layer, and the global features are enhanced through a feedforward network to obtain the overall global features. The feedforward network consists of two fully connected layers and linear units, and the expression for this process is as follows: ； in, Indicates a feedforward network layer. This represents the final global feature; 3-8. Input the features obtained in step 3-1 into the branches of the multi-scale convolutional module; first, pass them through a basic convolutional unit to obtain initial local features, the expression of which is as follows: ； in, Indicates passing through the basic convolutional unit The initial local features obtained after processing have a convolution kernel size of 3×3 and a stride of 1. Represents local features of the input; 3-9. Input the initial local features obtained in step 3-8 into dilated convolutions with different receptive fields to obtain three features focusing on different scales of content. These features are then concatenated with the original input features and aggregated through a basic convolutional unit. The expression for the entire process is as follows: ； in, This represents a basic convolutional unit with a kernel size of 1×1 and a stride of 1. Indicates the connection process. This represents a dilated convolution operation with an expansion rate of k, where k = 1, 3, 5. This represents the enhanced local features obtained; 3-10. Input the global and local features obtained in steps 3-7 and 3-9 into the selection and fusion module; firstly, use cosine similarity to calculate the similarity matrix of the input global and local features from both channel and spatial perspectives. and Then, based on the feature values ​​in the matrix that fall within the ranges [0,1] and [-1,0), the selection matrix is ​​calculated using the Sigmoid activation function to distinguish between similar and dissimilar features. Its expression is as follows: ； ； ； Where C() represents the calculation process of cosine similarity, Each value is between -1 and 1, and the closer it is to 1, the higher the similarity. 3-11. Use the selection matrix obtained in step 3-10 to filter the features generated by the two branches to distinguish between similar and dissimilar content; for features with high similarity, directly use cascade processing to retain the common information present in these features, the expression of which is as follows: ； ； ； in, , or Indicates similar features, This indicates similar features existing between global and local features, meaning their feature values ​​are between [0,1], and the dimension is... ; 3-12. For dissimilar features in the global and local features, a spatial feature transformation module is used to facilitate interaction, aiming to ensure that the global and local features obtained from steps 3-7 and 3-9 complement each other while preserving their respective features. This spatial feature transformation module needs to be learned based on different traits so that they can simultaneously focus on the unique features of both branches. The expression for this process is as follows: ； ； ； ； in, and , or These represent the different features and the modified features, respectively. Indicates the use of obtaining modulation coefficients and The spatial feature transformation module, in which p = trans or p = cnn in and The learnable parameters are obtained by processing with 3×3 and 1×1 convolutional kernels, with a stride of 1 for each. This represents dissimilar features that exist between global and local features, meaning their feature values ​​are between [-1, 0), and the dimension is... ; 3-13. Integrate the similar and dissimilar content obtained in steps 3-11 and 3-12 into the same feature space, while maintaining the same dimension as the initial feature space. The expression is as follows: ； in, This indicates the output feature selected from the fusion module, which is the output feature obtained after passing through the last module of the (n-1)th residual group, i.e., the output feature of the (n-1)th residual group. , Indicates the aggregation process. This represents a basic convolutional unit with reduced feature dimensions, a kernel size of 3×3, a stride of 1, and a number of channels set to [value missing]. ; Step 4: Extract deep features from the image The image is input into the upsampling module, which consists of basic convolutional units and pixel rearrangement. After the features are processed by this module, the image is restored from the feature space to the original image space, resulting in the final reconstructed image. The specific steps are as follows: 4-1. Input the depth features obtained in step 3 into the basic convolutional unit to obtain the initial reconstructed features, the expression of which is as follows: ； in, This represents the depth features obtained in step 3. Indicates the initial reconstruction features. This represents a basic convolutional unit with a kernel size of 3×3 and a stride of 1. 4-2. Input the initial reconstructed features obtained in step 4-1 into the pixel rearrangement unit to obtain the reconstructed features, the expression of which is as follows: ； in, Indicates reconstruction features, Indicates a pixel rearrangement unit; 4-3. The reconstructed features obtained in step 4-2 are restored to the original image space through basic convolutional units to obtain the final reconstructed image, the expression of which is as follows: ； in, This represents a basic convolutional unit with a kernel size of 3×3 and a stride of 1. This represents the reconstructed high-resolution image.

2. The method according to claim 1, characterized in that, include: The dual-attention Transformer module first obtains the query, key, and numerical vectors related to the features through linear transformation. At the same time, it obtains the weight matrix of the importance of different channels of the input features through the channel attention unit. Then, it calculates the spatial features that are applied to the numerical vector by the query and key vectors to obtain global features that pay attention to both space and channels. The multi-scale convolution module acquires local information at different scales through dilated convolutions with different dilation rates. It also aggregates with the original input features to obtain multifaceted local features. Without changing the size of the original features, it expands the receptive field by changing the convolution form, thereby enhancing the attention to local information.

3. The method according to claim 1, characterized in that, include: The fusion module selects and calculates the implicit similarities and differences between global and local features, converts them into corresponding weight matrices, adaptively achieves the fusion of similar features and the interaction of dissimilar features, and enhances the overall features of the image.

Images