A super-resolution reconstruction method based on a hybrid attention mechanism
By introducing a hybrid attention mechanism in image super-resolution reconstruction, combining channel and spatial attention modules, the performance limitations caused by dimensionality reduction operations in existing methods are addressed, thereby improving reconstruction results and speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-09
- Publication Date
- 2026-03-27
AI Technical Summary
In existing image super-resolution reconstruction methods, the channel attention module needs to perform dimensionality reduction operations, which leads to insufficient performance and difficulty in capturing contextual information in the spatial dimension, resulting in limited improvement in reconstruction effect.
A hybrid attention mechanism is adopted, combining the channel attention module ECA and the spatial attention module ESA to construct a hybrid attention module, which is integrated into the residual group to emphasize important information and capture dependencies, thereby improving feature representation.
It improves the quality of reconstructed images, increases reconstruction speed and effectiveness, and focuses on learning more important features by enhancing sensitivity in channel and spatial dimensions, taking into account inter-channel and spatial contextual relationships.
Smart Images
Figure CN115908130B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image processing, and particularly relates to a super-resolution reconstruction method based on a hybrid attention mechanism. BACKGROUND
[0002] Image resolution is an index depicting the pixel density and the degree of detail contained in an image. Generally speaking, the higher the resolution, the clearer the image. With the development of science and technology and the improvement of living standards, the demand for high-resolution images in various industries is increasing. However, due to various factors, the images obtained are often low-resolution images. To obtain high-resolution images, we can start from the hardware and software levels. Image super-resolution reconstruction is a method of obtaining high-resolution images through post-processing from the software level. It aims to reconstruct a low-resolution image degraded in quality into a high-resolution image with rich details and clear edges.
[0003] The network structure of image super-resolution reconstruction is generally deep and complex, so the training difficulty is large. In order to solve this problem, researchers based on the particularity of the image restoration task (i.e. the low-resolution image is very similar to the target image, and the same information between the two is more, and the different information is less, that is, the residual is small), the idea of residual learning is introduced. They divide the original end-to-end mapping into linear and nonlinear parts by adding a jump connection. Most of the same information is directly forwarded to the target image by the linear part (jump connection), without spending too much learning resource. Different information (residual information) is learned by the nonlinear part (network backbone), so that the quality of the final reconstructed image is greatly improved.
[0004] Due to the increasing depth of the network structure, the computing resource is still the bottleneck limiting its application and development. In order to further improve the reconstruction effect, researchers further introduced the attention mechanism, such as RCAN, which concentrates the limited computing resources on learning the residual information (such as edges, corners, etc.) that is beneficial to reconstruction, that is, allocates more weight to these information to obtain better feature expression, and then improves the reconstruction effect.
[0005] However, the channel attention module used in the existing method is mostly SE module. Due to the dimension reduction operation and the inability to perform cross-channel interaction, the performance of the module is insufficient. At the same time, due to the non-local similarity of the image itself, it is difficult to capture this non-local dependence only by using channel attention. SUMMARY
[0006] The purpose of this invention is to address the technical problems of existing methods that use attention in the channel dimension to emphasize residual information (such as edges, corners, etc.) that is beneficial to reconstruction, resulting in insufficient performance due to the need for dimensionality reduction operations in the channel attention module, and the difficulty for the channel attention module to capture contextual information in the spatial dimension, which makes the acquired deep feature representation less than ideal, thus limiting the improvement of the reconstructed image effect. Therefore, this invention proposes a super-resolution reconstruction method based on a hybrid attention mechanism.
[0007] A super-resolution reconstruction method based on a hybrid attention mechanism includes the following steps:
[0008] Step 1: Data collection and selection;
[0009] Step 2: Data preprocessing, including low-resolution image generation, dataset augmentation, dataset enhancement, data standardization, and obtaining the training set;
[0010] Step 3: Select the network optimizer and design the hyperparameters;
[0011] Step 4: Design and build the network, including the design and implementation of the shallow feature extraction part, the deep feature extraction part, the feature map amplification part, and the reconstruction part;
[0012] Step 5. Train and optimize the network to obtain the optimal model;
[0013] Step 6: Reconstruct the target image using the optimal model obtained in Step 5.
[0014] Step 1 includes the following sub-steps:
[0015] Step 1-1: Select and collect high-resolution training sets;
[0016] Step 1-2: Select the test set.
[0017] Step 2 includes the following sub-steps:
[0018] Step 2-1: Generate low-resolution images corresponding to several images in the training set;
[0019] Step 2-2: Expand the training set;
[0020] Steps 2-3: Perform dataset augmentation;
[0021] Steps 2-4: Standardize the data.
[0022] In step 3, select the ADAM optimizer to adjust the network parameters.
[0023] In step 4, the established network is a super-resolution reconstruction network based on a hybrid attention mechanism, specifically:
[0024] The network input LR→ shallow feature extraction layer, get feature map F0→ the first hybrid attention residual group, get feature map F1→ several second hybrid attention residual groups→ the third hybrid attention residual group, get feature map F G → the last two-dimensional convolution layer of the deep feature extraction part;
[0025] The shallow feature extraction layer, the last two-dimensional convolution layer of the deep feature extraction part→ the first feature fusion, get feature map F DF → feature map amplification part→ reconstruction part→ reconstructed image HR
[0026] The deep feature extraction part has a total of G hybrid attention residual groups with the same structure, and the G hybrid attention residual groups are sequentially connected from the input end to the output end; each hybrid attention residual group contains B basic residual blocks sequentially connected from the input end to the output end, and a hybrid attention module and a two-dimensional convolution layer constitute.
[0027] The hybrid attention module is located before the last two-dimensional convolution layer of the residual group, not in the residual block. This structure can strengthen important information and capture dependency while speeding up the training and reconstruction of the network model.
[0028] The input of the hybrid attention residual group→ the first basic residual block→ several second basic residual blocks→ the third basic residual block→ the hybrid attention module→ the last two-dimensional convolution layer in the hybrid attention residual group;
[0029] The input of the hybrid attention residual group, the last two-dimensional convolution layer in the hybrid attention residual group→ feature fusion→ the output of the hybrid attention residual group;
[0030] The input of the hybrid attention module→ the channel attention module ECA;
[0031] The input of the hybrid attention module→ the spatial attention module ESA;
[0032] The channel attention module, the spatial attention module→ the channel concatenation Contact→ the channel dimension reduction 1*1 convolution layer→ the output of the hybrid attention module;
[0033] The first hybrid attention residual group, the second hybrid attention residual group, and the third hybrid attention residual group have the same structure. Take the first hybrid attention residual group as an example as follows:
[0034] The feature map F0 input by the first hybrid attention residual group→ the first basic residual block, get feature map F 1,1→ several second basic residual blocks → a third basic residual block, to obtain a feature map F 1,B → a mixed attention module, to obtain a feature map R1 → the last two-dimensional convolution layer in the first mixed attention residual group;
[0035] The feature map F0 input by the first mixed attention residual group, the last two-dimensional convolution layer in the first mixed attention residual group → second feature fusion → the output F1 of the first mixed attention residual group;
[0036] The input F of the mixed attention module 1,B → a channel attention module ECA;
[0037] The input F of the mixed attention module 1,B → a spatial attention module ESA;
[0038] The channel attention module, the spatial attention module → the contact of channel splicing → the 1*1 convolution layer of channel dimension reduction → the output R1 of the mixed attention module;
[0039] The mixed attention module, as an important component of the residual group, integrates the channel attention module ECA which does not need dimension reduction and can consider cross-channel interaction and the lightweight and efficient spatial attention module ESA, so that the constructed network achieves a good balance between reconstruction speed and reconstruction effect.
[0040] In step 4, when using the established super-resolution reconstruction network based on the mixed attention mechanism, the following steps are included:
[0041] Step 4-1: shallow feature extraction is performed;
[0042] Shallow feature extraction is realized by the first two-dimensional convolution layer in the network, which is used to extract shallow features from the input low-resolution image I LR , specifically using formula (1);
[0043] F0=H SFE (I LR ) (1)
[0044] Where H SRE (·) is a two-dimensional convolution operation, and its output F0 is a shallow feature, which will be used as the input of the deep feature extraction part;
[0045] Step 4-2: deep feature extraction is performed;
[0046] The deep feature extraction part is composed of G channel and spatial attention residual groups (CSARGs), a long skip connection and a two-dimensional convolution layer, and is used to further extract deep features from the shallow features;
[0047] According to the shallow features obtained in step 4-1, deep features are further obtained, and the deep features are obtained by formula (2);
[0048] F DF =H DFE (F0) (2)
[0049] Wherein H DFE (·) represents the operation of the deep feature extraction part, and the output F DF is the deep feature.
[0050] Step 4-3: Enlarging the deep features;
[0051] The sub-pixel convolution module is used in the deep feature enlargement part, and this part is used to enlarge the deep features to the target image size. The deep feature enlargement is realized by formula (3)
[0052] F UP =H UP (F DF ) (3)
[0053] Wherein H UP (·) represents the operation of the sub-pixel convolution module, and F UP represents the enlarged features;
[0054] Step 4-4: Reconstructing the enlarged feature map;
[0055] The enlarged feature map is reconstructed, and this part is realized by using a two-dimensional convolution layer, and the reconstructed image is obtained by formula (4);
[0056] I SR =H REC (F UP )=H CSARN (I LR ) (4)
[0057] Wherein I SR is the final reconstructed image, H REC (·) is the operation of the reconstruction part, and H CSARN (·) is the operation of the super-resolution reconstruction network based on the hybrid attention mechanism.
[0058] I LR and I HRrespectively represent the low-resolution image and its corresponding high-resolution image of the network input, I SR represent the super-resolution image output by the network.
[0059] In step 4-2, the deep feature part uses the following sub-steps when in use:
[0060] The deep feature extraction part uses a hybrid attention residual group as a basic unit, which contains G such basic units, a long skip connection and a two-dimensional convolution layer.
[0061] Step 4.2.1: Obtain the output of the gth hybrid attention residual group;
[0062] The G hybrid attention residual groups in the deep feature extraction part are sequentially connected from the input end to the output end, and the output of the gth hybrid attention residual group can be obtained by formula (5);
[0063] F g = H g (F g-1 ) (5)
[0064] Where H g (·) is the operation of the gth hybrid attention residual group, and its input and output are F g-1 and F g respectively;
[0065] Step 4.2.2: Obtain the output of the Gth hybrid attention residual group according to step 4.2.1, and fuse it with the shallow feature F0 to form the final deep feature;
[0066] According to formula (5), formula (2) can be further expressed as formula (6);
[0067] F DF = F0 + W LSC F G = F0 + W LSC H G (F G-1 ) (6)
[0068] Where W LSC is the weight set of the last two-dimensional convolution layer in the deep feature extraction part, and FG is the output of the Gth hybrid attention residual group;
[0069] In step 4.2.1, the hybrid attention residual group uses the following sub-steps when in use:
[0070] Each mixed attention residual group is composed of B basic residual blocks, a short skip connection, a mixed attention module and a two-dimensional convolution layer. Among them, the basic residual block is a module commonly used in super-resolution reconstruction, which is composed of two two-dimensional convolution layers, a ReLU layer and an identity mapping;
[0071] Step 4.2.1.1: Obtain the output of the b-th basic residual block in the g-th mixed attention residual group;
[0072] The B basic residual blocks in each mixed attention residual group are sequentially connected from the input end to the output end. The output of the b-th basic residual block in the g-th mixed attention residual group can be obtained by formula (7)
[0073] F g,b =H g,b (F g,b-1 )=H g,b (H g,b-1 (…H g,1 (F g-1 )…)) (7)
[0074] Where H g,b (·) is the operation of the b-th basic residual block in the g-th mixed attention residual group, and the input and output are F g,b-1 and F g,b respectively;
[0075] Step 4.2.1.2: Obtain the output of the mixed attention module in the g-th mixed attention residual group;
[0076] The mixed attention module is composed of a channel attention module, a spatial attention module, a channel dimension Contact operation and a 1*1 convolution layer. Among them, the channel attention module adopts the ECA module which does not need to perform dimension reduction operation and can consider cross-channel interaction, and the spatial attention module adopts the ESA module which is light and relatively efficient. The mixed attention module first performs information emphasis and dependence capture in the channel dimension and space respectively, then concatenates the output features of the two dimensions, and finally reduces and fuses the output by using the 1*1 convolution layer.
[0077] According to step 4.2.1.1, the output of the B-th residual block in the g-th mixed attention residual group is obtained, and then the output of the mixed attention module is obtained by formula (8)
[0078]
[0079] Where R g is the output of the mixed attention module, W g,1 is the weight set of the 1*1 convolution layer, and [·,·] is the Contact operation in the channel dimension, and respectively represent the operation of the channel attention module and the operation of the spatial attention module of the gth mixed attention residual group, F g,B is the output of the Bth residual block in the gth mixed attention residual group;
[0080] Step 4.2.1.3: Feature fusion is performed between the input of the gth mixed attention residual group and the output of the mixed attention module obtained in 4.2.1.2 to obtain the output of the gth mixed attention residual group;
[0081] Therefore, formula (5) can be further represented as formula (9);
[0082] F g = H g (F g-1 ) = F g-1 + W g,2R g (9)
[0083] where W g,2 is the weight set of the last two-dimensional convolution layer in the gth mixed attention residual group.
[0084] Step 5 includes the following steps:
[0085] Step 5-1: using the training data obtained in step 2 to train the network built in step 4;
[0086] Step 5-2: using L1 loss to optimize the network. For a given training data set wherein and respectively represent the ith (i = 1, …, N) low-resolution image and the corresponding ith high-resolution image in the data set, it contains N LR images and corresponding HR images, and the optimization goal is to make the reconstructed image infinitely close to the original HR image, so that their difference is infinitely small, that is, to minimize the L1 loss, and formula (10) is used to calculate the loss;
[0087]
[0088] wherein Θ represents the parameter set in the network, and the optimization method of the loss function is gradient descent method;
[0089] Step 5-3: during the training process, after training a round, the optimized network is tested once using the test data set to save an optimal model.
[0090] Compared with the prior art, the present application has the following technical effects:
[0091] 1) In the present application, the mixed attention module is constructed by using the channel attention module ECA and the spatial attention module ESA, which can improve the sensitivity of important information in the channel dimension and space, make the network focus on learning more important features, and to some extent, consider the influence of the relationship between channels and the context relationship in space on feature expression, so as to improve the shortcomings of the existing method;
[0092] 2) The mixed attention module is integrated in the residual group instead of the basic residual block as in the existing method, and this structure can speed up the training and reconstruction speed of the network. BRIEF DESCRIPTION OF DRAWINGS
[0093] The present application will be further described below in conjunction with the drawings and examples:
[0094] Figure 1 The flowchart of the present application;
[0095] Figure 2 The structure diagram of the super-resolution reconstruction network based on the mixed attention mechanism in the present application;
[0096] Figure 3 The structure diagram of the mixed attention residual group in the present application. DETAILED DESCRIPTION
[0097] As Figure 1 A super-resolution reconstruction method based on a mixed attention mechanism, which comprises the following steps:
[0098] Step 1: collection and selection of data set;
[0099] Step 2: data preprocessing, including generation of low-resolution images, data set expansion, data set enhancement, data standardization, and obtaining a training set;
[0100] Step 3: selection of network optimizer and design of hyperparameters;
[0101] Step 4: network design and construction, including design and implementation of shallow feature extraction part, deep feature extraction part, feature map enlargement part and reconstruction part;
[0102] Step 5. Network training and optimization to obtain the optimal model;
[0103] Step 6: reconstructing the target image using the optimal model obtained in step 5.
[0104] In step 1, the following sub-steps are included:
[0105] Step 1-1: selection and collection of high-definition high-resolution training set;
[0106] Step 1-2: Selection of the test set is performed.
[0107] In step 2, the following sub-steps are included:
[0108] Step 2-1: Generation of low-resolution images corresponding to a number of images in the training set is performed;
[0109] Step 2-2: Augmentation of the training set is performed;
[0110] Step 2-3: Enhancement of the data set is performed;
[0111] Step 2-4: Standardization of the data is performed.
[0112] In step 3, the ADAM optimizer is selected to adjust the network parameters.
[0113] As shown in FIG. 4, Figure 2
[0114] The established network is a super-resolution reconstruction network based on a hybrid attention mechanism, specifically:
[0115] Network input LR→ shallow feature extraction layer 1, feature map F0 is obtained→ first hybrid attention residual group 2, feature map F1 is obtained→ a number of second hybrid attention residual groups 3→ third hybrid attention residual group 4, feature map F G → last two-dimensional convolution layer 5 of deep feature extraction part;
[0116] Shallow feature extraction layer 1, last two-dimensional convolution layer 5 of deep feature extraction part→ first feature fusion 17, feature map F DF → feature map enlargement part 6→ reconstruction part 7→ reconstructed image HR
[0117] The shallow feature extraction part includes: the first two-dimensional convolution layer 1 in the network.
[0118] The deep feature extraction part includes the first hybrid attention residual group 2, the second hybrid attention residual group 3, and the third hybrid attention residual group 4.
[0119] The feature map enlargement part includes: a module 6 at the end of the network composed of sub-pixel convolution layers.
[0120] The deep feature extraction part has G hybrid attention residual groups with the same structure, and the G hybrid attention residual groups are sequentially connected from the input end to the output end; each hybrid attention residual group includes B basic residual blocks sequentially connected from the input end to the output end, and a hybrid attention module and a two-dimensional convolution layer.
[0121] input of the mixed attention residual group → first basic residual block → several second basic residual blocks → third basic residual block → mixed attention module → last two-dimensional convolution layer in the mixed attention residual group;
[0122] input of the mixed attention residual group, last two-dimensional convolution layer in the mixed attention residual group → feature fusion → output of the mixed attention residual group;
[0123] input of the mixed attention module → channel attention module;
[0124] input of the mixed attention module → spatial attention module;
[0125] channel attention module, spatial attention module → Contact of channel concatenation → 1*1 convolution layer of channel dimension reduction → output of the mixed attention module;
[0126] the first mixed attention residual group 2, the second mixed attention residual group 3 and the third mixed attention residual group 4 have the same structure; the first mixed attention residual group 2 is taken as an example and specifically shown as follows: Figure 3
[0127] feature map F0 input of the first mixed attention residual group → first basic residual block 8, feature map F 1,1 → several second basic residual blocks 9 → third basic residual block 10, feature map F 1,B → mixed attention module 11, feature map R1 → last two-dimensional convolution layer 12 in the first mixed attention residual group;
[0128] feature map F0 input of the first mixed attention residual group, last two-dimensional convolution layer 12 in the first mixed attention residual group → second feature fusion 18 → output F1 of the first mixed attention residual group;
[0129] input F 1,B of the mixed attention module → channel attention module 13;
[0130] input F 1,B of the mixed attention module → spatial attention module 14;
[0131] channel attention module 13, spatial attention module 14 → Contact 15 of channel concatenation → 1*1 convolution layer 16 of channel dimension reduction → output R1 of the mixed attention module;
[0132] Firstly, the application utilizes the relatively advanced channel attention module ECA and the spatial attention module ESA in the constructed hybrid attention module to emphasize important information and capture dependency in the channel dimension and the spatial dimension respectively, then the features output by the two attention modules are channel spliced, and then a 1*1 dimension reduction layer is used for fusion to obtain better feature expression, thereby improving the quality of the reconstructed high-resolution image.
[0133] In step 4, when using the established hybrid attention mechanism based super-resolution reconstruction network, the following steps are included:
[0134] Step 4-1: shallow feature extraction is performed;
[0135] The shallow feature extraction is implemented by the first two-dimensional convolutional layer 1 in the network, which is used to extract shallow features from the input low-resolution image I LR , and specifically uses formula (1);
[0136] F0=H SFE (I LR ) (1)
[0137] Where H SRE (·) is a two-dimensional convolution operation, and its output F0 is the shallow feature, which will be used as the input of the deep feature extraction part;
[0138] Step 4-2: deep feature extraction is performed;
[0139] The deep feature extraction part is composed of G hybrid attention residual groups (channel and spatial attention residual group, CSARG), such as 2, 3, and 4, a long skip connection, and a two-dimensional convolutional layer 5, which is used to further extract deep features from the shallow features;
[0140] According to the shallow features obtained in step 4-1, deep features are further obtained, and the deep features are obtained by formula (2);
[0141] F DF =H DFE (F0) (2)
[0142] Where H DFE (·) is the operation of the deep feature extraction part, and its output F DF is the deep feature.
[0143] Step 4-3: the deep feature is enlarged;
[0144] The sub-pixel convolution module 6 is used in the deep feature amplification part, which is used to amplify the deep features to the target image size. The deep feature amplification is realized by formula (3)
[0145] F UP UP (F DF ) (3)
[0146] where H UP (·) represents the operation of the sub-pixel convolution module, F UP represents the amplified feature;
[0147] Step 4-4: reconstruct the amplified feature map;
[0148] The reconstructed feature map is reconstructed, which is realized by using a two-dimensional convolution layer 7, and the reconstructed image is obtained by formula (4);
[0149] I SR REC (F UP )=H CSARN (I LR ) (4)
[0150] where I SR is the final reconstructed image, H REC (·) is the operation of the reconstruction part, and H CSARN (·) is the operation of the super-resolution reconstruction network based on the hybrid attention mechanism.
[0151] I LR and I HR represent the low-resolution image and its corresponding high-resolution image input into the network, respectively, and I SR represents the super-resolution image output from the network.
[0152] In step 4-2, the deep feature part uses the following sub-steps when used:
[0153] The deep feature extraction part uses a hybrid attention residual group as a basic unit, which contains G such basic units, a long skip connection and a two-dimensional convolution layer.
[0154] Step 4.2.1: Obtain the output of the gth hybrid attention residual group;
[0155] The G hybrid attention residual groups in the deep feature extraction part are sequentially connected from the input end to the output end, and the output of the gth hybrid attention residual group can be obtained by formula (5);
[0156] F g g (Fg-1 ) (5)
[0157] where H g is the output of the gth mixed attention residual group; g-1 and F g ;
[0158] Step 4.2.2: The output of the Gth mixed attention residual group is obtained according to step 4.2.1, and is fused with the shallow feature F0 into the final deep feature;
[0159] According to formula (5), formula (2) can be further expressed as formula (6);
[0160] F DF = F0+ W LSC F G = F0+ W LSC H G (F G-1 ) (6)
[0161] where W LSC is the weight set of the last two-dimensional convolution layer in the deep feature extraction part, and F G is the output of the Gth mixed attention residual group;
[0162] In step 4.2.1, the mixed attention residual group is used in the following substeps:
[0163] Each mixed attention residual group is composed of B basic residual blocks such as 8, 9 and 10, a short skip connection, a mixed attention module such as 11 and a two-dimensional convolution layer such as 12. The basic residual block is a commonly used module in super-resolution reconstruction, which is composed of two two-dimensional convolution layers, a ReLU layer and an identity mapping;
[0164] Step 4.2.1.1: Obtain the output of the bth basic residual block in the gth mixed attention residual group;
[0165] The B basic residual blocks in each mixed attention residual group are sequentially connected from the input end to the output end, and the output of the bth basic residual block in the gth mixed attention residual group can be obtained by formula (7)
[0166] F g,b = H g,b (F g,b-1 ) = H g,b (H g,b-1 (…H g,1 (F g-1 )…)) (7)
[0167] where H g,b(·) is the operation of the gth basic residual block in the gth mixed attention residual group, and the input and output thereof are F g,b-1 and F g,b ;
[0168] Step 4.2.1.2: Obtain the output of the mixed attention module in the gth mixed attention residual group;
[0169] The mixed attention module is composed of a channel attention module, a spatial attention module, a channel dimension Contact operation as shown in 15, and a 1*1 convolution layer as shown in 16, wherein the channel attention adopts the ECA module as shown in 13 without dimension reduction operation and capable of considering cross-channel interaction, and the spatial attention module adopts the ESA module as shown in 14 which is light and relatively efficient. The mixed attention module first performs information emphasis and dependence capture on the channel dimension and the space respectively, then splices the output features of the two dimensions, and finally utilizes the 1*1 convolution layer to reduce dimension and fuse to obtain the output.
[0170] According to step 4.2.1.1, the output of the Bth residual block in the gth mixed attention residual group is obtained, and then the output of the mixed attention module is obtained by formula (8)
[0171]
[0172] wherein R g is the output of the mixed attention module, W g,1 is the weight set of the 1*1 convolution layer, [·,·] is the channel dimension Contact operation, and respectively represent the operation of the channel attention module and the operation of the spatial attention module of the gth mixed attention residual group, F g,B is the output of the Bth residual block in the gth mixed attention residual group;
[0173] Step 4.2.1.3: Perform feature fusion on the input of the gth mixed attention residual group and the output of the mixed attention module obtained in 4.2.1.2 to obtain the output of the gth mixed attention residual group;
[0174] Therefore, formula (5) can be further represented as formula (9);
[0175] F g = H g (F g-1 ) = F g-1 + W g,2 R g (9)
[0176] wherein W g,2is the weight set of the last two-dimensional convolution layer in the gth mixed attention residual group.
[0177] Step 5 comprises the following steps:
[0178] Step 5-1: using the training data obtained in step 2, the network built in step 4 is trained;
[0179] Step 5-2: the network is optimized using L1 loss. For a given training data set wherein and respectively represent the i(i=1,…,N)th low-resolution image and the corresponding i(th high-resolution image in the data set, it contains N LR images and corresponding HR images, the optimization goal is to make the reconstructed image infinitely close to the original HR image, and the difference between them is infinitely small, that is, to minimize the L1 loss, the loss is calculated using formula (10);
[0180]
[0181] wherein Θ represents the parameter set in the network, and the optimization method of the loss function is gradient descent method;
[0182] Step 5-3: during the training process, after training a round, the optimized network is tested once using the test data set to save an optimal model.
[0183] Embodiment: in this example, the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) commonly used in super-resolution reconstruction are used to evaluate the performance of the model. Table 1 is the comparison of the reconstruction results of the classic method RCAN, wherein the total training iteration number of the two models is 100 times.
[0184] Table 1 comparison of experimental results with RCAN
[0185]
[0186] As can be seen from Table 1, after training the same number of iterations, on the test data set Set5, the reconstruction result of the method proposed in the application is higher than that of RCAN when the magnification is 2.
[0187] In summary, the super-resolution reconstruction method based on the mixed attention mechanism proposed in the application can distinguish the importance of information conducive to detail recovery, and can improve the sensitivity to high-contribution features, so that the network can focus on learning more important features, while combining the context information in the spatial dimension to generate better feature expression, thereby improving the quality of the reconstructed image;
[0188] In addition, the ECA module and the ESA module configured in the mixed attention module are relatively light modules, and the mixed attention module is integrated in the residual group instead of the residual block, so that the constructed network achieves a good balance between reconstruction speed and reconstruction effect.
Claims
1. A super-resolution reconstruction method based on a hybrid attention mechanism, characterized in that, It comprises the following steps: Step 1: data collection and selection; Step 2: data preprocessing, including low-resolution image generation, data set expansion, data set enhancement, data standardization, and obtaining a training set; Step 3: selection of network optimizer and design of hyperparameters; Step 4: network design and construction, including shallow feature extraction, deep feature extraction, and feature map amplification; Step 5: network training and optimization to obtain an optimal model; Step 6: reconstructing the target image using the optimal model obtained in step 5; In step 4, the established network is a hybrid attention mechanism-based super-resolution reconstruction network, specifically: Network input LR→ shallow feature extraction layer, get feature map → first mixed attention residual group, get feature map → several second mixed attention residual groups → third mixed attention residual group, get feature map → the last two-dimensional convolution layer of the deep feature extraction part; Shallow feature extraction layer, the last two-dimensional convolution layer of deep feature extraction part → first feature fusion, get feature map → feature map enlargement part → reconstruction part → reconstructed image; The hybrid attention residual group is as follows: feature maps of the first mixed attention residual group input → first basic residual block, resulting in feature maps → several second basic residual blocks → third basic residual block, resulting in feature maps → mixed attention module, resulting in feature maps → last two-dimensional convolution layer in the first mixed attention residual group; the feature map input of the first mixed attention residual group the last two-dimensional convolution layer in the first mixed attention residual group → second feature fusion → the output of the first mixed attention residual group ; The hybrid attention module is as follows: Input to the mixed attention module → Channel attention module ECA; Input to the mixed attention module → spatial attention module ESA; Channel attention module ECA, spatial attention module ESA Contact of channel concatenation 1 of channel dimension reduction 1 Convolutional layer Output of hybrid attention module ; In step 4, when using the established network, the following steps are included: Step 4-1: shallow feature extraction; The shallow feature extraction is realized by the first two-dimensional convolution layer in the network, and is used to extract shallow features from the input low-resolution image , specifically using equation (1). ； wherein is a two-dimensional convolution operation, whose output is a shallow feature, which will be the input of the deep feature extraction part; Step 4-2: deep feature extraction; The deep feature extraction part includes G hybrid attention residual groups for further extracting deep features from shallow features; According to the shallow features obtained in step 4-1, deep features are further obtained, and the deep features are obtained by formula (2); ； wherein the operation of the deep feature extraction section, whose output is the deep feature; Step 4-3: amplifying the deep features; The deep feature amplification part uses a sub-pixel convolution module, which is used to amplify the deep features to the size of the target image; the deep feature amplification is realized by formula (3): ； wherein denotes the operation of a sub-pixel convolution module, denotes the amplified features; Step 4-4: reconstructing the amplified feature map; The amplified feature map is reconstructed, which is realized using a two-dimensional convolution layer, and the reconstructed image is obtained by formula (4); ； wherein, is the final reconstructed image, is the operation of reconstructing the part, is the operation of the super-resolution reconstruction network based on the hybrid attention mechanism; a low resolution image representing a network input, a super resolution image representing a network output; In step 4-2, the deep feature extraction part uses the following sub-steps when used: The deep feature extraction part uses a hybrid attention residual group as a basic unit, which contains G such basic units, a long skip connection, and a two-dimensional convolution layer; Step 4.2.1: obtaining the output of the gth hybrid attention residual group; The G hybrid attention residual groups in the deep feature extraction part are sequentially connected from the input end to the output end, and the output of the gth hybrid attention residual group can be obtained by formula (5); ； wherein is the operation of the gth mixed attention residual group, whose input and output are and ; Step 4.2.2: The output of the Gth mixed attention residual group is computed according to Step 4.2.1 and fused with the shallow feature into the final deep feature. Step 4.2.3: The final deep feature is computed according to Step 4.2.2 and Step 4.2.
1. According to formula (5), formula (2) can be further expressed as formula (6); ； wherein is a weight set of the last two-dimensional convolution layer in the deep feature extraction part, is the output of the Gth mixed attention residual group; In step 4.2.1, the hybrid attention residual group uses the following sub-steps when used: Each hybrid attention residual group consists of B basic residual blocks, a short skip connection, a hybrid attention module, and a two-dimensional convolution layer, where the basic residual block is a commonly used module in super-resolution reconstruction, consisting of two two-dimensional convolution layers, a ReLU layer, and an identity mapping; Step 4.2.1.1: obtaining the output of the bth basic residual block in the gth hybrid attention residual group; The B basic residual blocks in each hybrid attention residual group are sequentially connected from the input end to the output end, and the output of the bth basic residual block in the gth hybrid attention residual group can be obtained by formula (7): ； wherein is the operation of the g-th basic residual block in the g-th mixed attention residual group, whose input and output are respectively and ; Step 4.2.1.2: obtaining the output of the hybrid attention module in the gth hybrid attention residual group; The mixed attention module is composed of a channel attention module, a spatial attention module, a channel dimension Contact operation and a 1 1convolutional layer; wherein the channel attention module adopts the ECA module without dimension reduction operation and capable of considering cross-channel interaction, and the spatial attention module adopts the ESA module which is light and relatively efficient; the mixed attention module first respectively performs information emphasis and dependence capture on the channel dimension and the space, then splices the output features of the two dimensions, and finally utilizes the 1 1convolutional layer to reduce dimension and fuse to obtain the output; The output of the Bth residual block in the gth mixed attention residual group is obtained according to step 4.2.1.1.1, and then the output of the mixed attention module is obtained by formula (8): ； wherein is the output of the mixed attention module, is 1 1 a set of weights of a convolutional layer, is a contact operation on the channel dimension, and denote the operation of the channel attention module ECA and the operation of the spatial attention module ESA of the g-th mixed attention residual group, respectively, is the output of the B-th residual block in the g-th mixed attention residual group; Step 4.2.1.3: The input of the gth mixed attention residual group is fused with the output of the mixed attention module obtained in step 4.2.1.2 to obtain the output of the gth mixed attention residual group; Therefore, formula (5) can be further expressed as formula (9); ； wherein is the weight set of the last two-dimensional convolution layer in the gth mixed attention residual group.
2. The method of claim 1, wherein, In step 1, the following sub-steps are included: Step 1-1: Selection and collection of high-definition high-resolution training set are performed; Step 1-2: Selection of test set is performed.
3. The method of claim 1, wherein, In step 2, the following sub-steps are included: Step 2-1: Generation of low-resolution images corresponding to a plurality of images in the training set is performed; Step 2-2: Augmentation of the training set is performed; Step 2-3: Enhancement of the data set is performed; Step 2-4: Standardization of the data is performed.
4. The method according to claim 1 or 2 or 3, characterized in that, Step 5 includes the following steps: Step 5-1: The network built in step 4 is trained using the training data obtained in step 2; Step 5-2: Use The loss is used to optimize the network; for a given training dataset ,in and They represent the first and second elements in the dataset, respectively. Zhang low-resolution image and the corresponding first A high-resolution image, which contains N The optimization goal is to make the reconstructed image infinitely close to the original HR image, minimizing their difference, and thus... To minimize the loss, use equation (10) to calculate the loss; ； wherein represents the parameter set in the network, and the optimization method of the loss function is gradient descent method, ; Step 5-3: During the training process, after each round of training, the optimized network is tested once using the test data set to save an optimal model.