Image super-resolution reconstruction method and device based on high-frequency feature enhancement

Abstract

The present invention discloses an image super-resolution reconstruction method and device based on high-frequency feature enhancement, and relates to the technical field of image super-resolution reconstruction. In the training stage of the image super-resolution reconstruction model of high-frequency feature enhancement, the present invention adopts a multi-scale sampling network to extract high-frequency features, and introduces a feature mask module in the multi-scale sampling network to improve the accuracy of feature fusion. At the same time, a lightweight high-frequency enhancement attention mechanism is introduced and integrated with the multi-scale sampling network to construct a high-frequency feature self-adjustment module. This process effectively integrates high-frequency features of different scales and levels into the network through learnable parameters, so that the feature extraction capability is greatly improved. Then, a mapping relationship between high-frequency features of different layers in multiple high-frequency feature self-adjustment modules is established through a mask matrix produced by a feature adjustment layer, so that the interaction between features is greatly enhanced, thereby greatly enhancing the utilization rate of features, and thus the quality of the reconstructed image is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution reconstruction, and in particular to an image super-resolution reconstruction method and device based on high-frequency feature enhancement. Background Art

[0002] Image super-resolution technology plays an important role in the fields of medical imaging, remote sensing monitoring and computational photography. It aims to upgrade low-resolution images to high resolution to enhance the clarity and details of the image. At present, the main super-resolution reconstruction methods include interpolation, reconstruction and learning. The interpolation method generally uses bicubic interpolation and geometric feature-based interpolation. Although it is simple, it is easy to produce image oscillation and artifacts. In contrast, reconstruction methods such as maximum a posteriori estimation and iterative back projection have improved the reconstruction effect, but the calculation effect is low and is greatly affected by the magnification factor. Learning methods, such as domain embedding and sparse representation, achieve HR image reconstruction by learning the mapping function of HR image and LR image sample pairs or using the internal similarity of the image. In recent years, with the rapid development of deep learning technology, convolutional neural networks (CNN) have been widely used in super-resolution tasks, and a series of efficient image super-resolution networks, such as SRCNN, EDSR, RCAN and MCSR, have been proposed, which provide new ideas and methods for the further development of super-resolution reconstruction technology.

[0003] Current super-resolution networks often focus on image reconstruction performance while ignoring the complexity and practical feasibility of model training; the computational cost of super-resolution models is closely related to the number and structure of models; in order to reduce the number of network parameters, lightweight image super-resolution networks usually use simple feature extraction and fusion methods to complete image reconstruction. However, this simple fusion method may limit the feature extraction capability because it uses fixed and inefficient convolution kernels; another common design method is to use series or dense connections, such as residual dense networks (RDN). The series connection method may lead to insufficient interaction between features, while the dense connection method may reduce the utilization of features, thereby affecting network performance.

[0004] Therefore, the conventional simple feature extraction and fusion adopted by the current end-to-end lightweight image super-resolution network, as well as the use of series and dense jumper connections, have limited feature extraction capabilities and insufficient interaction between features, which greatly reduces the utilization of features, resulting in poor quality of reconstructed images and limiting its use in practical applications. Summary of the invention

[0005] The embodiments of the present invention provide an image super-resolution reconstruction method and device based on high-frequency feature enhancement, which can solve the problems in the prior art that the feature extraction capability is limited and the interaction between features is insufficient, which greatly reduces the utilization rate of the features and makes the quality of the reconstructed image poor.

[0006] The embodiment of the present invention provides an image super-resolution reconstruction method based on high-frequency feature enhancement, comprising the following steps:

[0007] Construct an image super-resolution reconstruction model framework with high-frequency feature enhancement, which includes a shallow feature extraction module and a reconstruction module;

[0008] In the model framework, upsampling layers and downsampling layers of different scales are used to construct a multi-scale sampling network, two convolutional layers and a depth-separable convolution cascade are used to construct a feature mask module FMB, and downsampling layers and upsampling layers of different scales are used to construct a lightweight high-frequency enhanced attention mechanism HFG-ESA. The feature mask module FMB is used to train the high-frequency feature refinement ability of the multi-scale sampling network to obtain the trained multi-scale sampling network, and the trained multi-scale sampling network is fused with the lightweight high-frequency enhanced attention mechanism to construct a high-frequency feature self-adjustment module.

[0009] Six high-frequency feature self-adjustment modules are used and divided into two groups, a mask matrix is ​​generated using a high-frequency feature adjustment layer, and a mapping between the first three high-frequency feature self-adjustment modules and the last three high-frequency feature self-adjustment modules is constructed using the mask matrix to obtain a high-frequency feature mutual adjustment module, and multiple high-frequency feature mutual adjustment modules are fused into a deep feature extraction module within the reconstruction model framework to complete the training of the model, and the deep feature extraction module, the shallow feature extraction module and the reconstruction module are fused within the reconstruction model to obtain a trained high-frequency feature enhanced image super-resolution reconstruction model;

[0010] The low-resolution image to be reconstructed is input into the trained high-frequency feature enhanced image super-resolution reconstruction model, and the image features are extracted through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; the shallow feature map is extracted through multiple high-frequency feature mutual adjustment modules that map to each other in the deep feature extraction module to obtain a deep feature map; the features in the deep feature map are extracted through the convolution layer and the sub-pixel convolution layer included in the reconstruction module, the feature elements in the deep feature map and the shallow feature map are added, and pixel reconstruction is performed to obtain a reconstructed high-resolution image.

[0011] Preferably, the high-frequency feature enhanced image super-resolution reconstruction model framework comprises:

[0012] The high-frequency feature enhanced image super-resolution reconstruction model framework consists of three parts: image shallow feature extraction module, image deep feature extraction module and image reconstruction module, and the formulas used are:

[0013] F s =W conv3 (I LR )

[0014] F D =W HFMM (W HFMM (W HFMM (F S )))

[0015] I SR =W conv3 (W pixelshufle (W D ))+Bic(F S )

[0016] Where: I LR represents a low-resolution image; W conv3 (·) indicates a convolution with a kernel size of 3*3; F s Represents shallow features; W HFMM (·) represents the nonlinear mapping learned by the high-frequency feature mutual adjustment module; F D Represents the shallow feature F s The deep features obtained by four HFMM modules; W pixelshufle (·) indicates sub-pixel convolution layer; Bic(·) indicates bicubic interpolation upsampling; I SR represents the final reconstructed super-resolution image; W D Represents the deep features to be processed.

[0017] Preferably, the multi-scale sampling network comprises:

[0018] The multi-scale sampling network extracts multi-scale high-frequency features by downsampling the input features by eight times, four times, and two times, respectively. The formula used is:

[0019]

[0020] Where: F1 is the upper input of the high frequency self-modulation module; They are eightfold, fourfold, and twofold downsampling after maximum pooling, respectively; Bicubic interpolation upsampling is eightfold, fourfold, and twofold respectively; F x8 、F x4 、F x2 These are the output features of F1 after eight times, four times, and two times of upsampling.

[0021] Preferably, the feature mask module comprises:

[0022] The feature mask module FMB consists of two 1*1 convolutions and one 3*3 depth-separable convolution, and the formula used is:

[0023]

[0024] in: They are the double, quadruple, and eightfold high-frequency features extracted by the multi-scale sampling network; W FMB (·) is the transfer function of the feature mask module; α, β, and γ are the feature masks of each branch respectively; are the output characteristics of the three branches respectively; F hf is the sum of the high-frequency information output by the three branches; δ(·) is the Sigmoid activation function; is the element-wise multiplication operation; W conv1 (*) indicates 1*1 convolution; W dwconv1 (·) represents a 1*1 depthwise separable convolution; F up is the final high-frequency output feature; F1 is the upper input of the feature mask module.

[0025] Preferably, the formula used by the high-frequency feature self-adjustment module is:

[0026]

[0027] F up =W hf (F1)

[0028] F down =σ(W conv3 (σ(W conv3 (F2)))+W conv3 (F2)

[0029]

[0030] in: is the input of the module; split(·) is the channel split operation; F1 and F2 are the upper and lower branch inputs of the channel split respectively; W hf (·) is a multi-scale sampling network; F up is the upper branch output; σ(·) is the activation function GELU; F down is the output of the lower branch; concat(·) is the feature concatenation F up Operation; W HFG-ESA (·) High-frequency feature guided enhanced spatial attention (HFG-ESA); is the output of the module; Wconv3 (·) indicates a convolution with a kernel size of 3*3.

[0031] Preferably, the high-frequency feature mutual adjustment module includes:

[0032] The high-frequency feature mutual adjustment module includes six high-frequency feature self-adjustment modules;

[0033] The high-frequency feature mutual adjustment module generates a mask matrix through the feature adjustment layer, and uses the mask matrix to adjust the output of the last three high-frequency feature self-adjustment modules to achieve effective mapping of the outputs of the first three high-frequency feature self-adjustment modules and establish the relationship between high-frequency features in different periods. The formula used is:

[0034]

[0035] Where: W conv1 ( ＊ ) represents 1*1 convolution; is the output feature of the i-th high-frequency feature self-adjustment module in the high-frequency feature mutual adjustment module; is the output feature of the i+3th high-frequency feature self-adjustment module in the high-frequency feature mutual adjustment module; F i,i+3 The output feature of the i-th high-frequency feature self-adjustment module is used as the previous feature, and the output feature of the i+3-th high-frequency feature self-adjustment module is used as the current feature; Mod( ＊ ) is the remainder function.

[0036] Preferably, the steps of extracting the high-frequency feature enhanced image super-resolution reconstruction model are:

[0037] The image shallow feature extraction module consists of a convolution with a convolution kernel size of 3*3. LR First, a 3*3 convolution is performed to extract shallow features to obtain shallow features F S ;

[0038] The deep feature extraction module of the image is composed of multiple high-frequency feature mutual adjustment modules, each of which is composed of six high-frequency feature self-adjustment modules. S The deep feature F is obtained by multiple high-frequency feature mutual adjustment modules D ;

[0039] The image reconstruction module consists of a sub-pixel convolution layer and a 3*3 convolution layer. D After the sub-pixel convolution layer and 3*3 convolution, the shallow feature F is upsampled by bicubic interpolation. S Add together to get the reconstructed super-resolution image I SR .

[0040] The embodiment of the present invention further provides an image super-resolution reconstruction device based on high-frequency feature enhancement, comprising:

[0041] A model building module is used to build an image super-resolution reconstruction model framework with high-frequency feature enhancement, which includes a shallow feature extraction module and a reconstruction module;

[0042] The model training module is used to construct a multi-scale sampling network using upsampling layers and downsampling layers of different scales in the model framework, construct a feature mask module FMB using two convolutional layers and a depth-separable convolution cascade, and construct a lightweight high-frequency enhanced attention mechanism HFG-ESA using downsampling layers and upsampling layers of different scales; the feature mask module FMB is used to train the high-frequency feature refinement capability of the multi-scale sampling network to obtain the trained multi-scale sampling network, and the trained multi-scale sampling network is integrated with the lightweight high-frequency enhanced attention mechanism to construct a high-frequency feature self-adjustment module;

[0043] Six high-frequency feature self-adjustment modules are used and divided into two groups, a mask matrix is ​​generated using a high-frequency feature adjustment layer, and a mapping between the first three high-frequency feature self-adjustment modules and the last three high-frequency feature self-adjustment modules is constructed using the mask matrix to obtain a high-frequency feature mutual adjustment module, and multiple high-frequency feature mutual adjustment modules are fused into a deep feature extraction module within the reconstruction model framework to complete the training of the model, and the deep feature extraction module, the shallow feature extraction module and the reconstruction module are fused within the reconstruction model to obtain a trained high-frequency feature enhanced image super-resolution reconstruction model;

[0044] The image reconstruction module inputs the low-resolution image to be reconstructed into the trained high-frequency feature enhanced image super-resolution reconstruction model, extracts features of the image through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; extracts features of the shallow feature map through multiple high-frequency feature mutual adjustment modules mapped to each other in the deep feature extraction module to obtain a deep feature map; extracts features in the deep feature map through the convolution layer and sub-pixel convolution layer contained in the reconstruction module, adds the feature elements in the deep feature map and the shallow feature map, and performs pixel reconstruction to obtain a reconstructed high-resolution image.

[0045] The embodiment of the present invention provides an image super-resolution reconstruction method and device based on high-frequency feature enhancement. Compared with the prior art, the advantages thereof are as follows:

[0046] In the training stage of the image super-resolution reconstruction model with high-frequency feature enhancement, the present invention adopts a multi-scale sampling network to extract high-frequency features, and introduces a feature mask module in the multi-scale sampling network to improve the accuracy of feature fusion. At the same time, a lightweight high-frequency enhancement attention mechanism is introduced and integrated with the multi-scale sampling network to construct a high-frequency feature self-adjustment module. This process effectively integrates high-frequency features of different scales and levels into the network through learnable parameters, which greatly improves the feature extraction capability. Then, a mapping relationship between high-frequency features of different layers in multiple high-frequency feature self-adjustment modules is established through the mask matrix produced by the feature adjustment layer, which greatly enhances the interaction between features, thereby greatly enhancing the utilization rate of features, and thus greatly improving the quality of the reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of the main flow of an image super-resolution reconstruction method and device based on high-frequency feature enhancement provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of the algorithm structure of an image super-resolution reconstruction method and device based on high-frequency feature enhancement provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of visual comparison of an image reconstructed by an algorithm of an image super-resolution reconstruction method and device based on high-frequency feature enhancement provided by an embodiment of the present invention and four-fold reconstruction of img025 in Urban100 by other algorithms;

[0050] Figure 4 A schematic diagram of visual comparison of an image reconstructed by an algorithm of the present invention and an image super-resolution reconstruction method and device based on high-frequency feature enhancement provided by an embodiment of the present invention and four-fold reconstruction of img073 in Urban100 by other algorithms;

[0051] Figure 5 A schematic diagram of high-frequency information of a feature graph of a high-frequency feature self-adjustment module effectiveness analysis result of an image super-resolution reconstruction method and device based on high-frequency feature enhancement provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of similar pixel point analysis of a feature graph of a high-frequency feature self-adjustment module effectiveness analysis result of an image super-resolution reconstruction method and device based on high-frequency feature enhancement provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0054] See also Figure 1 to Figure 6 The embodiment of the present invention provides an image super-resolution reconstruction method based on high-frequency feature enhancement, comprising the following steps:

[0055] Step 1: Design an image super-resolution reconstruction model framework based on high-frequency feature enhancement.

[0056] Step 2: Propose a multi-scale sampling network to extract high-frequency features through sampling, and introduce a feature mask module to further refine these extracted high-frequency features.

[0057] Step 3: Propose a lightweight HFG-ESA, which further increases the attention to the high-frequency information of the image while maintaining the focus on spatial features, thereby improving the network's ability to represent high-frequency information.

[0058] Step 4: Construct a high-frequency feature self-adjustment module to achieve feature fusion with learnable parameters, which greatly increases the accuracy of high-frequency features extracted by the network.

[0059] Step 5: Design a high-frequency feature mutual adjustment module and use the feature adjustment layer to efficiently establish a nonlinear mapping relationship between the high-frequency features of images at different periods.

[0060] In step 1, the image super-resolution reconstruction model framework based on high-frequency feature enhancement is designed:

[0061] This framework consists of three parts: image shallow feature extraction, image deep feature extraction and image reconstruction; the formula is as follows:

[0062] F s =W conv3 (I LR )

[0063] F D =W HFMM (W HFMM (W HFMM (F S )))

[0064] I SR =W conv3 (W pixelshufle (W D))+Bic(F S )

[0065] Where: I LR represents a low-resolution image; W conv3 (·) indicates a convolution with a kernel size of 3*3; F s Represents shallow features; W HFMM (·) represents the nonlinear mapping learned by the high-frequency feature mutual adjustment module; F D Represents the shallow feature F s The deep features obtained by four HFMM modules; W pixelshufle (·) indicates sub-pixel convolution layer; Bic(·) indicates bicubic interpolation upsampling; I SR represents the final reconstructed super-resolution image; W D Represents the deep features to be processed.

[0066] In step 2, a multi-scale sampling network is proposed and a feature mask module is introduced, including:

[0067] Most current super-resolution models obtain high-frequency features by increasing the number of feature extraction modules, but there are problems such as fixed extracted features and low precision. The present invention proposes a multi-scale sampling network to sample and obtain high-frequency information, significantly reducing the parameters of the stacked convolutional layers.

[0068] The multi-scale sampling network proposed in the present invention extracts multi-scale high-frequency features by downsampling the input features by eight times, four times, and two times, respectively; the formula is as follows:

[0069]

[0070] Where: F1 is the upper input of the high frequency self-modulation module; They are eightfold, fourfold, and twofold downsampling after maximum pooling, respectively; The bicubic interpolation is upsampled eight times, four times, and two times respectively; the final F x8 、F x4 、F x2 These are the output features of F1 after eight times, four times, and two times of upsampling.

[0071] Most existing super-resolution models extract high-frequency information by stacking a large number of convolutional extraction layers, which will lead to a large amount of redundant calculations. To address this problem, the present invention introduces a feature mask block to improve the network's ability to represent cross-scale high-frequency information, and further improve the accuracy of information fusion without increasing redundant calculations. The formula is as follows:

[0072]

[0073]

[0074] in: They are the double, quadruple, and eightfold high-frequency features extracted by the multi-scale sampling network. The feature mask module FMB consists of two 1*1 convolutions and one deep classifiable convolution. FMB (·) is the transfer function of the feature mask module; α, β, and γ are the feature masks of each branch respectively; are the output characteristics of the three branches respectively; F hf is the sum of the high-frequency information output by the three branches; δ(·) is the Sigmoid activation function; is the element-wise multiplication operation; W conv1 (*) indicates 1*1 convolution; W dwconv1 (·) represents a 1*1 depthwise separable convolution; F up is the final high-frequency output feature; F1 is the upper input of the feature mask module.

[0075] In step three, a lightweight high-frequency enhanced attention mechanism (HFG-ESA) is proposed to enhance the network's ability to represent high-frequency features. The HFG-ESA of the present invention uses the concept of a multi-scale sampling network to first downsample the input features and then upsample them to the original size. Finally, high-frequency features are further extracted by element-wise subtraction to enhance the input of the convolution group.

[0076] In step 4, a high-frequency feature self-modulation module (HFSM) is constructed to enhance the utilization of high-frequency features; the formula is as follows:

[0077]

[0078] F up =W hf (F1)

[0079] F down =σ(W conv3 (σ(W conv3 (F2)))+W conv3 (F2)

[0080]

[0081] in: is the input of the HFSM module; split(·) is the channel split operation; F1 and F2 are the upper and lower branch inputs of the channel split respectively; W hf (·) is a multi-scale sampling network; F up is the upper branch output; σ(·) is the activation function GELU; F downis the output of the lower branch; concat(·) is the feature concatenation F up Operation; W HFG-ESA (·) High-frequency feature guided enhanced spatial attention (HFG-ESA); is the output of the HFSM module; W conv3 (·) indicates a convolution with a kernel size of 3*3; the value of i ranges from 2 to 6.

[0082] Finally, in step five, in order to establish an effective mapping relationship between high-frequency feature vectors of different layers, the present invention proposes a high-frequency feature mutual adjustment (HFMM) module; the implementation method of the HFMM module is: a mask matrix is ​​generated by adjusting the layer to adjust the output of the subsequent three HFSM modules, so as to achieve effective mapping of the outputs of the three previous high-frequency feature self-adjustment (HFSM) modules, thereby establishing the relationship between high-frequency features in different periods; there is the following formula:

[0083]

[0084] Where: W conv1 ( ＊ ) represents 1*1 convolution; is the output feature of the i-th HFSM module in the current HFMM module; is the output feature of the i+3th high-frequency feature self-adjustment module in the high-frequency feature mutual adjustment module; F i,i+3 The output features of the i-th HFSM module are used as the previous features, the output features of the i+3-th HFSM module are used as the current features, and the output features are obtained after the adjustment layer, where the value of i ranges from 1 to 3; Mod( ＊ ) is the remainder function.

[0085] In order to solve the problems of insufficient mapping relationship and information redundancy that may be caused by jumper connection, the present invention introduces a HFMM module and uses a feature adjustment layer to generate a mask matrix to characterize the mapping relationship between high-frequency features of different layers.

[0086] The algorithm framework of the embodiment of the present invention is as follows Figure 2 As shown in the figure, the algorithm consists of three modules, namely, image shallow feature extraction, image deep feature extraction and image reconstruction.

[0087] The shallow feature extraction module consists of a convolution with a convolution kernel size of 3*3. LR First, a 3*3 convolution is performed to extract shallow features to obtain shallow features F S .

[0088] The deep feature extraction module is composed of multiple high-frequency feature mutual adjustment modules, each of which is composed of six high-frequency feature self-adjustment modules. S The deep feature F is obtained by multiple high-frequency feature mutual adjustment modules D .

[0089] The image reconstruction module consists of a sub-pixel convolution layer and a 3*3 convolution layer. D After the sub-pixel convolution layer and 3*3 convolution, the shallow feature F is upsampled by bicubic interpolation. S Add together to get the reconstructed super-resolution image I SR .

[0090] While keeping the model lightweight, the present invention uses a multi-scale sampling network to extract more accurate high-frequency features. At the same time, a high-frequency feature mutual adjustment module is designed to enhance the model's ability to characterize high-frequency features. First, an image super-resolution reconstruction model framework based on high-frequency feature enhancement is constructed, including image shallow feature extraction, image deep feature extraction, and image reconstruction. In the deep feature extraction module, a multi-scale sampling network is used to extract high-frequency features, and a feature mask block is introduced to improve the accuracy of feature fusion. Subsequently, an enhanced spatial attention mechanism based on high-frequency features (HFG-ESA) is introduced to further enhance the network's attention to high-frequency features. Then, the multi-scale sampling network and HFG-ESA are combined to construct a high-frequency feature self-adjustment module, which improves the high-frequency feature extraction capability. In order to characterize the mapping relationship between high-frequency features of different layers, a high-frequency feature mutual adjustment module is designed to reduce model redundant calculations and improve the utilization of early high-frequency features. In summary, the present invention solves the problem of performance degradation of lightweight image super-resolution networks when pursuing low parameter amounts.

[0091] The technical effects of the present invention are described in detail below in combination with performance testing and experimental analysis; the increase in scale aggravates the loss of high-frequency features of the image, and the high-frequency feature enhancement module of the present invention can effectively restore them; quantitative analysis shows that the model of the present invention is significantly superior to the advanced lightweight image super-resolution reconstruction model at a 4-fold scaling factor, especially in restoring high-frequency features; the comparison results are shown in Table 1.

[0092] Table 1 Comparison of indicators under 4x scaling factor

[0093]

[0094]

[0095] From Table 1, it can be found that the model proposed in the present invention has significantly higher PSNR and SSIM indicators than other models while keeping the number of parameters low under a 4-fold scaling factor.

[0096] In order to further demonstrate the superiority of the network model proposed in this paper, a 4-fold image reconstruction experiment was conducted on the Urban100 dataset with complex textures. The images reconstructed using this algorithm were compared with the results of other algorithms. The comparison results are shown in Figure 2. Figure 3 and Figure 4 As shown; It can be clearly seen from these image results that compared with other models, the algorithm model proposed in the present invention can reconstruct images more realistically; Figure 3 It can be seen that the lines in the original super-resolution image are straight, and the images reconstructed by other models have visual problems such as blurred details and severe jaggedness, while the image reconstructed by the algorithm proposed in the present invention is basically consistent with the original image; this shows that the algorithm model proposed in the present invention has obvious superiority over other algorithm models in image super-resolution reconstruction.

[0097] In order to further verify the effectiveness of the high-frequency feature self-adjustment proposed in the present invention, a visual comparative analysis of the feature maps was carried out; the original input was processed with sampling factors of 2, 4 and 8 to obtain the corresponding high-frequency information feature maps; the observation results showed that these feature maps were basically similar to the feature maps of high-resolution images, indicating that the high-frequency features output by the model proposed in the present invention had a better effect; in order to further quantify the differences, the visualized feature maps were compared pixel by pixel; the results showed that the high-frequency features output by the high-frequency feature self-adjustment module proposed in the present invention had a feature map similarity ratio of up to 42.52%, which was more than three times that of the original input features; this shows that the high-frequency feature self-adjustment module can enhance the extraction of high-frequency features and has a good effect on reconstructing super-resolution images.

[0098] In order to further demonstrate the effectiveness of the high-frequency feature mutual adjustment proposed in the present invention, the present invention conducted an ablation experiment and designed four network models for comparison; they are respectively a series high-frequency feature self-adjustment module structure, a densely connected high-frequency feature self-adjustment module structure, a jump-connected high-frequency feature self-adjustment module structure and a high-frequency feature mutual adjustment structure; these four modules are deep feature extraction modules, named Reuse_none, Reuse_dense, Reuse_skip, and Reuse_mod respectively; after 300 rounds of training, a test with a scaling factor of 4 was performed on the Set14 dataset; the test results are shown in Table 2, where × indicates that this operation is not adopted, and √ indicates that this operation is adopted.

[0099] Table 2 Analysis of the effectiveness of feature mutual adjustment

[0100]

[0101] It can be concluded from the observations in Table 2 that, compared with directly using jumper connections and dense connections, the present invention establishes a mapping relationship between high-frequency features of different layers through adjustment operations, and achieves the best reconstruction effect with minimal increase in the number of parameters and calculations.

[0102] The embodiment of the present invention further provides an image super-resolution reconstruction device based on high-frequency feature enhancement, comprising:

[0103] The model building module is used to build an image super-resolution reconstruction model framework with high-frequency feature enhancement, which includes a shallow feature extraction module and a reconstruction module.

[0104] The model training module is used to construct a multi-scale sampling network using upsampling layers and downsampling layers of different scales in the model framework, construct a feature mask module FMB using two convolutional layers and a depth-wise separable convolution cascade, and construct a lightweight high-frequency enhanced attention mechanism HFG-ESA using downsampling layers and upsampling layers of different scales; the feature mask module FMB is used to train the high-frequency feature refinement capability of the multi-scale sampling network to obtain the trained multi-scale sampling network, and the trained multi-scale sampling network is integrated with the lightweight high-frequency enhanced attention mechanism to construct a high-frequency feature self-adjustment module.

[0105] Six high-frequency feature self-adjustment modules are adopted and divided into two groups equally. The high-frequency feature adjustment layer is used to generate a mask matrix. The mask matrix is ​​used to construct the mapping between the first three high-frequency feature self-adjustment modules and the last three high-frequency feature self-adjustment modules to obtain a high-frequency feature mutual adjustment module. Multiple high-frequency feature mutual adjustment modules are fused into a deep feature extraction module within the reconstruction model framework to complete the training of the model and the deep feature extraction module, the shallow feature extraction module and the reconstruction module are fused within the reconstruction model to obtain the trained high-frequency feature enhanced image super-resolution reconstruction model.

[0106] The image reconstruction module inputs the low-resolution image to be reconstructed into the trained high-frequency feature enhanced image super-resolution reconstruction model, extracts features of the image through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; extracts features of the shallow feature map through multiple high-frequency feature mutual adjustment modules mapped to each other in the deep feature extraction module to obtain a deep feature map; extracts features in the deep feature map through the convolution layer and sub-pixel convolution layer contained in the reconstruction module, adds the feature elements in the deep feature map and the shallow feature map, and performs pixel reconstruction to obtain a reconstructed high-resolution image.

[0107] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An image super-resolution reconstruction method based on high-frequency feature enhancement, characterized in that: The following steps are involved: Construct an image super-resolution reconstruction model framework with high-frequency feature enhancement, which includes a shallow feature extraction module and a reconstruction module; In the model framework, upsampling layers and downsampling layers of different scales are used to build a multi-scale sampling network, two convolutional layers and a depth-separable convolutional cascade are used to build a feature mask module FMB, and downsampling layers and upsampling layers of different scales are used to build a lightweight high-frequency enhanced attention mechanism HFG-ESA. The feature mask module FMB is used to train the high-frequency feature refinement ability of the multi-scale sampling network to obtain the trained multi-scale sampling network. The trained multi-scale sampling network is then integrated with the lightweight high-frequency enhanced attention mechanism to construct a high-frequency feature self-adjustment module. Six high-frequency feature self-adjustment modules are used and divided into two groups, a mask matrix is ​​generated using a high-frequency feature adjustment layer, and a mapping between the first three high-frequency feature self-adjustment modules and the last three high-frequency feature self-adjustment modules is constructed using the mask matrix to obtain a high-frequency feature mutual adjustment module, and multiple high-frequency feature mutual adjustment modules are fused into a deep feature extraction module within the reconstruction model framework to complete the training of the model, and the deep feature extraction module, the shallow feature extraction module and the reconstruction module are fused within the reconstruction model to obtain a trained high-frequency feature enhanced image super-resolution reconstruction model; The low-resolution image to be reconstructed is input into the trained high-frequency feature enhanced image super-resolution reconstruction model, and the image features are extracted through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; the shallow feature map is extracted through multiple high-frequency feature mutual adjustment modules that map to each other in the deep feature extraction module to obtain a deep feature map; the features in the deep feature map are extracted through the convolution layer and the sub-pixel convolution layer included in the reconstruction module, the feature elements in the deep feature map and the shallow feature map are added, and pixel reconstruction is performed to obtain a reconstructed high-resolution image.

2. The image super-resolution reconstruction method based on high-frequency feature enhancement according to claim 1, characterized in that: The high-frequency feature enhanced image super-resolution reconstruction model framework includes: The high-frequency feature enhanced image super-resolution reconstruction model framework consists of three parts: image shallow feature extraction module, image deep feature extraction module and image reconstruction module, and the formulas used are: F s =W conv3 (I LR ) F D =W HFMM (W HFMM (W HFMM (F S ))) I SR =W conv3 (W pixelshufle (W D ))+Bic(F S ) Where: I LR represents a low-resolution image; W conv3 (·) indicates a convolution with a kernel size of 3*3; F s Represents shallow features; W HFMM (·) represents the nonlinear mapping learned by the high-frequency feature mutual adjustment module; F D Represents the shallow feature F s The deep features obtained by four HFMM modules; W pixelshufle (·) indicates sub-pixel convolution layer; Bic(·) indicates bicubic interpolation upsampling; I SR represents the final reconstructed super-resolution image; W D Represents the deep features to be processed.

3. The image super-resolution reconstruction method based on high-frequency feature enhancement according to claim 1, characterized in that: The multi-scale sampling network comprises: The multi-scale sampling network extracts multi-scale high-frequency features by downsampling the input features by eight times, four times, and two times, respectively. The formula used is: Among them: F1 is the upper input of the high-frequency self-modulation module; They are eightfold, fourfold, and twofold downsampling after maximum pooling, respectively; Bicubic interpolation upsampling is eightfold, fourfold, and twofold respectively; F x8 、F x4 、F x2 These are the output features of F1 after eight times, four times, and two times of upsampling.

4. The image super-resolution reconstruction method based on high-frequency feature enhancement according to claim 1, characterized in that: The feature mask module comprises: The feature mask module FMB consists of two 1*1 convolutions and one 3*3 depth-separable convolution, and the formula used is: in: They are the double, quadruple, and eightfold high-frequency features extracted by the multi-scale sampling network; W FMB (·) is the transfer function of the feature mask module; α, β, and γ are the feature masks of each branch respectively; are the output characteristics of the three branches respectively; F hf is the sum of the high-frequency information output by the three branches; δ(·) is the Sigmoid activation function; is the element-wise multiplication operation; W conv1 (*) indicates 1*1 convolution; W dwconv1 (·) represents a 1*1 depthwise separable convolution; F up is the final high-frequency output feature; F1 is the upper input of the feature mask module.

5. The image super-resolution reconstruction method based on high-frequency feature enhancement according to claim 1, characterized in that: The formula used by the high-frequency characteristic self-adjustment module is: F up =W hf (F1) F down =σ(W conv3 (σ(W conv3 (F2)))+W conv3 (F2)) in: is the input of the module; split(·) is the channel split operation; F1 and F2 are the upper and lower branch inputs of the channel split respectively; W hf (·) is a multi-scale sampling network; F up is the upper branch output; σ(·) is the activation function GELU; F down is the output of the lower branch; concat(·) is the feature concatenation F up Operation; W HFG-ESA (·) High-frequency feature guided enhanced spatial attention (HFG-ESA); is the output of the module; W conv3 (·) indicates a convolution with a kernel size of 3*3.

6. The image super-resolution reconstruction method based on high-frequency feature enhancement according to claim 1, characterized in that: The high-frequency feature mutual adjustment module includes: The high-frequency feature mutual adjustment module includes six high-frequency feature self-adjustment modules; The high-frequency feature mutual adjustment module generates a mask matrix through the feature adjustment layer, and uses the mask matrix to adjust the output of the last three high-frequency feature self-adjustment modules to achieve effective mapping of the outputs of the first three high-frequency feature self-adjustment modules and establish the relationship between high-frequency features in different periods. The formula used is: Where: W conv1 ( ＊ ) represents 1*1 convolution; is the output feature of the i-th high-frequency feature self-adjustment module in the high-frequency feature mutual adjustment module; is the output feature of the i+3th high-frequency feature self-adjustment module in the high-frequency feature mutual adjustment module; F i,i+3 The output feature of the i-th high-frequency feature self-adjustment module is used as the previous feature, and the output feature of the i+3-th high-frequency feature self-adjustment module is used as the current feature; Mod( ＊ ) is the remainder function.

7. The image super-resolution reconstruction method based on high-frequency feature enhancement according to claim 2 is characterized in that: The extraction steps of the image super-resolution reconstruction model with high-frequency feature enhancement are as follows: The image shallow feature extraction module consists of a convolution with a convolution kernel size of 3*3. LR First, a 3*3 convolution is performed to extract shallow features to obtain shallow features F S ; The deep feature extraction module of the image is composed of multiple high-frequency feature mutual adjustment modules, each of which is composed of six high-frequency feature self-adjustment modules. S The deep feature F is obtained by adjusting multiple high-frequency feature mutual adjustment modules. D ; The image reconstruction module consists of a sub-pixel convolution layer and a 3*3 convolution layer. D After the sub-pixel convolution layer and 3*3 convolution, the shallow feature F is upsampled by bicubic interpolation. S Add together to get the reconstructed super-resolution image I SR .

8. An image super-resolution reconstruction device based on high-frequency feature enhancement, characterized in that: include: A model building module is used to build an image super-resolution reconstruction model framework with high-frequency feature enhancement, which includes a shallow feature extraction module and a reconstruction module; The model training module is used to construct a multi-scale sampling network using upsampling layers and downsampling layers of different scales in the model framework, construct a feature mask module FMB using two convolutional layers and a depth-wise separable convolutional cascade, and construct a lightweight high-frequency enhanced attention mechanism HFG-ESA using downsampling layers and upsampling layers of different scales; The feature mask module FMB is used to train the high-frequency feature refinement ability of the multi-scale sampling network to obtain the trained multi-scale sampling network. The trained multi-scale sampling network is then integrated with the lightweight high-frequency enhanced attention mechanism to construct a high-frequency feature self-adjustment module. Six high-frequency feature self-adjustment modules are used and divided into two groups, a mask matrix is ​​generated using a high-frequency feature adjustment layer, and a mapping between the first three high-frequency feature self-adjustment modules and the last three high-frequency feature self-adjustment modules is constructed using the mask matrix to obtain a high-frequency feature mutual adjustment module, and multiple high-frequency feature mutual adjustment modules are fused into a deep feature extraction module within the reconstruction model framework to complete the training of the model, and the deep feature extraction module, the shallow feature extraction module and the reconstruction module are fused within the reconstruction model to obtain a trained high-frequency feature enhanced image super-resolution reconstruction model; The image reconstruction module inputs the low-resolution image to be reconstructed into the trained high-frequency feature enhanced image super-resolution reconstruction model, extracts features of the image through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; extracts features of the shallow feature map through multiple high-frequency feature mutual adjustment modules mapped to each other in the deep feature extraction module to obtain a deep feature map; extracts features in the deep feature map through the convolution layer and sub-pixel convolution layer contained in the reconstruction module, adds the feature elements in the deep feature map and the shallow feature map, and performs pixel reconstruction to obtain a reconstructed high-resolution image.

Images