Image super-resolution reconstruction method based on multilevel intermediate feature processing

Abstract

The invention provides an image super-resolution reconstruction method based on multilevel intermediate feature processing, aiming at the problems of information extraction limitation and high-frequency detail loss of a traditional image super-resolution method. The method comprises three processing steps: firstly, extracting shallow layer features from an input low-resolution image; then, designing a parallel strategy to extract feature information streams from different levels and adaptively calibrate the feature information streams; in addition, a correlation between adjacent feature maps of the intermediate features is modeled and a weight assignment for each pixel is calculated; and finally, adding all information streams through up-sampling recovery and interpolation results of the original image to obtain reconstructed image data. According to the method provided by the invention, the problems of insufficient information extraction and texture detail loss in the existing image super-resolution technology are solved, the reconstructed image details are more accurate, and the perception quality of the image is improved.

Description

Technical Field

[0001] The present invention relates to the problem of image super-resolution reconstruction in the field of digital image processing, and in particular to an image super-resolution reconstruction method based on multi-level intermediate feature processing. Background Art

[0002] As the whole society has entered the era of informatization and digitalization in recent years, image super-resolution (SR) technology has been widely used in the production and life of human society. Higher-quality images can improve the visual comfort in daily life and entertainment. At the same time, high-quality images are the prerequisite for performing advanced tasks in computer vision processing. In many cases, due to comprehensive factors such as equipment cost, transmission bandwidth, and storage space, some low-resolution (LR) images still exist in large quantities in people's lives. Reconstructing high-resolution (HR) images through image super-resolution algorithms is not easily affected by environmental factors, hardware performance limits, and other factors. However, most existing algorithms do not fully extract and utilize the features of the original low-resolution images, so research on how to reconstruct high-resolution images with higher quality is still in progress.

[0003] Most existing deep learning-based image super-resolution algorithms have achieved good results. However, in the stage of extracting features, they are usually extracted from a simple small filter stack, which lacks flexibility in dealing with spatial contexts of different scales. In addition, intermediate features are not fully utilized before being passed to the next layer, which leads to the omission of important information and the loss of high-frequency details. Existing methods lack sufficient consideration of the impact of feature information at different levels and intermediate feature information on the reconstruction of super-resolution images. For the above reasons, how to fully extract and utilize features to reconstruct high-resolution images to ensure that the reconstruction results have more accurate details, slighter distortions, and milder artifacts is of great significance to the research in the field of image super-resolution. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an image super-resolution reconstruction method based on multi-level intermediate feature processing to improve the reconstruction quality of high-resolution images.

[0005] The following technical measures are adopted to solve the above technical problems: an image super-resolution reconstruction method based on multi-level intermediate feature processing mainly includes the following steps:

[0006] (1) Feature extraction: A 3×3 regular convolution is used to extract shallow feature expressions from the original input LR image.

[0007] (2) Feature Mapping: The features are further extracted and utilized through the Enhanced Feature Mining Module (EFM). In EFM, a self-calibrated multi-scale module is first designed to extract hierarchical features. Parallel branches with different receptive fields capture different information expressions at each level and then perform feature fusion. A calibration branch is used to merge the underlying basic information into the information at each level. The information at each level is adjusted as a whole to obtain the output intermediate feature map. Secondly, the dual-branch attention module will further utilize the intermediate features. First, the generated intermediate feature map uses a three-dimensional convolution kernel to model the correlation of the feature maps in the adjacent space to obtain the spatial context information of the digital image. At the same time, the confidence of each pixel position is calculated for the intermediate feature map to obtain the corresponding pixel weight distribution. Finally, they are integrated through the gated fusion node.

[0008] (3) Upsampling restoration: Based on the feature map obtained after processing in (2), it is enlarged to the upsampling size through interpolation. The important information and convolution fine-tuning are retained through the designed feature attention component, and finally fused with the bicubic interpolation map of the original image to form the final image data.

[0009] Preferably, the adaptively calibrated multi-scale module in step (2) constructs three branches with different receptive fields to simultaneously extract features at different levels, and uses a 1×1 convolution kernel for feature fusion, while using a 3×3 convolution kernel to open up another parallel calibration branch to explicitly merge basic information from the bottom layer into the captured multi-level information.

[0010] Preferably, the dual-branch attention block consists of a layer attention component (LA) and a feature attention component (FA). The main components of the layer attention component include a 3D convolution kernel and a sigmoid function. First, the hierarchical feature f MS Input to the 3D convolution layer, adjacent feature maps are convolved in three-dimensional space to model the correlation between adjacent features and obtain the correlation coefficient between adjacent feature maps. Then, the coefficient is mapped to (0, 1) through the Sigmoid function to generate a three-dimensional attention matrix. After completing the corresponding correlation coefficient calculation, the input feature f MS Perform matrix multiplication with the attention matrix to generate a weighted feature map to strengthen the hierarchical relationship between adjacent features. Finally, the input features are added to the output features in the channel direction in a skip connection manner. This residual structure can ensure sufficient information flow and improve training efficiency. The specific layer attention component calculation formula is:

[0011]

[0012] Among them, H 3D and H SRepresents 3D convolution function and Sigmoid activation function respectively. LA The output of the presentation layer attention component.

[0013] Preferably, the feature attention component directly uses a 1×1 convolution to re-nonlinearly adjust the input. Then, the Sigmoid function assigns weights to each pixel of the input feature map to obtain an attention map. This method takes into account the different contributions of pixels at different positions, thereby avoiding the loss of some information due to the reduction in the dimensionality of feature information. At the same time, the input features are adjusted through a convolution layer of size 3 to reduce redundant information. Finally, the attention map is multiplied by the adjusted input features to emphasize more important information. The specific formula of the feature attention component is:

[0014]

[0015] Among them, H S is the Sigmoid activation function. C1 and H C3 Represent 1×1 and 3×3 convolution operations respectively.

[0016] Preferably, after the feature information is generated by the layer attention component and the feature attention component, they are integrated through the gated fusion node. Specifically, the gated fusion node connects the feature information output by the two attention components on the channel and adaptively fuses the information using a convolutional layer of size 1×1. The final dual-branch attention block output calculation formula is:

[0017] f DAB =H GF (f LA ,f FA )

[0018] Among them, H GF and f DAB They represent the features generated by GF function and DAB respectively.

[0019] Preferably, the upsampling recovery layer is composed of a linear interpolation function, a feature attention component and a convolution layer. The feature attention component is introduced into the upsampling recovery layer to help the network pay attention to important changes in feature information. The final reconstructed image is obtained by adding the output features of the upsampling recovery layer and the original low-resolution image interpolated to the reconstructed image size. This method can retain the basic structural information in the original image and avoid the loss of structural information in the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the algorithm network structure of the present invention.

[0021] Figure 2 Schematic diagram of the calibrated multi-scale module.

[0022] Figure 3 Schematic diagram of the dual-branch attention block.

[0023] Figure 4 Schematic diagram of the upsampling recovery layer.

[0024] Figure 5 A visual qualitative comparison on the benchmark with a magnification factor of ×4. DETAILED DESCRIPTION

[0025] The present invention is further described below in conjunction with the accompanying drawings. It is necessary to point out here that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Persons skilled in the art in this field may make some non-essential improvements and adjustments to the present invention based on the above content of the present invention, which still fall within the scope of protection of the present invention.

[0026] Combination Figure 1 , specifically involving an image super-resolution reconstruction method based on multi-level intermediate feature processing, which can be divided into the following steps:

[0027] (4) Feature extraction: A shallow feature expression is extracted from the original input LR image through a 3×3 regular convolution.

[0028] The calculation formula for shallow feature extraction is:

[0029] f0=H SFE (I LR )

[0030] Among them I LR is the input LR image, H SFE represents a shallow feature extraction function, which is a convolution kernel of size 3×3.

[0031] f0 is the extracted shallow feature.

[0032] (5) Feature Mapping: The features are further extracted and utilized by the Enhanced Feature Mining Module (EFM). In EFM, a self-calibrated multi-scale module is first designed to extract hierarchical features. Parallel branches with different receptive fields capture different information expressions at each level and then perform feature fusion. A calibration branch is used to merge the underlying basic information into the information at each level. The information at each level is adjusted as a whole to obtain the output intermediate feature map. Secondly, the dual-branch attention module will further utilize the intermediate features. First, the generated intermediate feature map uses a three-dimensional convolution kernel to model the correlation of the feature maps in the adjacent space to obtain the spatial context information of the digital image. At the same time, the confidence of each pixel position is calculated for the intermediate feature map to obtain the corresponding pixel weight distribution. Finally, they are integrated through the gated fusion node.

[0033] (6) Upsampling restoration: Based on the feature map obtained after processing in (2), it is enlarged to the upsampling size through interpolation. The important information and convolution fine-tuning are retained through the designed feature attention component, and finally fused with the bicubic interpolation map of the original image to form the final image data.

[0034] Specifically, in step (2), in order to make full use of image features and help the neural network generate more discriminative representations, the present invention designs an adaptively calibrated multi-scale module (CMS), which adopts a method of 3-branch parallel information extraction and 1-branch adaptive calibration.

[0035] In the specific implementation process, Figure 2 As shown in the figure, three branches with different receptive fields are constructed to extract features at different levels at the same time. In order to extract more advanced semantic information and ensure computational efficiency as much as possible, in the proposed structure, all feature extraction branches use convolution kernels of size 3×3, and the range of the receptive field is determined by the number of convolution kernels. At the same time, in one of the convolution branches, a hole convolution (marked in green) is used to replace the regular convolution to obtain a larger receptive field. In order to avoid the loss of local information due to the lack of correlation of information obtained from a long distance when the expansion rate is too large, resulting in blurring and artifacts in the reconstructed image. Therefore, the expansion rate is set to 2 to expand the receptive field while avoiding the problem of information loss.

[0036] In the three feature extraction branches, 1×1 convolution is used to reduce the feature dimension to reduce computational complexity. The three branches with different receptive fields capture feature information at different levels to identify complex texture details. The geometric details and semantic information of the image features are then connected on the channel. Since these features contain redundant information, a 1×1 convolution kernel is used for feature fusion. On the other hand, this convolution kernel is also used to change the channel dimension to be consistent with the input dimension.

[0037] As the network goes deeper, feature information becomes more abstract, which is likely to weaken the information from the bottom layer. To address this problem, another parallel calibration branch is opened up using a 3×3 convolution kernel to explicitly merge the basic information from the bottom layer into the captured multi-level information, optimizing the information flow of the network. This strategy enables the model to holistically adjust the hierarchical information and strengthen the interaction between information.

[0038] In step (2), in order to make full use of the additional information contained in the intermediate features to improve the comprehensiveness and comprehensiveness of the acquired feature information, the dual-branch attention block (DAB) extracts more useful information from the intermediate features to guide image reconstruction.

[0039] In the specific implementation process, Figure 3 As shown, the dual-branch attention block consists of a layer attention component (LA) and a feature attention component (FA). The goal of the layer attention component is to allow the network to learn the correlation dependencies between adjacent feature maps so as to accurately infer the contextual information in their low-resolution images and achieve high-quality restoration.

[0040] like Figure 3 As shown in Figure 2, the main components of the layer attention component include a 3D convolution kernel and a Sigmoid function. First, the hierarchical feature f MS Input to the 3D convolution layer, adjacent feature maps are convolved in three-dimensional space to model the correlation between adjacent features and obtain the correlation coefficient between adjacent feature maps. Then, the coefficient is mapped to (0, 1) through the Sigmoid function to generate a three-dimensional attention matrix. After completing the corresponding correlation coefficient calculation, the input feature f MS Perform matrix multiplication with the attention matrix to generate a weighted feature map to strengthen the hierarchical relationship between adjacent features. Finally, the input features are added to the output features in the channel direction in a skip connection manner. This residual structure can ensure sufficient information flow and improve training efficiency. The specific layer attention component calculation formula is:

[0041]

[0042] Among them, H 3D and H S Represents 3D convolution function and Sigmoid activation function respectively. LA The output of the presentation layer attention component.

[0043] In addition, in order to emphasize important information and make the algorithm model pay attention to more details of the image, a feature attention component is proposed to synergistically improve the super-resolution performance of the image, ensuring attention to the detailed information of the features at the pixel level when the feature information flow is passed forward.

[0044] Specifically, first, the input is directly re-nonlinearly adjusted using a 1×1 convolution. Then, the Sigmoid function assigns weights to each pixel of the input feature map to obtain an attention map. This method takes into account the different contributions of pixels at different positions, thereby avoiding the loss of some information due to the reduction in the dimensionality of feature information. At the same time, the input features are adjusted through a convolution layer of size 3 to reduce redundant information. Finally, the attention map is multiplied by the adjusted input features to emphasize more important information. The specific formula of the feature attention component is:

[0045]

[0046] Among them, H Sis the Sigmoid activation function. C1 and H C3 Represent 1×1 and 3×3 convolution operations respectively.

[0047] After the feature information is generated by the layer attention component and the feature attention component, they are integrated through the gated fusion node. Specifically, the gated fusion node connects the feature information output by the two attention components on the channel and adaptively fuses the information using a convolutional layer of size 1×1. The final dual-branch attention block output calculation formula is:

[0048] f DAB =H GF (f LA ,f FA )

[0049] Among them, H GF and f DAB They represent the features generated by GF function and DAB respectively. Input feature f MS , the output feature f of the layer attention component LA , the output feature f of the feature attention component FA The features output by the final two-branch attention block have the same number of channels and spatial resolution.

[0050] In step (3) shown in FIG. Figure 4 The upsampling recovery layer shown is composed of a linear interpolation function, a feature attention component, and a convolutional layer. It upsamples the output high-level features to the size of the reconstructed image for final mapping. As mentioned earlier, the feature attention component helps the network to effectively utilize features, so the feature attention component is introduced in the upsampling recovery layer to help the network pay attention to important changes in feature information. The final reconstructed image is obtained by adding the output features of the upsampling recovery layer to the original low-resolution image interpolated to the size of the reconstructed image. This method can retain the basic structural information in the original image and avoid the loss of structural information in the original image.

[0051] In order to better illustrate the effectiveness of the present invention, the present invention is used to conduct reconstruction experiments on public benchmark datasets in the field of image super-resolution, and then compared with methods such as EDSR, RDN, SRMDNF, SRFBN and PAN, and the peak signal-to-noise ratio PSNR and structural similarity SSIM of the reconstructed image are calculated to measure the algorithm effect. Table 1 shows the SSIM and PSNR comparison between the present invention and other advanced image super-resolution methods, indicating that the present algorithm has more advantages than other advanced methods.

[0052] Figure 5In order to compare the visual effects of the present invention with other advanced methods, the test images are selected from pictures with rich texture backgrounds. Experiments have shown that most of the compared image super-resolution algorithms cannot restore textures and line patterns more accurately, and will produce obvious distortion and blurring caused by spatial aliasing for dense lines in buildings. On the other hand, the present invention is more successful in reconstructing textures and edges in super-resolution images, and can obtain clearer details, and the reconstructed images have fewer artifacts.

[0053] Table 1 Quantitative comparison with advanced algorithms when the scale factor is ×4

[0054]

Claims

1. An image super-resolution reconstruction method based on multi-level intermediate feature processing, characterized in that The following steps are involved: (1) Feature extraction: A 3×3 regular convolution is used to extract shallow feature expressions from the original input LR image. (2) Feature Mapping: The features are further extracted and utilized through the Enhanced Feature Mining Module (EFM). In the EFM, a self-calibrated multi-scale module is first designed to extract hierarchical features. Parallel branches with different receptive fields capture different information expressions at each level and then perform feature fusion. A calibration branch is used to merge the underlying basic information into the information at each level. The information at each level is adjusted as a whole to obtain the output intermediate feature map. Secondly, the dual-branch attention module will further utilize the intermediate features. First, the layer attention component will generate the intermediate feature map using a three-dimensional convolution kernel to model the correlation of the feature maps in the adjacent space to obtain the spatial context information of the digital image. At the same time, the feature attention component calculates the confidence of each pixel position in the intermediate feature map to obtain the corresponding pixel weight distribution. Finally, they are integrated through the gated fusion node. (3) Upsampling recovery: Based on the feature map obtained after processing in (2), the upsampling recovery layer enlarges it to the upsampling size through interpolation, retains important information and convolution fine-tuning through the designed feature attention component, and finally merges it with the interpolation map of the original image to form the final image data.

2. The image super-resolution reconstruction method based on multi-level intermediate feature processing according to claim 1 is characterized in that In the extraction stage, the low-resolution image is subjected to a 3×3 conventional convolution to extract shallow feature information f0 from the original input LR image. The formula is expressed as: f0=H SFE (I LR ) Among them I LR is the input LR image, H SFE represents the shallow feature extraction function, which is a convolution kernel of size 3×3.

3. The image super-resolution reconstruction method based on multi-level intermediate feature processing according to claim 1 is characterized in that: The self-calibrated multi-scale module first uses parallel extraction kernel branches with different receptive fields to extract information from the input. Then, a 1×1 convolution kernel is used for fusion, and a 3×3 convolution kernel is used as a calibration branch to merge the underlying basic information into the captured information at each level, so that the model can adjust the level information as a whole and strengthen the interaction between information.

4. The image super-resolution reconstruction method based on multi-level intermediate feature processing according to claim 1 is characterized in that: The layer attention component first models the correlation between adjacent feature maps in three-dimensional space to obtain a correlation coefficient matrix, and then multiplies the matrix with the original input features to generate a weighted feature map and use the original input features as information supplement. The calculation formula can be expressed as: Among them, H 3D and H S Represents 3D convolution function and Sigmoid activation function, F LA The output of the presentation layer attention component.

5. The image super-resolution reconstruction method based on multi-level intermediate feature processing according to claim 1, characterized in that: The feature attention component aims to preserve image details by directly using 1×1 convolution to perform nonlinear adjustment on the intermediate feature map, and calculates the weight of each pixel at the pixel level for the intermediate feature map to obtain different confidence levels for pixels at different positions, thus avoiding the loss of information caused by dimensionality reduction during the transmission of feature information. The calculation formula is expressed as: Among them, H S is the Sigmoid activation function, H C1 and H C3 They represent 1×1 and 3×3 convolution operations respectively. Finally, in order to adaptively fuse information, they are integrated through the gated fusion node. The final dual-branch attention block can be expressed as: f DAB =H GF (f LA ,f FA ) Among them, H GF and f DAB Represent the features generated by GF function and DAB respectively, and input feature f MS , the output feature f of the layer attention component LA , the output feature f of the feature attention component FA The features output by the final two-branch attention block have the same number of channels and spatial resolution.

6. The image super-resolution reconstruction method based on multi-level intermediate feature processing according to claim 1, characterized in that: The upsampling recovery stage upsamples the output high-level features to the size of the reconstructed image, and finally fine-tunes them through two 3×3 convolutions. A feature attention component is introduced in the upsampling recovery layer to help the network pay attention to important changes in feature information. The final reconstructed image is obtained by adding the output features of the upsampling recovery layer and the original low-resolution image interpolated to the size of the reconstructed image.

Images