Image super-resolution reconstruction method based on multi-scale residual feature fusion
The image super-resolution reconstruction method using multi-scale residual feature fusion solves the problems of poor generality and large number of parameters in real image reconstruction, and achieves lightweight and efficient image restoration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING NORMAL UNIVERSITY
- Filing Date
- 2021-10-26
- Publication Date
- 2026-04-17
AI Technical Summary
Existing super-resolution networks have poor versatility in real image reconstruction and have a huge number of parameters, making deployment difficult.
An image super-resolution reconstruction method based on multi-scale residual feature fusion is adopted, which includes multi-scale feature extraction, residual feature fusion, enhanced attention module and adaptive upsampling, and combined with Charbonnier Loss function to construct a lightweight super-resolution reconstruction model.
It improves the versatility of image super-resolution reconstruction and significantly reduces the number of algorithm parameters, achieving lightweight and efficient image restoration.
Smart Images
Figure SMS_90 
Figure QLYQS_1 
Figure QLYQS_2
Abstract
Description
Technical Field
[0001] This invention relates to an image processing method, and more particularly to an image super-resolution reconstruction method based on multi-scale residual feature fusion. Background Technology
[0002] Existing super-resolution networks all generate high- and low-resolution image pairs on public datasets using fixed degradation methods such as known blur kernels. This results in poor generality of super-resolution reconstruction networks, making them unsuitable for real-world applications. Therefore, image super-resolution reconstruction based on real images has become a current research hotspot. Furthermore, super-resolution reconstruction algorithms based on real images suffer from a large number of parameters. While this large number of parameters can guarantee reconstruction performance, it hinders the deployment and application of the algorithms.
[0003] Therefore, in order to solve the above-mentioned technical problems, it is urgent to propose a new technical approach. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide an image super-resolution reconstruction method based on multi-scale residual feature fusion, which can effectively repair real images, ensure the versatility of the method, and at the same time, significantly reduce the algorithm parameters to achieve lightweighting while ensuring the reconstruction performance of the method.
[0005] This invention provides an image super-resolution reconstruction method based on multi-scale residual feature fusion, comprising the following steps:
[0006] S1. Acquire images at different resolutions and preprocess them to obtain high- and low-resolution image pairs;
[0007] S2. Construct a multi-scale feature extraction module based on depthwise separable convolution to extract features from preprocessed high- and low-resolution image pairs and output feature maps;
[0008] S3. Construct a residual feature fusion module to perform residual feature fusion processing on the output feature map;
[0009] S4. Construct an enhanced attention module to process the feature map after residual feature fusion;
[0010] S5. The feature map output in step S4 is upsampled using an adaptive upsampling module to generate a super-resolution image;
[0011] S6. Construct a loss function module based on Charbonnier Loss and process the super-resolution image;
[0012] S7. Construct a super-resolution image reconstruction model based on multi-scale residual feature fusion, and input the super-resolution image processed in step S6 into the super-resolution reconstruction model for training;
[0013] S8. Input the image to be processed into a super-resolution image reconstruction model based on multi-scale residual feature fusion for processing, and obtain the super-resolution reconstructed image information of the image to be processed.
[0014] Furthermore, in step S2, the multi-scale feature extraction module extracts features using the following formula:
[0015] ;
[0016] ;
[0017] ;
[0018] ;
[0019] ;
[0020] ;
[0021] ;
[0022] ;
[0023] ;
[0024] in: Let represent a linear rectified function, 'a' be a learnable constant, 'W' represent the weights, 'b' represent the bias parameters, the superscript of 'b' indicates the current layer number, the subscript of 'W' indicates the size of the convolution kernel, and the first parameter in the superscript of 'W' indicates the current layer number of the weights, and the second parameter indicates the position of the current weights in the convolution of that layer. The square brackets [] represent convolution operations; [] represent concatenation operations, which concatenate feature maps along the channel dimension. This represents the feature map output by the convolutional layer, and its subscript indicates which layer and which convolution it comes from.
[0025] The extraction process is as follows: Input feature map Passing through two respectively Convolution is used to increase or decrease dimensionality to obtain feature maps. and ,Then Execute once Channel separable convolution to obtain receptive field Feature map And then Execute once Channel separable convolution to obtain receptive field Feature map Then cascade , and Obtain multi-scale feature maps ;right Execute once Channel separable convolution obtains receptive field as Feature map And then Execute once Convolution to obtain the receptive field Feature map , final cascade , Use after Convolution reduces dimensionality, resulting in a better output feature map. Number of channels and Consistent.
[0026] Furthermore, in step S3, the residual feature fusion module is executed based on the following formula:
[0027] ;
[0028] ;
[0029] ;
[0030] ;
[0031] ;
[0032] ;
[0033] ;
[0034] ;
[0035] ;
[0036] Where: MSDSB represents the feature extraction module constructed in step S2; This represents the intermediate features in the residual feature fusion module, which are continuously updated during the module's execution. This represents the output feature map of each feature extraction module, and its subscript indicates which feature extraction module the feature map comes from.
[0037] The residual feature fusion process is as follows:
[0038] Input feature map First, the feature extraction module obtains its output feature map. ;
[0039] feature map The intermediate features are obtained by adding them to the input feature map. ;
[0040] The second and third feature extraction modules were used sequentially to extract the intermediate features. Perform feature extraction operations to obtain feature maps and And use residual connections to connect the feature maps with Add and update intermediate features ;
[0041] The fourth feature extraction module is used to extract intermediate features. The fourth feature map was obtained by performing feature extraction. ;
[0042] feature map To feature map Cascaded along the channel dimension, and through Convolutional processing is used to fuse and reduce the number of channels to obtain residual feature maps. ;
[0043] The input feature map is added to the residual feature map to obtain the output feature map. .
[0044] Furthermore, in step S4, the enhanced attention module is executed based on the following formula:
[0045] ;
[0046] ;
[0047] ;
[0048] ;
[0049] ;
[0050] ;
[0051] The specific process is as follows: A global average pooling is performed on the feature map output by the residual feature fusion module, and then the features are vectorized to obtain feature vectors. ;
[0052] The relationships between channels are adaptively established using two fully connected layers and an activation function, and the channel mask is obtained by compressing the feature vectors to the range of 0 to 1 using the Sigmoid function. ;
[0053] Use channel mask For the original feature channel Perform weighted processing and update to obtain a feature map calibrated by the channel dimension. ;
[0054] The feature map after processing by the channel attention module conduct Convolution dimensionality reduction to 1 channel Then execute twice in sequence. Convolution is used to capture the spatial relationships of an image through a large receptive field, while the second... Convolution reduces the number of channels to 1;
[0055] The feature map is compressed to a value between 0 and 1 using the Sigmoid function to generate a mask with spatial dimensions. Finally, the mask and the feature map are multiplied together to obtain the feature map after module calibration. .
[0056] Furthermore, in step S6, the loss function module adopts the following formula:
[0057] ;in, This represents the image generated by the network; I is the real image, and i, j, and k represent the corresponding pixels in the image's length, width, and color channels, respectively. The constant is set to 0.01, where h, w, and c represent the length, width, and color channels of the image, respectively.
[0058] Furthermore, the super-resolution reconstruction model is as follows:
[0059]
[0060] Where F represents the super-resolution network model proposed in this paper. This represents the parameters in the super-resolution model presented in this paper, with the ultimate goal of updating the parameters. This makes the loss function Minimize, that is:
[0061] .
[0062] The beneficial effects of this invention are as follows: This invention can effectively restore real images, avoiding the problem of poor versatility in super-resolution tasks. Moreover, through the operation of depthwise separable convolution and reusable convolution, the number of parameters of the algorithm is greatly reduced, and it has lightweight characteristics while ensuring reconstruction performance. Attached Figure Description
[0063] The present invention will be further described below with reference to the accompanying drawings and embodiments:
[0064] Figure 1 This is a flowchart of the present invention.
[0065] Figure 2 This is a schematic diagram of the feature extraction module of the present invention.
[0066] Figure 3 This is a schematic diagram of the residual feature fusion structure of the present invention.
[0067] Figure 4 This is a schematic diagram of the enhanced attention module structure of the present invention.
[0068] Figure 5 This is a schematic diagram of the super-resolution network structure of the present invention. Detailed Implementation
[0069] The present invention will be further described in detail below with reference to the accompanying drawings:
[0070] This invention provides an image super-resolution reconstruction method based on multi-scale residual feature fusion, comprising the following steps:
[0071] S1. Acquire images at different resolutions and preprocess them to obtain high- and low-resolution image pairs; wherein, existing image acquisition equipment is used to acquire images at different resolutions;
[0072] S2. Construct a multi-scale feature extraction module based on depthwise separable convolution to extract features from preprocessed high- and low-resolution image pairs and output feature maps;
[0073] S3. Construct a residual feature fusion module to perform residual feature fusion processing on the output feature map;
[0074] S4. Construct an enhanced attention module to process the feature map after residual feature fusion;
[0075] S5. The feature map output in step S4 is upsampled using an adaptive upsampling module to generate a super-resolution image;
[0076] S6. Construct a loss function module based on Charbonnier Loss and process the super-resolution image;
[0077] S7. Construct a super-resolution image reconstruction model based on multi-scale residual feature fusion, and input the super-resolution image processed in step S6 into the super-resolution reconstruction model for training;
[0078] S8. Input the image to be processed into the super-resolution image reconstruction model based on multi-scale residual feature fusion for processing to obtain the super-resolution reconstructed image information of the image to be processed. Through the above method, the real image can be effectively restored, avoiding the problem of poor generality in super-resolution tasks. Moreover, through the operation of depthwise separable convolution and reusable convolution, the number of parameters of the algorithm is greatly reduced, and it has lightweight characteristics while ensuring reconstruction performance.
[0079] In this embodiment, in step S2, the multi-scale feature extraction module extracts features using the following formula:
[0080] ;
[0081] ;
[0082] ;
[0083] ;
[0084] ;
[0085] ;
[0086] ;
[0087] ;
[0088] ;
[0089] in: Let represent a linear rectified function, 'a' be a learnable constant, 'W' represent the weights, 'b' represent the bias parameters, the superscript of 'b' indicates the current layer number, the subscript of 'W' indicates the size of the convolution kernel, and the first parameter in the superscript of 'W' indicates the current layer number of the weights, and the second parameter indicates the position of the current weights in the convolution of that layer. The square brackets [] represent convolution operations; [] represent concatenation operations, which concatenate feature maps along the channel dimension. This represents the feature map output by the convolutional layer, and its subscript indicates which layer and which convolution it comes from.
[0090] The extraction process is as follows: Input feature map Passing through two respectively Convolution is used to increase or decrease dimensionality to obtain feature maps. and ,Then Execute once Channel separable convolution to obtain receptive field Feature map And then Execute once Channel separable convolution to obtain receptive field Feature map Then cascade , and Obtain multi-scale feature maps ;right Execute once Channel separable convolution obtains receptive field as Feature map And then Execute once Convolution to obtain the receptive field Feature map , final cascade , Use after Convolution reduces dimensionality, resulting in a better output feature map. Number of channels and In summary, the above methods effectively lightweight the entire algorithm process, consume fewer resources, and ensure the accuracy of the final result.
[0091] In this embodiment, in step S3, the residual feature fusion module performs the operation based on the following formula:
[0092] ;
[0093] ;
[0094] ;
[0095] ;
[0096] ;
[0097] ;
[0098] ;
[0099] ;
[0100] ;
[0101] Where: MSDSB represents the feature extraction module constructed in step S2; This represents the intermediate features in the residual feature fusion module, which are continuously updated during the module's execution. This represents the output feature map of each feature extraction module, and its subscript indicates which feature extraction module the feature map comes from.
[0102] The residual feature fusion process is as follows:
[0103] Input feature map First, the feature extraction module obtains its output feature map. ;
[0104] feature map The intermediate features are obtained by adding them to the input feature map. ;
[0105] The second and third feature extraction modules were used sequentially to extract the intermediate features. Perform feature extraction operations to obtain feature maps and And use residual connections to connect the feature maps with Add and update intermediate features ;
[0106] The fourth feature extraction module is used to extract intermediate features. The fourth feature map was obtained by performing feature extraction. ;
[0107] feature map To feature map Cascaded along the channel dimension, and through Convolutional processing is used to fuse and reduce the number of channels to obtain residual feature maps. ;
[0108] The input feature map is added to the residual feature map to obtain the output feature map. The above methods can effectively ensure the accuracy of the final results.
[0109] In this embodiment, in step S4, the enhanced attention module is executed based on the following formula:
[0110] ;
[0111] ;
[0112] ;
[0113] ;
[0114] ;
[0115] ;
[0116] The specific process is as follows: A global average pooling is performed on the feature map output by the residual feature fusion module, and then the features are vectorized to obtain feature vectors. ;
[0117] The relationships between channels are adaptively established using two fully connected layers and an activation function, and the channel mask is obtained by compressing the feature vectors to the range of 0 to 1 using the Sigmoid function. ;
[0118] Use channel mask For the original feature channel Perform weighted processing and update to obtain a feature map calibrated by the channel dimension. ;
[0119] The feature map after processing by the channel attention module conduct Convolution dimensionality reduction to 1 channel Then execute twice in sequence. Convolution is used to capture the spatial relationships of an image through a large receptive field, while the second... Convolution reduces the number of channels to 1;
[0120] The feature map is compressed to a value between 0 and 1 using the Sigmoid function to generate a mask with spatial dimensions. Finally, the mask and the feature map are multiplied together to obtain the feature map after module calibration. .
[0121] Furthermore, in step S6, the loss function module adopts the following formula:
[0122] ;in, The image generated by the network is the final output super-resolution image; I represents the real image, i.e., the original image, where i, j, and k represent the corresponding pixels in the image's length, width, and color channels, respectively. The constant is set to 0.01, where h, w, and c represent the length, width, and color channels of the image, respectively.
[0123] In this embodiment, the super-resolution reconstruction model is as follows:
[0124]
[0125] Where F represents the super-resolution network model proposed in this paper. This represents the parameters in the super-resolution model presented in this paper, with the ultimate goal of updating the parameters. This makes the loss function Minimize, that is:
[0126] At this point, the final super-resolution image is output. .
[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. An image super-resolution reconstruction method based on multi-scale residual feature fusion, characterized in that: Includes the following steps: S1. Acquire images at different resolutions and preprocess them to obtain high- and low-resolution image pairs; S2. Construct a multi-scale feature extraction module based on depthwise separable convolution to extract features from preprocessed high- and low-resolution image pairs and output feature maps; S3. Construct a residual feature fusion module to perform residual feature fusion processing on the output feature map; S4. Construct an enhanced attention module to process the feature map after residual feature fusion; S5. The feature map output in step S4 is upsampled using an adaptive upsampling module to generate a super-resolution image; S6. Construct a loss function module based on Charbonnier Loss and process the super-resolution image; S7. Construct a super-resolution image reconstruction model based on multi-scale residual feature fusion, and input the super-resolution image processed in step S6 into the super-resolution reconstruction model for training; S8. Input the image to be processed into a super-resolution image reconstruction model based on multi-scale residual feature fusion for processing, and obtain the super-resolution reconstructed image information of the image to be processed; In step S2, the multi-scale feature extraction module extracts features using the following formula: ； ； ； ； ； ； ； ； ； in: Let W represent the linear rectified function, where a is a learnable constant, W represents the weights, b represents the bias parameters, the superscript of b indicates the current layer number, the subscript of W indicates the size of the convolution kernel, and the first parameter in the superscript of W indicates the current layer number of the weights, and the second parameter indicates the position of the current weights in the convolution of the current layer. The square brackets [] represent convolution operations; [] represent concatenation operations, which concatenate feature maps along the channel dimension. This represents the feature map output by the convolutional layer, and its subscript indicates which layer and which convolution it comes from. The extraction process is as follows: Input feature map Passing through two respectively Convolution is used to increase or decrease dimensionality to obtain feature maps. and ,Then Execute once Channel separable convolution to obtain receptive field Feature map And then Execute once Channel separable convolution to obtain receptive field Feature map Then cascade , and Obtain multi-scale feature maps ;right Execute once Channel separable convolution obtains receptive field as Feature map And then Execute once Convolution to obtain the receptive field Feature map , final cascade , Use after Convolution reduces dimensionality, resulting in a better output feature map. Number of channels and Consistent; In step S3, the residual feature fusion module is executed based on the following formula: ； ； ； ； ； ； ； ； ； Where: MSDSB represents the feature extraction module constructed in step S2; This represents the intermediate features in the residual feature fusion module, which are continuously updated during the module's execution. This represents the output feature map of each feature extraction module, and its subscript indicates which feature extraction module the feature map comes from. The residual feature fusion process is as follows: Input feature map First, the feature extraction module obtains its output feature map. ; feature map The intermediate features are obtained by adding them to the input feature map. ; The second and third feature extraction modules were used sequentially to process the intermediate features. Perform feature extraction operations to obtain feature maps and And use residual connections to connect the feature maps with Add and update intermediate features ; The fourth feature extraction module is used to extract intermediate features. The fourth feature map was obtained by performing feature extraction. ; feature map To feature map Cascaded along the channel dimension, and through Convolutional processing is used to fuse and reduce the number of channels to obtain residual feature maps. ; The input feature map is added to the residual feature map to obtain the output feature map. ; In step S4, the enhanced attention module is executed based on the following formula: ； ； ； ； ； ； The specific process is as follows: A global average pooling is performed on the feature map output by the residual feature fusion module, and then the features are vectorized to obtain feature vectors. ; The relationships between channels are adaptively established using two fully connected layers and an activation function, and the channel mask is obtained by compressing the feature vectors to the range of 0 to 1 using the Sigmoid function. ; Use channel mask For the original feature channel Perform weighted processing and update to obtain a feature map calibrated by the channel dimension. ; The feature map after processing by the channel attention module conduct Convolution dimensionality reduction to 1 channel Then execute twice in sequence. Convolution is used to capture the spatial relationships of an image through a large receptive field, while the second... Convolution reduces the number of channels to 1; The feature map is compressed to a value between 0 and 1 using the Sigmoid function to generate a mask with spatial dimensions. Finally, the mask and the feature map are multiplied together to obtain the feature map after module calibration. .
2. The image super-resolution reconstruction method based on multi-scale residual feature fusion according to claim 1, characterized in that: In step S6, the loss function module uses the following formula: ;in, This represents the image generated by the network; I is the real image, and i, j, and k represent the corresponding pixels in the image's length, width, and color channels, respectively. The constant is set to 0.01, where h, w, and c represent the length, width, and color channels of the image, respectively.
3. The image super-resolution reconstruction method based on multi-scale residual feature fusion according to claim 1, characterized in that: The super-resolution reconstruction model is as follows: Where F represents the super-resolution network model proposed in this paper. This represents the parameters in the super-resolution model presented in this paper, with the ultimate goal of updating the parameters. This makes the loss function Minimize, that is: 。