Image super-resolution restoration method, system, electronic device and storage medium

Abstract

The present disclosure provides a method, system, electronic device and computer-readable storage medium for super-resolution restoration of an image, the method comprising: using a convolution kernel to process the image to be processed to obtain a low-resolution image, then using a 1×1 convolution kernel for dimensional conversion, and using a preset number of convolution kernels with different receptive fields of adaptive learning weights to extract multi-scale and hierarchical shallow features; using a double-layer nested residual module to extract deep features, and recursively executing a preset number of times; fusing the features extracted by each convolution kernel of the shallow features with the features obtained by the corresponding number of recursive executions to obtain each fused feature; using a preset number of reconstruction branches, using sub-pixel convolution kernels to reconstruct images for each obtained fused feature, and adding all branch results to generate a final high-resolution image. The disclosed technical solution can solve the problem of complex parameters of the image restoration model and weak representation ability of the network.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to a method for super-resolution restoration of an image, a system for super-resolution restoration of an image, an electronic device, and a computer-readable storage medium. Background Art

[0002] Aerial photography technology can quickly obtain ground information and bring convenience to people in some special environments. However, when the aerial photography flies to a certain height, the pixel points in the aerial image will contain a larger scene size, resulting in a low image resolution. For this reason, the image needs to be processed for resolution recovery. At present, the hot research on super-resolution reconstruction of aerial images is mainly focused on image super-resolution reconstruction algorithms based on deep learning, such as using sparse representation-based methods, using compressed sensing, and using a large number of samples to train a dictionary corresponding to high resolution and low resolution, and using sparse reconstruction of high-resolution images; the method based on deep learning is to obtain the relationship between low-resolution images and high-resolution images through learning, and then use deep convolutional neural network training samples to obtain high-frequency information to supplement the information of the original image and reconstruct high-resolution images.

[0003] In the current image restoration methods based on deep learning, the number of trainable network layers and channels determines the performance of the model. Although heavyweight neural networks have good performance, they have a large number of parameters. It is difficult to give full play to their advantages under resource-constrained conditions, and it is difficult to balance performance and scale. In addition, in recent years, many super-resolution methods based on convolutional neural networks have ignored the interdependence between channels, which hinders the network's representation capabilities. Summary of the invention

[0004] In order to at least solve the technical problems in the prior art that the performance and scale of the neural network model for image restoration are difficult to balance, the parameters are complex, and the network's representation ability is weak, the present disclosure provides a super-resolution restoration method, system, electronic device and computer-readable storage medium for an image, which uses a lightweight network to significantly reduce network parameters while improving image resolution through an improved deep learning method, and uses a multi-level and multi-scale approach to extract shallow features, enriching feature information, promoting the flow of information between channels, reducing parameters, and enhancing network expression capabilities.

[0005] In a first aspect, the present disclosure provides a method for super-resolution restoration of an image, the method comprising:

[0006] The image to be processed is processed using a convolution kernel to obtain a low-resolution image;

[0007] Using a 1×1 convolution kernel to perform dimension transformation on the low-resolution image, and using a preset number of convolution kernels with different receptive fields and adaptive learning weights to extract multi-scale and hierarchical shallow feature information from the low-resolution image after dimension transformation;

[0008] Extracting deep features from the low-resolution image using a double-layer nested residual module, and recursively performing the preset number of times when extracting deep feature information;

[0009] The features extracted by each convolution kernel with different receptive fields when extracting shallow features are fused with the features obtained by completing the corresponding number of recursive executions in extracting deep features through dot multiplication to obtain each fused feature;

[0010] Using the preset number of reconstruction branches, each fusion feature obtained is reconstructed using a sub-pixel convolution kernel, and the results obtained by all branches are added together to generate a final high-resolution image.

[0011] The number of channels of feature information can be adjusted through dimensional conversion, and the preset number can be 3. The convolution kernels of three different receptive fields with adaptive learning weights are divided into three branches, and gradually weighted superposition is performed to extract multi-scale and hierarchical shallow feature information from the low-resolution image after dimensional conversion; three residual groups with local jump connections are used to extract deep features from the low-resolution image, and the residual group contains a double-layer nested residual block that uses a strategy of first expansion and then compression to reduce the number of channels and the amount of parameters. Through the multi-channel reconstruction branch (module), the image is reconstructed using a 1×1 convolution layer and a sub-pixel convolution kernel for each fusion feature obtained, and the results obtained from all branches are added to generate a high-resolution image with the same size as the original image.

[0012] Furthermore, the preset number is 3, and the convolution kernels of different receptive fields include a 1×1 convolution kernel, a 3×3 convolution kernel, and a 5×5 convolution kernel;

[0013] The method of extracting multi-scale and hierarchical shallow feature information from the low-resolution image after dimension conversion using a preset number of convolution kernels with different receptive fields and adaptive learning weights includes:

[0014] The Fs obtained by dimensionally transforming the low-resolution image using a 1×1 convolution kernel 0 Use 1×1 convolution kernel to extract shallow features to get feature Fs 1 ;

[0015] The obtained Fs 1 Multiply it with the corresponding adaptive learning weight and then add it to Fs 0 After adding the points, a 3×3 convolution kernel is used to extract shallow features to obtain feature Fs 2 ;

[0016] The obtained Fs 2 Multiply it with the corresponding adaptive learning weight and then add it to Fs 0 After adding the points, use a 5×5 convolution kernel to extract shallow features to obtain feature Fs 3 .

[0017] Furthermore, the extracting deep features of the low-resolution image by using a double-layer nested residual module, and recursively performing the preset number of times when extracting deep feature information, includes:

[0018] Construct an embedded module with a convolution kernel of 1×1, which includes a channel shuffling module;

[0019] The residual module with a convolution kernel of 3×3 is used as the external module, and a channel shuffling module is added to the external module. The embedded module and the external module are then combined to form a double-layer nested residual module.

[0020] A plurality of double-layer nested residual modules are used to extract deep features from the low-resolution image, and the deep features are extracted three times in a recursive manner to obtain deep features FD 1 , FD 2 and FD 3 .

[0021] The deep features of the low-resolution image are extracted by adding a residual group of local skip connections, wherein the residual group includes a double-layer nested residual block and uses a strategy of first expansion and then compression to reduce the number of channels and the amount of parameters; and when extracting the deep feature information, the preset number of times is performed in a recursive manner, including:

[0022] Construct a convolution kernel embedded module. The embedded module is to transform the commonly used residual module. The two 3×3 convolution kernels at the beginning and end of the module are changed to 1×1 convolution kernels, so as to achieve the effect of slowly increasing parameters and add channel confusion;

[0023] As an external module, the residual module is composed of two 3×3 convolution kernels, an activation function and a channel shuffling function. The first convolution kernel not only extracts features but also expands the feature channel to obtain rich information. The second convolution kernel compresses the feature channel while screening features, which is conducive to the transmission of effective information. The embedded module and the external module are combined to form a double-layer nested residual module.

[0024] Furthermore, the channel scrambling module is specifically used for:

[0025] First, compress the channel through a 1×1 convolution layer, then group the compressed channels through a group convolution and perform channel shuffling, then restore the number of output channels to the number of input channels through a second 1×1 convolution layer, perform global average pooling on the output result, and use the preset logistic regression function to calculate the channel weight and multiply it with the corresponding channel.

[0026] By performing channel shuffling, the flow of information between channels is achieved.

[0027] Furthermore, the features extracted by the convolution kernels of different receptive fields when extracting shallow features are respectively fused with the features obtained by completing the corresponding number of recursive executions in extracting deep features by means of dot multiplication, including:

[0028] The feature Fs obtained when extracting shallow features 1 The result FD obtained by the first recursion when extracting deep features 1 Fusion of features by dot product;

[0029] The feature Fs obtained when extracting shallow features 2 The feature FD obtained by the second recursion when extracting deep features 2 Fusion of features by dot product;

[0030] The feature Fs obtained when extracting shallow features 3 The feature FD obtained by the third recursion when extracting deep features 3 The features are fused by point multiplication.

[0031] Furthermore, the using the preset number of reconstruction branches to reconstruct images using sub-pixel convolution kernels for each obtained fusion feature includes:

[0032] Set a corresponding reconstruction branch for each fusion feature;

[0033] For each reconstruction branch, the corresponding fusion features are used to reconstruct the image using a 1×1 convolution layer and a sub-pixel convolution layer.

[0034] Furthermore, the method further comprises:

[0035] The preset image dataset is used to train the model parameters in the image restoration process, and the ablation experiment is performed with PSNR (Peak signal-to-noise ratio) as the evaluation indicator.

[0036] In a second aspect, the present disclosure further provides a super-resolution restoration system for an image, comprising:

[0037] A preprocessing module, which is configured to process the image to be processed using a convolution kernel to obtain a low-resolution image;

[0038] A shallow feature extraction module, which is configured to use a 1×1 convolution kernel to perform dimension transformation on the low-resolution image, and use a preset number of convolution kernels with different receptive fields and adaptive learning weights to extract multi-scale and hierarchical shallow feature information from the dimensionally transformed low-resolution image;

[0039] A deep feature extraction module, configured to extract deep features from the low-resolution image using a double-layer nested residual module, and recursively perform the preset number of times when extracting deep feature information;

[0040] A fusion module is configured to fuse the features extracted by each convolution kernel of different receptive fields in extracting shallow features with the features obtained by completing the corresponding number of recursive executions in extracting deep features by dot multiplication to obtain each fused feature;

[0041] The image reconstruction module is configured to use the preset number of reconstruction branches to reconstruct images using sub-pixel convolution kernels for each fusion feature obtained, and add the results obtained by all branches to generate a final high-resolution image.

[0042] In a third aspect, the present disclosure provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the image super-resolution restoration method as described in any one of the first aspects.

[0043] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image super-resolution restoration method described in any one of the above-mentioned first aspects is implemented.

[0044] Beneficial effects:

[0045] The present invention provides a method, system, electronic device and computer-readable storage medium for super-resolution restoration of an image. The method uses a convolution kernel to process an image to be processed to obtain a low-resolution image. A 1×1 convolution kernel is used to perform dimension transformation on the low-resolution image, and a preset number of convolution kernels with different receptive fields and adaptive learning weights are used to extract multi-scale and hierarchical shallow feature information from the low-resolution image after dimension transformation. A double-layer nested residual module is used to extract deep features from the low-resolution image, and the deep feature information is extracted by recursively executing the preset number of times. The features extracted by each convolution kernel with different receptive fields when extracting shallow features are respectively fused with the features obtained by completing the corresponding number of recursive executions in extracting deep features by dot multiplication to obtain each fused feature. The preset number of reconstruction branches are used to reconstruct the image using sub-pixel convolution kernels for each fused feature obtained, and the results obtained by all branches are added together to generate a final high-resolution image. The technical solution disclosed in the present invention uses a lightweight network to improve image resolution through an improved deep learning method, and uses a nested block method to reduce parameters. Considering that shallow features contain a large amount of low-frequency information, a multi-level and multi-scale approach is used to extract shallow features, and then a channel attention shuffle module is used to improve the representation ability of the network and achieve a balance between performance and scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a process of a super-resolution restoration method of an image provided in the first embodiment of the present disclosure;

[0047] Figure 2 A schematic diagram of a shallow feature extraction process provided in the first embodiment of the present disclosure;

[0048] Figure 3 A schematic diagram of an embedded module structure provided in Embodiment 1 of the present disclosure;

[0049] Figure 4 A schematic diagram of a double-layer nested residual module structure provided in Embodiment 1 of the present disclosure;

[0050] Figure 5 A schematic diagram of a recursive process of deep feature extraction provided in the first embodiment of the present disclosure;

[0051] Figure 6 A schematic diagram of the structure of a channel scrambling module provided in the first embodiment of the present disclosure;

[0052] Figure 7 A schematic diagram of a fusion process of shallow features and deep features provided in the first embodiment of the present disclosure;

[0053] Figure 8A schematic diagram of a complete image super-resolution restoration process provided in the first embodiment of the present disclosure;

[0054] Fig. 9 An architecture diagram of an image super-resolution restoration system provided in Embodiment 2 of the present disclosure;

[0055] Fig.10 This is an architecture diagram of an electronic device provided in Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention, rather than to limit the present invention.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence; and, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.

[0058] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings.

[0059] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present disclosure, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0060] The following specific embodiments describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the technical problems of the difficulty in balancing the performance and scale of the neural network model for image restoration, the complexity of parameters, and the weak representation ability of the network. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0061] Figure 1 A schematic diagram of a process of a super-resolution restoration method for an image provided in the first embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes:

[0062] Step S101: Processing the image to be processed using a convolution kernel to obtain a low-resolution image;

[0063] Step S102: using a 1×1 convolution kernel to perform dimension conversion on the low-resolution image, and using a preset number of convolution kernels with different receptive fields and adaptive learning weights to extract multi-scale and hierarchical shallow feature information from the dimension-converted low-resolution image;

[0064] Step S103: extracting deep features from the low-resolution image by using a double-layer nested residual module, and recursively performing the preset number of times when extracting deep feature information;

[0065] Step S104: The features extracted by each convolution kernel of different receptive fields when extracting shallow features are fused with the features obtained by completing the corresponding number of recursive executions in extracting deep features by dot multiplication to obtain each fused feature;

[0066] Step S105: using the preset number of reconstruction branches, respectively reconstructing images using sub-pixel convolution kernels for each obtained fusion feature, and adding the results obtained by all branches to generate a final high-resolution image.

[0067] In order to solve the problem that the pixel points in the aerial image contain a large scene size, resulting in low image resolution, this embodiment reconstructs the image through super-resolution restoration of the image. First, an aerial image is input, and then a 3×3 convolution kernel is used to obtain a low-resolution image as the input of the network structure. The low resolution refers to the image that has not been reconstructed. The aerial image is dimensionalized using a 1×1 convolution kernel. Considering that the shallow features contain a large amount of low-frequency information, convolution kernels with different receptive fields of adaptively learned weights are used to extract multi-scale and hierarchical shallow feature information, enriching the shallow feature information. When extracting deep features, a double-layer nested residual block is used to increase the number of channels. While obtaining richer information, the strategy of first expansion and then compression is used to reduce parameters. Finally, the channel attention shuffling module is used to promote the flow of information between channels and enhance the network expression ability. Then the features extracted by each convolution kernel of different receptive fields of shallow features are fused with the features obtained by completing the corresponding number of recursive executions in extracting deep features through dot multiplication. The deep features obtained by each recursive execution are fused with shallow features of different levels extracted by different convolution kernels in the shallow features to enhance the feature's discriminative power and noise resistance. It integrates information at multiple levels to solve the problem of lack of feature invariance, and can effectively increase the descriptor's discriminative power by fusing multiple complementary features. For each fused feature obtained, reconstruction branches are set respectively, and the image is reconstructed using sub-pixel convolution kernels. The results obtained by each branch are added to generate the final high-resolution aerial image, which is the reconstructed graphic.

[0068] In one implementation of this embodiment, the dataset image adopts DIV2K (super-resolution reconstruction dataset) and is divided into a training set and a test set; the training set is bicubic (BI) down-sampled to obtain paired low-resolution and high-resolution training images, and the image is randomly rotated in the vertical and horizontal directions to enhance the data;

[0069] The training data is set up in batches, with 16 image blocks per batch, and the size is 48×48. The optimizer is the ADAM optimizer. The number of single iterations of all batches in the propagation is set to 1000. The weights are normalized during training, and the learning rate is set to 10 -3 The learning rate is decayed by half every 200 iterations. The model is implemented using PyTorch

[34] and trained on a Nvidia 2080 Ti GPU.

[0070] Using the above nested residual neural network training dataset, the compression rate of the first 1×1 convolution in the channel shuffle attention module is set to 48, the number of convolution groups is set to 3, and the number of hierarchical feature fusion blocks and multi-scale feature fusion blocks in the recursive structure is set to 3.

[0071] Five benchmark test sets are used to verify the effectiveness of the method proposed in this embodiment, and the test sets are Set5, Set14, B100, Urban100 and Manga109.

[0072] This embodiment uses a lightweight network to significantly reduce network parameters while improving image resolution through an improved deep learning method, uses a multi-level and multi-scale approach to extract shallow features, enriches feature information, uses a double-layer nested residual block approach to increase the number of channels, and while obtaining richer information, uses a strategy of first expansion and then compression to reduce parameters and enhance network expression capabilities. A good balance has been achieved in terms of performance and model scale.

[0073] Furthermore, the preset number is 3, and the convolution kernels of different receptive fields include a 1×1 convolution kernel, a 3×3 convolution kernel, and a 5×5 convolution kernel;

[0074] The method of extracting multi-scale and hierarchical shallow feature information from the low-resolution image after dimension conversion using a preset number of convolution kernels with different receptive fields and adaptive learning weights includes:

[0075] The Fs obtained by dimensionally transforming the low-resolution image using a 1×1 convolution kernel 0 Use 1×1 convolution kernel to extract shallow features to get feature Fs 1 ;

[0076] The obtained Fs 1 Multiply it with the corresponding adaptive learning weight and then add it to Fs0 After adding the points, a 3×3 convolution kernel is used to extract shallow features to obtain feature Fs 2 ;

[0077] The obtained Fs 2 Multiply it with the corresponding adaptive learning weight and then add it to Fs 0 After adding the points, use a 5×5 convolution kernel to extract shallow features to obtain feature Fs 3 .

[0078] The image to be processed is processed using the convolution kernel to obtain a low-resolution image F 0 , the process of shallow feature extraction is as follows Figure 2 As shown, it includes: using a 1×1 convolution kernel to process the low-resolution image F 0 Perform dimension conversion to obtain Fs 0 .

[0079] The obtained Fs 0 Use 1×1 convolution kernel to extract shallow features to get Fs 1 .

[0080] F s1 =H C1 (F s0 )

[0081] The obtained Fs 1 With adaptively learned weights α 1 After multiplication, add Fs 0 After adding the points, a 3×3 convolution kernel is used to extract shallow features to obtain Fs 2 ;

[0082] F s2 =H C2 (F s0 +α 1 F s1 )

[0083] The obtained Fs 2 With adaptively learned weights α 2 After multiplication, add Fs 0 After adding the points, use a 5×5 convolution kernel to get Fs 3 ;

[0084] F s3 =H C3 (F s0 +α 2 F s2 )

[0085] H C1 , H C2 and H C3They respectively represent extraction using a 1×1 convolution kernel, extraction using a 3×3 convolution kernel, and extraction using a 5×5 convolution kernel. Using convolution kernels with different receptive fields with adaptive learning weights can extract shallow feature information of multiple scales and levels, enriching the feature information.

[0086] Furthermore, the extracting deep features of the low-resolution image by using a double-layer nested residual module, and recursively performing the preset number of times when extracting deep feature information, includes:

[0087] Construct an embedded module with a convolution kernel of 1×1, which includes a channel shuffling module;

[0088] The residual module with a convolution kernel of 3×3 is used as the external module, and a channel shuffling module is added to the external module. The embedded module and the external module are then combined to form a double-layer nested residual module.

[0089] A plurality of double-layer nested residual modules are used to extract deep features from the low-resolution image, and the deep features are extracted three times in a recursive manner to obtain deep features FD 1 , FD 2 and FD 3 .

[0090] The double-layer nested residual module for deep feature extraction consists of an external module and an embedded module. First, an embedded module with a convolution kernel of 1×1 is constructed. The structure of the embedded module is as follows: Figure 3 As shown in the figure, it includes: 1×1 convolution kernel, ReLU (Rectified Linear Unit, linear rectification function), the second 1×1 convolution kernel and channel shuffling module CSAM (Channel-Spatial Attention Module). ReLU, also known as the rectified linear unit, is a commonly used activation function in artificial neural networks. It is a nonlinear function represented by the ramp function and its variants. The double-layer nested residual module structure is shown in the figure. Figure 4 As shown in Fig. 1, built-in modules and channel shuffling modules are added to the residual module with a convolution kernel of 3×3. Multiple double-layer nested residual modules are used to extract deep features to complete a recursion. Each recursion obtains the corresponding deep features. The recursive process is as follows: Figure 5 As shown, after 3 recursions, the deep feature FD is obtained. 1 , FD 2 and FD 3 .

[0091] Furthermore, the channel scrambling module is specifically used for:

[0092] First, compress the channel through a 1×1 convolution layer, then group the compressed channels through a group convolution and perform channel shuffling, then restore the number of output channels to the number of input channels through a second 1×1 convolution layer, perform global average pooling on the output result, and use the preset logistic regression function to calculate the channel weight and multiply it with the corresponding channel.

[0093] The channel scrambling module structure is as follows Figure 6 As shown in the figure, first use a 1×1 convolution layer to compress the channel to reduce parameters, then use a group convolution to group the channels, and then perform channel shuffling. After the second 1×1 convolution layer, the number of output channels is restored to the number of input channels. The global average pooling is performed on the result, and a preset logistic regression function such as the softmax function is used to calculate the channel weights and multiply them with the corresponding channels. The channel attention shuffling module can promote the flow of information between channels and enhance the network expression ability.

[0094] Furthermore, the features extracted by the convolution kernels of different receptive fields when extracting shallow features are respectively fused with the features obtained by completing the corresponding number of recursive executions in extracting deep features by means of dot multiplication, including:

[0095] The feature Fs obtained when extracting shallow features 1 The result FD obtained by the first recursion when extracting deep features 1 Fusion of features by dot product;

[0096] The feature Fs obtained when extracting shallow features 2 The feature FD obtained by the second recursion when extracting deep features 2 Fusion of features by dot product;

[0097] The feature Fs obtained when extracting shallow features 3 The feature FD obtained by the third recursion when extracting deep features 3 The features are fused by point multiplication.

[0098] The fusion process of shallow features and deep features is as follows: Figure 7 As shown, the result Fs obtained by shallow feature extraction 1 The result FD obtained by the first recursion of deep feature extraction 1 The features are fused by dot multiplication, and the fusion is performed according to the dot multiplication of their respective weight parameters. Similarly, FS 2 With FD 2 , FS 3 With FD 3 Fusion is performed to obtain fusion features including a variety of different shallow features and deep features, enriching feature parameters.

[0099] Furthermore, the using the preset number of reconstruction branches to reconstruct images using sub-pixel convolution kernels for each obtained fusion feature includes:

[0100] Set a corresponding reconstruction branch for each fusion feature;

[0101] For each reconstruction branch, the corresponding fusion features are used to reconstruct the image using a 1×1 convolution layer and a sub-pixel convolution layer.

[0102] For each fused feature, the image is reconstructed using a 1×1 convolution layer and a sub-pixel convolution layer (Sub-pixel in the figure), and multiple reconstructed images are added together to obtain a super-resolution aerial image. The generated super-resolution image contains richer semantic information and has higher clarity.

[0103] Furthermore, the method further comprises:

[0104] The preset image dataset is used to train the model parameters in the image restoration process, and the peak signal-to-noise ratio (PSNR) is used as the evaluation indicator for ablation experiments.

[0105] The preset image data set can be trained using images in the Stanford Drone Dataset (UAV video image data set), and an ablation experiment is performed using PSNR as an evaluation indicator to determine the contribution of each part and adjust parameters. Compared with other lightweight network models, the model proposed in this embodiment achieves a better balance between performance and model scale.

[0106] For easier understanding, please combine Figure 8 As shown, the super-resolution restoration process of the image includes shallow feature extraction, deep feature extraction, feature fusion and image reconstruction. The low-resolution image is dimensionalized by using a 1×1 convolution kernel, and a preset number of convolution kernels with different receptive fields with adaptive learning weights are used to extract multi-scale and hierarchical shallow feature information; deep features are extracted by using a double-layer nested residual module, and the double-layer nested residual module includes multiple channel attention scrambling modules to promote the flow of information between channels, and multiple deep features are extracted three times in a recursive manner; deep and shallow feature fusion, the features obtained by the shallow feature 1×1 convolution kernel are fused with the first recursion of the deep feature extraction step in a point-to-point manner, and the features obtained by the 3×3 convolution kernel and the 5×5 convolution kernel in the convolution kernels with different receptive fields are fused with the second and third recursions in a point-to-point manner respectively; three reconstruction branches are used to reconstruct the image using sub-pixel convolution kernels for the three fused features obtained by feature fusion, and the results obtained by the three branches are added to generate the final high-resolution aerial image.

[0107] Fig. 9The second embodiment of the present disclosure provides an architecture diagram of a super-resolution restoration system for an image, such as Fig. 9 As shown, including:

[0108] A preprocessing module 11, which is configured to process the image to be processed using a convolution kernel to obtain a low-resolution image;

[0109] A shallow feature extraction module 12 is configured to use a 1×1 convolution kernel to perform dimension conversion on the low-resolution image, and use a preset number of convolution kernels with different receptive fields and adaptive learning weights to extract multi-scale and hierarchical shallow feature information from the dimension-converted low-resolution image;

[0110] A deep feature extraction module 13, which is configured to extract deep features from the low-resolution image by using a double-layer nested residual module, and recursively execute the preset number of times when extracting deep feature information;

[0111] A fusion module 14 is configured to fuse the features extracted by each convolution kernel of different receptive fields in extracting shallow features with the features obtained by completing the corresponding number of recursive executions in extracting deep features by means of dot multiplication to obtain each fused feature;

[0112] The image reconstruction module 15 is configured to use the preset number of reconstruction branches to reconstruct images using sub-pixel convolution kernels for each fusion feature obtained, and add the results obtained by all branches to generate a final high-resolution image.

[0113] Furthermore, the preset number is 3, and the convolution kernels of different receptive fields include a 1×1 convolution kernel, a 3×3 convolution kernel, and a 5×5 convolution kernel;

[0114] The shallow feature extraction module 12 is specifically configured as follows:

[0115] The Fs obtained by dimensionally transforming the low-resolution image using a 1×1 convolution kernel 0 Use 1×1 convolution kernel to extract shallow features to get feature Fs 1 ;

[0116] The obtained Fs 1 Multiply it with the corresponding adaptive learning weight and then add it to Fs 0 After adding the points, a 3×3 convolution kernel is used to extract shallow features to obtain feature Fs 2 ;

[0117] The obtained Fs 2 Multiply it with the corresponding adaptive learning weight and then add it to Fs 0 After adding the points, use a 5×5 convolution kernel to extract shallow features to obtain feature Fs 3 .

[0118] Furthermore, the deep feature extraction module 13 is specifically configured as follows:

[0119] Construct an embedded module with a convolution kernel of 1×1, which includes a channel shuffling module;

[0120] The residual module with a convolution kernel of 3×3 is used as the external module, and a channel shuffling module is added to the external module. The embedded module and the external module are then combined to form a double-layer nested residual module.

[0121] A plurality of double-layer nested residual modules are used to extract deep features from the low-resolution image, and the deep features are extracted three times in a recursive manner to obtain deep features FD 1 , FD 2 and FD 3 .

[0122] Furthermore, the channel scrambling module is specifically used for:

[0123] First, compress the channel through a 1×1 convolution layer, then group the compressed channels through a group convolution and perform channel shuffling, then restore the number of output channels to the number of input channels through a second 1×1 convolution layer, perform global average pooling on the output result, and use the preset logistic regression function to calculate the channel weight and multiply it with the corresponding channel.

[0124] Furthermore, the fusion module 14 is specifically configured as follows:

[0125] The feature Fs obtained when extracting shallow features 1 The result FD obtained by the first recursion when extracting deep features 1 Fusion of features by dot product;

[0126] The feature Fs obtained when extracting shallow features 2 The feature FD obtained by the second recursion when extracting deep features 2 Fusion of features by dot product;

[0127] The feature Fs obtained when extracting shallow features 3 The feature FD obtained by the third recursion when extracting deep features 3 The features are fused by point multiplication.

[0128] Furthermore, the image reconstruction module 15 is specifically configured as follows:

[0129] Set a corresponding reconstruction branch for each fusion feature;

[0130] For each reconstruction branch, the corresponding fusion features are used to reconstruct the image using a 1×1 convolution layer and a sub-pixel convolution layer.

[0131] Furthermore, the system also includes a training module 16,

[0132] The training module 16 is configured to use a preset image data set to train model parameters in the image restoration process, and to perform ablation experiments using PSNR as an evaluation indicator.

[0133] The image super-resolution restoration system of the disclosed embodiment is used to implement the image super-resolution restoration method in method embodiment 1, so the description is relatively simple. For details, please refer to the relevant description in the previous method embodiment 1, which will not be repeated here.

[0134] In addition, if Fig.10 As shown, the third embodiment of the present disclosure further provides an electronic device, including a memory 10 and a processor 20, wherein the memory 10 stores a computer program, and when the processor 20 runs the computer program stored in the memory 10, the processor 20 executes the above-mentioned various possible methods.

[0135] The memory 10 is connected to the processor 20. The memory 10 may be a flash memory, a read-only memory or other memory. The processor 20 may be a central processing unit or a single-chip microcomputer.

[0136] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the above-mentioned various possible methods.

[0137] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0138] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A super-resolution restoration method for an image, characterized in that: The method comprises: The image to be processed is processed using a convolution kernel to obtain a low-resolution image; Using a 1×1 convolution kernel to perform dimension transformation on the low-resolution image, and using a preset number of convolution kernels with different receptive fields and adaptive learning weights to extract multi-scale and hierarchical shallow feature information from the low-resolution image after dimension transformation; Construct an embedded module with a convolution kernel of 1×1, which includes a channel shuffling module; The residual module with a convolution kernel of 3×3 is used as the external module, and a channel shuffling module is added to the external module. The embedded module and the external module are then combined to form a double-layer nested residual module. Extracting deep features from the low-resolution image using a double-layer nested residual module, and recursively performing the preset number of times when extracting deep feature information; The features extracted by each convolution kernel with different receptive fields when extracting shallow features are fused with the features obtained by completing the corresponding number of recursive executions in extracting deep features through dot multiplication to obtain each fused feature; Using the preset number of reconstruction branches, each fusion feature obtained is reconstructed using a sub-pixel convolution kernel, and the results obtained by all branches are added together to generate a final high-resolution image.

2. The method according to claim 1, characterized in that The preset number is 3, and the convolution kernels of different receptive fields include a 1×1 convolution kernel, a 3×3 convolution kernel, and a 5×5 convolution kernel; The method of extracting multi-scale and hierarchical shallow feature information from the low-resolution image after dimension conversion using a preset number of convolution kernels with different receptive fields and adaptive learning weights includes: The low-resolution image Fs0 obtained by dimension conversion using a 1×1 convolution kernel is subjected to shallow feature extraction using a 1×1 convolution kernel to obtain feature Fs1; The obtained Fs1 is multiplied by the corresponding adaptive learning weight, then added to Fs0, and a 3×3 convolution kernel is used to extract shallow features to obtain feature Fs2; The obtained Fs2 is multiplied by the corresponding adaptively learned weight, then added to Fs0, and a 5×5 convolution kernel is used to extract shallow features to obtain feature Fs3.

3. The method according to claim 2, characterized in that The method of extracting deep features from the low-resolution image by using a double-layer nested residual module, and recursively performing the preset number of times when extracting deep feature information, includes: A plurality of double-layer nested residual modules are used to extract deep features from the low-resolution image, and the deep features are extracted by recursively performing the operation three times to obtain deep features FD1, FD2 and FD3 respectively.

4. The method according to claim 3, characterized in that The channel scrambling module is specifically used for: First, compress the channel through a 1×1 convolution layer, then group the compressed channels through a group convolution and perform channel shuffling, then restore the number of output channels to the number of input channels through a second 1×1 convolution layer, perform global average pooling on the output result, and use the preset logistic regression function to calculate the channel weight and multiply it with the corresponding channel.

5. The method according to claim 3, characterized in that: The features extracted by each convolution kernel of different receptive fields when extracting shallow features are respectively fused with the features obtained by completing the corresponding number of recursive executions in extracting deep features by means of dot multiplication, including: The feature Fs1 obtained when extracting shallow features and the result FD1 obtained by the first recursion when extracting deep features are fused by dot multiplication; The feature Fs2 obtained when extracting shallow features and the feature FD2 obtained by the second recursion when extracting deep features are fused by dot multiplication; The feature Fs3 obtained when extracting shallow features and the feature FD3 obtained by the third recursion when extracting deep features are fused by dot multiplication.

6. The method according to claim 1, characterized in that The using the preset number of reconstruction branches to reconstruct images using sub-pixel convolution kernels for each obtained fusion feature respectively includes: Set a corresponding reconstruction branch for each fusion feature; For each reconstruction branch, the corresponding fusion features are used to reconstruct the image using a 1×1 convolution layer and a sub-pixel convolution layer.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The preset image dataset is used to train the model parameters in the image restoration process, and the peak signal-to-noise ratio (PSNR) is used as the evaluation indicator for ablation experiments.

8. A super-resolution restoration system for an image, characterized in that: include: A preprocessing module, which is configured to process the image to be processed using a convolution kernel to obtain a low-resolution image; A shallow feature extraction module, which is configured to use a 1×1 convolution kernel to perform dimension transformation on the low-resolution image, and use a preset number of convolution kernels with different receptive fields and adaptive learning weights to extract multi-scale and hierarchical shallow feature information from the dimensionally transformed low-resolution image; A deep feature extraction module configured to construct an embedded module with a convolution kernel of 1×1, including a channel shuffling module; and, The residual module with a convolution kernel of 3×3 is used as the external module, and a channel shuffling module is added to the external module. The embedded module and the external module are then combined to form a double-layer nested residual module. The deep feature extraction module is further configured to extract deep features from the low-resolution image using a double-layer nested residual module, and recursively perform the preset number of times when extracting deep feature information; A fusion module is configured to fuse the features extracted by each convolution kernel of different receptive fields in extracting shallow features with the features obtained by completing the corresponding number of recursive executions in extracting deep features by dot multiplication to obtain each fused feature; The image reconstruction module is configured to use the preset number of reconstruction branches to reconstruct images using sub-pixel convolution kernels for each fusion feature obtained, and add the results obtained by all branches to generate a final high-resolution image.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the image super-resolution restoration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for super-resolution restoration of an image according to any one of claims 1 to 7 is implemented.

Images