An image super-resolution reconstruction method based on edge guidance and bidirectional rectification residual learning

Abstract

This invention discloses an image super-resolution reconstruction method based on edge guidance and bidirectional correction residual learning, comprising: acquiring a target low-resolution image; inputting the target low-resolution image into a pre-trained image super-resolution reconstruction model, and obtaining a reconstructed high-resolution image based on the output of the image super-resolution reconstruction model; wherein the image super-resolution reconstruction model includes an edge feature extraction module, a low-resolution image processing branch and an edge information image processing branch, a fusion module and an upsampling reconstruction module; the low-resolution image processing branch and the edge information image processing branch are each composed of a shallow feature extraction module and several edge correction modules connected in series; this invention fully utilizes various features of the input image and the learning ability of residual learning to perform super-resolution reconstruction of the low-resolution image, improve the image resolution, and thus enhance the image display effect.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an image super-resolution reconstruction method based on edge guidance and bidirectional correction residual learning. Background Technology

[0002] With the continuous development of smart hardware and the internet, we are gradually entering the multimedia era. Among multimedia data, images and videos are the most frequently encountered types. In recent years, thanks to breakthroughs in key display technologies, high-definition display devices have become extremely popular. High-definition images can present more details and structure, providing a better visual experience. However, due to practical limitations, high-definition image data often cannot be directly obtained through sensors, failing to provide users with the best visual experience on high-definition display devices. Therefore, reconstructing ordinary images into high-definition images using super-resolution technology has become an indispensable step in using high-definition display devices. There are many situations where high-definition images cannot be directly obtained through sensors. For example, the size limitations of devices restrict the optical zoom adjustment capabilities of cameras, making it impossible to directly obtain high-definition images. Digital zoom, however, can adjust the image through certain algorithms to simulate optical zoom. Super-resolution algorithms, as one type of such algorithms, largely determine the magnification of digital zoom and the quality of the result. Therefore, super-resolution reconstruction is an important scientific research field closely related to our lives.

[0003] Super-resolution (SR) reconstruction uses software technology to reconstruct existing low-resolution images, improving image resolution while restoring lost high-frequency details. It is characterized by low cost and high practicality, and has become a research hotspot in the field of image processing.

[0004] Deep neural networks have achieved good results in image super-resolution tasks. Dong C et al. first proposed a three-layer CNN model SRCNN (Dong C, Loy CC, He K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Trans. Pattern Anal. Mach. Intell, 2016, 38(2): 295-307.) to learn the end-to-end mapping from LR images to HR images. Compared with traditional feature engineering-based methods, the application of CNN has greatly improved the reconstruction effect. However, the disadvantage is that the model has a very shallow number of layers and cannot fully utilize the learning ability of the model. To address this shortcoming, Kim J et al. proposed the VDSR model (Kim J, Lee JK, Lee KM. Accurate image super-resolution using very deep convolutional networks[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June 27-30, 2016, 1646-1654.). VDSR has a 20-layer deep network, which solves the problem of network convergence difficulty. However, the above methods also have certain drawbacks, namely that most operations are performed in a high-dimensional space, resulting in high time and space costs. In order to solve the computational efficiency problem and make full use of deep learning technology to automatically improve image resolution, Dong C et al. proposed the FSRCNN framework (Dong C, Loy CC, Tang.X. Accelerating the super-resolution convolutional neural network[C]. Proceedings of the Computer Vision-ECCV-14th European Conference, Part II, 2016, 391-407.), which constrains convolution to a low-dimensional space.Compared to SRCNN, FSRCNN achieves real-time performance, but its complexity in feature processing remains high. To address this issue, Shi W et al. proposed the ESPCN framework (Shi W, Caballero J, Huszar F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June 27-30, 2016, 1874-1883.), which aggregates low-dimensional feature maps while projecting onto a high-dimensional space to reconstruct HR images. Existing methods do not fully utilize the input image and fail to take full advantage of various features of image details, which have great potential to help in reconstructing HR images. Summary of the Invention

[0005] The purpose of this invention is to provide an image super-resolution reconstruction method based on edge guidance and bidirectional correction residual learning. This method fully utilizes various features of the input image and the learning ability of residual learning to perform super-resolution reconstruction on low-resolution images, thereby improving the image resolution and enhancing the image display effect.

[0006] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] Firstly, an image super-resolution reconstruction method based on edge guidance and bidirectional corrected residual learning is provided, including:

[0008] Acquire a low-resolution image of the target;

[0009] The target low-resolution image is input into a pre-trained image super-resolution reconstruction model, and the reconstructed high-resolution image is obtained based on the output of the image super-resolution reconstruction model.

[0010] The image super-resolution reconstruction model includes an edge feature extraction module, a low-resolution image processing branch and an edge information image processing branch, a fusion module and an upsampling reconstruction module; the low-resolution image processing branch and the edge information image processing branch are each composed of a shallow feature extraction module and several edge correction modules connected in series.

[0011] The processing steps of the image super-resolution reconstruction model include: using an edge feature extraction module to extract edge information from a low-resolution image X to obtain an edge information image G; using the low-resolution image X as the input of a low-resolution image processing branch and the edge information image G as the input of an edge information image processing branch, and extracting the first shallow feature F0 and the second shallow feature G0 respectively through a shallow feature extraction module; fusing and interacting the information of the two branches in each edge correction module; adding the output of the two-branch fusion module Fusion_Block in the last edge correction module with the output of the corresponding shallow feature extraction module, and finally performing feature fusion through a fusion module; the fusion result is then processed through an upsampling reconstruction module to obtain the reconstructed high-resolution image.

[0012] In some embodiments, the processing procedure of the image super-resolution reconstruction model specifically includes: taking a low-resolution image X as input to a low-resolution image processing branch, extracting a first shallow feature F0 from the low-resolution image X using a first shallow feature extraction module; and extracting the first deep feature F0 from the shallow feature F0 using a first deep feature extraction module RCA_Block. R The first RCA_Block contains several RCA convolutional layers, each including a residual block and a CA block. Channel attention is implemented using the residual and CA blocks. After stacking several RCA convolutional layers, a regular convolutional layer is added, along with a short connection. The output of this convolutional layer is then summed element-wise with the input F0 of the first RCA_Block to obtain the output F of the first RCA_Block. R ;

[0013] The edge information image G is used as the input to the edge information image processing branch. The second shallow feature extraction module is used to extract the second shallow feature G0 of the edge information image G. The second RCA_Block is used to extract the first deep feature G from the shallow feature G0. R The second RCA_Block contains several RCA convolutional layers, each including a residual block and a CA block. Channel attention is implemented using the residual and CA blocks. After stacking several RCA convolutional layers, a regular convolutional layer is added, along with a short connection. The output of this convolutional layer is then summed element-wise with the input G0 of the second RCA_Block to obtain the output G of the first RCA_Block. R ;

[0014] After obtaining the deep features of each of the two branches, deep feature fusion is then performed.

[0015] First, feature fusion is performed on the low-resolution image branches; Fusion_Block consists of a cat(·) and several RCA convolutional layers; F R With G R In the input Fusion_Block, the channel dimensions are concatenated using the cat() function. Thus, the two input shapes of Fusion_Block are [16, 64, 48, 48], and the concatenated shape is [16, 128, 48, 48]. This method fuses deep features from edge information image branches in the low-resolution image branch. The fused features are then input into several RCA convolutional layers. Unlike RCA_Block, the number of input and output feature channels in the RCA convolutional layers of Fusion_Block is twice that of RCA_Block, allowing for better fusion of deep features from both parts. A separate convolutional layer is added at the end of Fusion_Block, and the output is used for the next stage of deep feature processing.

[0016] At the same time, the edge information image branch also performs feature fusion; the input of the Fusion_Block of this branch is first the F R The process involves re-extracting the edge information of the features, and then comparing the extracted results with G. R The cat(·) function is used to concatenate the channels, and then the input is Fusion_Block for feature fusion. Finally, a separate convolutional layer is added, and the output is used for the next round of processing. Through these two feature fusions, a bidirectional correction mechanism is formed.

[0017] The output of each branch in its final stage is F. FDM With F FDMG Then, a long connection is formed between each of the two outputs to fully utilize the features of the input image. The shallow features and deep features are added element-wise. Next, the two long-connected features are input into the final Fusion_Block for feature fusion to obtain the final fused features. The final fused features are then input into the upsampling layer for upsampling. The upsampling parameters are set according to the specified magnification factor. The upsampling result is then input into the reconstruction layer, which is a simple convolutional layer. After reconstruction is completed, the output of the entire model is obtained, which is the super-resolution reconstructed image.

[0018] Experiments show that, compared to the method of fusing edge information image features into low-resolution image branches alone, the super-resolution reconstruction with a bidirectional correction mechanism is more effective.

[0019] In some embodiments, both the low-resolution image processing branch and the edge information image processing branch include four edge correction modules.

[0020] In some embodiments, the edge feature extraction module extracts the edge information of the image using a gradient value algorithm or a neighborhood maximum pixel difference algorithm.

[0021] In some embodiments, the shallow feature extraction module is structured as a single convolutional layer, let H... SF (·) represents a shallow feature extraction operator. In the low-resolution image processing branch, the input is image X and the output is F0; in the edge information image processing branch, the input is edge information image G and the output is G0.

[0022] F0 = H SF (X)

[0023] G0 = H SF (G).

[0024] In some embodiments, in an edge correction module, each of the low-resolution image processing branch and the edge information image processing branch has an RCA_Block concatenated with a Fusion_Block. Furthermore, the output of the RCA_Block in the low-resolution image processing branch is simultaneously fed into the Fusion_Block in the edge information image processing branch after edge feature extraction by the edge feature extraction module for feature fusion. Conversely, the output of the RCA_Block in the edge information image processing branch is simultaneously fed into the Fusion_Block in the low-resolution image processing branch for feature fusion. The inputs to the two branches' RCA_Blocks in the first stage are F0 and G0, respectively, and the outputs of the two branches' Fusion_Blocks in each stage serve as the inputs to the two branches' RCA_Blocks in the next edge correction module.

[0025] In some embodiments, the RCA_Block structure is a combination of several RCA convolutional layers, and short links are used to fuse feature information from different levels to learn more residual information; a total of l RCA convolutional layers are set. Representing the i-th RCA convolutional layer, RCA_Block is shown in the following formula:

[0026]

[0027] A channel attention (CA) module is added to the RCA convolutional layer. The CA block generates different channel attentions requiring a constant weight z that reflects the importance of different channels, as well as a mechanism to assign the weights to the channel feature information. The constant weight z is implemented through average pooling: assuming the input feature map is X = [x1, ..., x...] c ,···,xC ,], where c∈{0,···,C-1} is the c-th input channel, and C is the total number of input channels. Then, average pooling is performed on it, and the result of the pooling is: It is a one-dimensional tensor with length C, as shown in the following equation:

[0028]

[0029] Where H and W are the height and width of the input feature map, H GP (·) represents global average pooling; the weight allocation mechanism is implemented through CNN gating, using the sigmoid activation function as a switch, which is multiplied with the input, as shown in the following equation:

[0030] s = ReLU(W U δ(W D z))

[0031] Where δ(·) represents the sigmoid gate; W D It performs channel downsampling on the result of global average pooling, and then uses convolution to restore the output channels to their original state. W U It involves upsampling the channels, transforming the output channel into r times the input channel through convolution.

[0032] Combining the CA block with a residual block consisting of conv-relu-conv results in a channel attention RCA convolutional layer, as shown in the following equation:

[0033] F R =F R-1 +H CA (H Residual (F R-1 ))

[0034] Where F R-1 H is the output of the previous RCA convolutional layer. CA (·) represents the channel attention module, H Residual (·) represents a residual block.

[0035] In some embodiments, the fusion module Fusion_Block consists of a cat(·) layer, several RCA convolutional layers, and a convolutional layer connected in series. First, the two input feature maps are concatenated through the cat(·) layer, then processed and fused in several RCA convolutional layers, and finally, the input convolutional layer performs the final feature fusion, as shown in the following equation:

[0036] F FDM =Conv(H RCAs (cat(F DF ,FDFG )))

[0037] Here, cat(·) represents concatenating the input along the channel dimension, H RCAs (·) represents several cascaded RCA convolutional layers, and Conv(·) represents the last convolutional layer;

[0038] The deep features of the low-resolution image are fused with the deep features of the edge image by the fusion module to correct the deep features of the low-resolution image. Similarly, for the edge image branch, the information of the low-resolution image branch is added to correct the deep features of the edge image, forming a two-way correction effect.

[0039] In some embodiments, the upsampling reconstruction module includes an upsampling layer and a reconstruction layer; wherein the upsampling layer is a sub-pixel convolutional layer, and H is set... UP (·) is the upsampling operator, as shown in the following equation:

[0040] F UP =H UP (F FDM )

[0041] Wherein, input F FDM This represents the output of the last feature fusion layer;

[0042] The reconstruction layer uses a simple convolutional layer to upsample the output F. UP Make final adjustments and optimizations, and set H. REC (·) represents this process, as shown in the following equation:

[0043] Y = H REC (F UP )

[0044] Where Y is the output of the reconstruction layer, i.e. the final reconstructed high-resolution image.

[0045] In some embodiments, the training process of the image super-resolution reconstruction model includes:

[0046] The HR images are degraded to obtain the corresponding LR images, and the training set {X} is constructed accordingly. i ,Y i}, China X i For LR images, Y i For the corresponding HR image, the subscript i indicates the i-th one in the LR or HR image;

[0047] The image super-resolution reconstruction model is trained using the training set. The ADAM optimization method and backpropagation are used to optimize and adjust the parameters in the model until the preset conditions are met, thus obtaining a trained image super-resolution reconstruction model.

[0048] Furthermore, the degradation treatment includes:

[0049] X=(Y*k)↓ s

[0050] Where X is the LR image, Y is the HR image, * represents the convolution operation, k is the blur kernel, ↓ s This indicates a downsampling by a factor of s.

[0051] In some embodiments, during training, the L1 loss of both the high-resolution image and the super-resolution reconstructed image is used as the model's loss function.

[0052] In a second aspect, the present invention provides an image super-resolution reconstruction device based on edge guidance and bidirectional correction residual learning, including a processor and a storage medium;

[0053] The storage medium is used to store instructions;

[0054] The processor is configured to operate according to the instructions to perform the steps of the method according to the first aspect.

[0055] Thirdly, the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0056] Advantages of this invention:

[0057] 1. This invention fully utilizes the feature information of the input low-resolution image to separately reconstruct the high-frequency information, which is the most critical for super-resolution reconstruction, and focuses the recovery on the high-frequency information, thus effectively improving the quality of image reconstruction.

[0058] 2. Due to the limited performance of single-branch networks, this invention utilizes a dual-branch network to reconstruct the high-frequency information of the image using another network. During the reconstruction process, the two branch networks combine with each other to obtain deep feature information, forming a bidirectional correction effect and improving the reconstruction quality.

[0059] 3. This invention integrates a Residual Network and a Channel Attention Module into the core module of the main network, and introduces edge detail branches and bidirectional correction ideas, enabling the model to learn richer high-frequency features and have stronger learning capabilities. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the image super-resolution reconstruction method of the present invention;

[0061] Figure 2 This is a schematic diagram of the RCA_Block network structure in an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the Fusion_Block network structure in an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the original high-resolution image in an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of the image obtained after processing by the Bicubic method in an embodiment of the present invention;

[0065] Figure 6 This is a schematic diagram of the image obtained after EDSR processing in an embodiment of the present invention;

[0066] Figure 7 This is a schematic diagram of the image obtained after D-DBPN processing in an embodiment of the present invention;

[0067] Figure 8 This is a schematic diagram of the image obtained after SRResNet processing in an embodiment of the present invention;

[0068] Figure 9 This is a schematic diagram of the image obtained after processing by the image super-resolution reconstruction method of the present invention. Detailed Implementation

[0069] The technical solutions in the embodiments of the invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some examples of the present invention, and not all examples. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention.

[0070] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0071] In the description of this invention, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0072] Example 1

[0073] As shown in Title 1, an image super-resolution reconstruction method based on edge guidance and bidirectional corrected residual learning includes:

[0074] Acquire a low-resolution image of the target;

[0075] The target low-resolution image is input into a pre-trained image super-resolution reconstruction model, and the reconstructed high-resolution image is obtained based on the output of the image super-resolution reconstruction model.

[0076] The image super-resolution reconstruction model includes an edge feature extraction module, a low-resolution image processing branch and an edge information image processing branch, a fusion module and an upsampling reconstruction module; the low-resolution image processing branch and the edge information image processing branch are each composed of a shallow feature extraction module and several edge correction modules connected in series.

[0077] The processing steps of the image super-resolution reconstruction model include: using an edge feature extraction module to extract edge information from a low-resolution image X to obtain an edge information image G; using the low-resolution image X as the input of a low-resolution image processing branch and the edge information image G as the input of an edge information image processing branch, and extracting the first shallow feature F0 and the second shallow feature G0 respectively through a shallow feature extraction module; fusing and interacting the information of the two branches in each edge correction module; adding the output of the two-branch fusion module Fusion_Block in the last edge correction module with the output of the corresponding shallow feature extraction module, and finally performing feature fusion through a fusion module; the fusion result is then processed through an upsampling reconstruction module to obtain the reconstructed high-resolution image.

[0078] In some embodiments, the edge feature extraction module uses two methods:

[0079] 1. Gradient value:

[0080] G = Gradient(X) (1)

[0081] Where X is the LR image, Gradient(·) is the function to calculate the gradient of the target value, and G is the edge information image containing the gradient features of X.

[0082] 2. Maximum neighbor difference (MNB): The maximum pixel difference in the neighborhood.

[0083]

[0084] Where N8((i,j)) is the set of the eight pixel values ​​surrounding position (i,j) in the LR image X. MND(X(i,j)) represents the maximum difference between the pixel value at (i,j) and its eight surrounding pixel values. Applying this algorithm to all pixels of image X extracts the edge information of the input image, resulting in an image G of the same size as the input image X, containing the edge information of the original image.

[0085] Furthermore, a deep learning network model is constructed, consisting of two processing branches: a low-resolution image processing branch and an edge information image processing branch. Both branches comprise a shallow feature extraction module and several edge correction modules connected in series. The input to the low-resolution image processing branch is the LR image X to be reconstructed, while the input to the edge information image processing branch is the output G of the LR image after the edge feature extraction module. In each edge correction module, the information from the two branches is fused and interacted. In the final edge correction module, the outputs of the two branches (Fusion_Block) are added to the outputs of the corresponding shallow feature extraction modules, and then fused through a Fusion_Block. The fused result is then passed through an upsampling reconstruction module to obtain the final reconstruction result.

[0086] Furthermore, the shallow feature extraction module (Head_conv) is structured as a single convolutional layer, let H... SF (·) represents the shallow feature extraction operator. In the low-resolution image processing branch, the input is image X, and the output is F0. In the edge information image processing branch, the input is edge information image G, and the output is G0, as shown in formulas (3) and (4):

[0087] F0 = H SF (X) (3)

[0088] G0 = H SF (G) (4)

[0089] Furthermore, in an edge correction module, each of the low-resolution image processing branch and the edge information image processing branch has an RCA_Block concatenated with a Fusion_Block. Additionally, the output of the RCA_Block in the low-resolution image processing branch is simultaneously fed into the Fusion_Block in the edge information image processing branch after edge feature extraction by the edge feature extraction module for feature fusion. Conversely, the output of the RCA_Block in the edge information image processing branch is simultaneously fed into the Fusion_Block in the low-resolution image processing branch for feature fusion. The inputs to the two RCA_Blocks in the first stage are F0 and G0, respectively, and the outputs of the Fusion_Blocks in each stage serve as the inputs to the two RCA_Blocks in the next edge correction module.

[0090] Furthermore, the RCA_Block structure is a combination of several RCA convolutional layers, and short connections are used to fuse feature information from different levels to learn more residual information. It is set that there are a total of l RCA convolutional layers. Let RCA_Block represent the i-th RCA convolutional layer, as shown in equation (5):

[0091]

[0092] In SR models such as VDSR and EDSR, feature maps for different channels are processed uniformly without different operations for each channel. However, each channel actually contains different feature information, and channel attention mechanisms can allow the network to give more attention to certain channels. Therefore, introducing a channel attention mechanism is essential. This invention adds a channel attention module (CA) to the RCA convolutional layer. The CA block needs a constant weight z that reflects the importance of different channels and a mechanism to assign the weights to the channel feature information to generate different channel attention. The constant weight z is implemented through average pooling: assuming the input feature map is X = [x1, ..., x...] c ,···,x C ,], where c∈{0,···,C-1} is the c-th input channel, and C is the total number of input channels. Then, average pooling is performed on it, and the result of the pooling is: It is a one-dimensional tensor with length C, as shown in formula (6):

[0093]

[0094] Where H and W are the height and width of the input feature map, H GP(·) represents global average pooling. The weight allocation mechanism can be implemented through CNN gating, using the sigmoid activation function as a switch, and combining it with the input in a multiplicative manner, as shown in formula (7):

[0095] s = ReLU(W U δ(W D z)) (7)

[0096] Where δ(·) represents the sigmoid gate; W D It performs channel downsampling on the result of global average pooling, and then uses convolution to restore the output channels to their original state. W U It performs channel upsampling, transforming the output channel into r times the size of the input channel through convolution.

[0097] The CA block is combined with a residual block consisting of conv-relu-conv to obtain the channel attention RCA convolutional layer, as shown in Equation (8):

[0098] F R =F R-1 +H CA (H Residual (F R-1 (8)

[0099] Where F R-1 H is the output of the previous RCA convolutional layer. CA (·) represents the channel attention module, H Residual (·) represents a residual block.

[0100] Furthermore, Fusion_Block consists of a cat(·) layer, several RCA convolutional layers, and a concatenated convolutional layer. First, the two input feature maps are concatenated through the cat(·) layer, then processed and fused in several RCA convolutional layers, and finally fed into a convolutional layer for final feature fusion, as shown in formula (9).

[0101] F FDM =Conv(H RCAs (cat(F DF ,F DFG ))) (9)

[0102] Here, cat(·) represents concatenating the input along the channel dimension, H RCAs(·) represents several cascaded RCA convolutional layers, and Conv(·) represents the final convolutional layer. This module fuses the deep features of the low-resolution image with the deep features of the edge image to correct the deep features of the low-resolution image; similarly, for the edge image branch, the information of the low-resolution image branch is added to correct the deep features of the edge image, forming a bidirectional correction effect.

[0103] Furthermore, the upsampling reconstruction module comprises two parts: an upsampling layer and a reconstruction layer. The upsampling layer uses a sub-pixel convolutional layer, with H set... UP (·) is the upsampling operator, as shown in formula (10):

[0104] F UP =H UP (F FDM (10)

[0105] Wherein, input F FDM This represents the output of the last feature fusion layer.

[0106] Furthermore, the reconstruction layer uses a simple convolutional layer (Tail_conv) to upsample the output F. UP Make final adjustments and optimizations, and set H. REC (·) represents this process, as shown in formula (11):

[0107] Y = H REC (F UP (11)

[0108] Where Y is the output of the reconstruction layer, i.e. the final reconstructed high-resolution image.

[0109] In some embodiments, an image super-resolution reconstruction method based on edge guidance and bidirectional corrected residual learning includes the following steps:

[0110] Step 1: Prepare input data

[0111] Obtain a public dataset for training and testing; in this embodiment, the DIV2K dataset is used, which contains 800 images for training; the HR images are degraded to obtain the corresponding LR images, thereby constructing the training set {X}. i ,Y i}, China X i For LR images, Y i For the corresponding HR image, the subscript i indicates the i-th one in the LR or HR image;

[0112] Furthermore, the degradation model formula for the degradation treatment is: X=(Y*k)↓ s

[0113] Where X is the LR image, Y is the HR image, * represents the convolution operation, k is the blur kernel, ↓ s This indicates a downsampling factor of s. In this invention, the downsampling factor s is proposed to be 2, 3, or 4 times, and the downsampling method is proposed to be bicubic downsampling.

[0114] In this embodiment, the HR image and LR image are segmented into image blocks of size 48×48, and the segmented images are aggregated to obtain the training set; when training the network using the training set, the number of batch samples N is set to 16.

[0115] Step 2: Edge Feature Extraction

[0116] This embodiment employs a method that extracts the input gradient values. The edge feature extraction module extracts the edge information of the image while ignoring smooth regions.

[0117] Step 3: Shallow Feature Extraction

[0118] This invention uses a simple convolutional layer to extract shallow feature information from the input image X and the edge information image G, such as... Figure 1 As shown. The head_conv layer of the upper branch has 3 input channels, representing the three channels of the RGB image; the head_conv layer of the lower branch has 1 input channel, representing the extracted image edge information. The output feature channels of both branches are set to 64, and the convolution kernel size is 3×3. This module obtains the shallow features F0 and G0 of the two inputs.

[0119] Step 4: Edge Correction Module

[0120] The edge correction module consists of an RCA_Block cascaded with a Fusion_Block. The shallow features F0 and G0 obtained from the head_conv layer are fed into their respective branches, such as... Figure 2The RCA_Block deep feature extraction module shown contains several RCA convolutional layers, each including a residual block and a CA block. Channel attention is implemented using the residual and CA blocks. After stacking several RCA convolutional layers, a regular convolutional layer is added at the end of the RCA_Block, with 64 channels, a 3×3 kernel size, and a short connection. The output of this convolutional layer is element-wise summed with the input of the entire module to obtain the output of the entire module. This final convolutional layer and short connection stabilize deep features, allowing the network to continue learning even in deep convolutional layers. In this invention, the number of RCA convolutional layers is set to 10, ensuring optimal network performance. This module allows the deep feature outputs F0 and G0 on their respective branches to be obtained. R With G R Where R represents the R-th RCA_Block in the network. After obtaining the deep features of each of the two branches, deep feature fusion is performed. First, feature fusion is performed on the low-resolution image branch. The Fusion_Block consists of a cat(·) and several RCA convolutional layers; F R With G R In the Fusion_Block input, the channel dimensions are concatenated using the cat() function. The two input shapes of Fusion_Block are [16, 64, 48, 48], and the concatenated shape is [16, 128, 48, 48]. This method fuses deep features from the edge information image branch in the low-resolution image branch. The fused features are then input into several RCA convolutional layers. Unlike RCA_Block, the RCA convolutional layers in Fusion_Block have 128 input and output feature channels, twice that of RCA_Block, allowing for better fusion of the deep features from both parts. A separate convolutional layer with 128 input channels and 64 output channels is added at the end of Fusion_Block, allowing the output of Fusion_Block to continue processing deep features in the next stage. Simultaneously, feature fusion is also performed in the edge information image branch. The input of the Fusion_Block in this branch is not directly concatenated with the F... R With G R Instead, first put F R The process involves re-extracting the edge information of the features, and then comparing the extracted results with G. RThe `cat()` function is used for channel concatenation, followed by feature fusion via Fusion_Block. A separate convolutional layer is then added at the end, with 128 input channels and 64 output channels. The output is then used for the next round of processing. These two feature fusions create a bidirectional correction mechanism. Experiments show that the bidirectional correction mechanism achieves better super-resolution reconstruction results compared to fusing edge information image features into low-resolution image branches alone.

[0121] Step 5: Iterative Model Correction

[0122] The number of edge correction modules in this invention is set to 4. Experiments show that the reconstruction effect of the network with 4 edge correction modules is better than that with 2, 6, 8, and 10 modules.

[0123] Step Six: Upsampling Reconstruction Module

[0124] Assume that the output of each branch in its final stage is F. FDM With F FDMG Then, a long connection is formed between each of the two outputs to fully utilize the features of the input image, that is, the shallow features and deep features are added element-wise. Next, these two long-connected features are input into the final Fusion_Block for feature fusion to obtain the final fused features. The final fused features are then input into an upsampling layer for upsampling, with upsampling parameters set according to a specified magnification factor. The upsampling result is then input into the reconstruction layer, a simple convolutional layer with 64 input feature channels and 3 output channels, representing an RGB three-channel image. After reconstruction, the output of the entire network, i.e., the super-resolution reconstructed image, is obtained.

[0125] In this embodiment, when training the deep neural network for image super-resolution reconstruction based on edge-guided and bidirectional corrected residual learning, the ADAM optimization method and backpropagation are used to optimize and adjust the parameters in the network. A total of 1000 epochs are set, with an initial learning rate of 10. -4 The learning rate is halved every 200 epochs, and the L1 loss of both the high-resolution image and the super-resolution reconstructed image is used as the loss function of the network.

[0126] The following is a combination of Table 1 and... Figures 4 to 9 The present invention will be further illustrated through the effect evaluation of the embodiments.

[0127] In this embodiment, Peak Signal-to-Noise Ratio (PSNR), Structure Similarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) are used as objective evaluation metrics for the results. Lower LPIPS values ​​indicate higher perceptual quality.

[0128] In this embodiment, images from the Urban100 dataset are used as test images. Table 1 lists the average PSNR, SSIM, and LPIPS of the resulting images obtained by this embodiment and other advanced image super-resolution methods. As can be seen from Table 1, the results obtained by the method in this embodiment are significantly improved compared with the results of other methods.

[0129] Table 1

[0130]

[0131] Figures 4 to 9 The images shown are an original high-resolution (HR) image and super-resolution images obtained by processing with Bicubic, EDSR, D-DBPN, SRResNet, and the method of this embodiment. The images obtained by EDSR, D-DBPN, and SRResNet are generally clear, but some local areas exhibit blurred details. The image obtained by the method of this embodiment has richer high-frequency details. Both the visual effect and objective evaluation metrics of the results from this embodiment demonstrate that the method of this invention can achieve better image super-resolution reconstruction quality while maintaining network interpretability.

[0132] Example 2

[0133] Secondly, this embodiment provides an image super-resolution reconstruction device based on edge guidance and bidirectional correction residual learning, including a processor and a storage medium;

[0134] The storage medium is used to store instructions;

[0135] The processor is configured to operate according to the instructions to perform the steps of the method according to Embodiment 1.

[0136] Example 3

[0137] Thirdly, this embodiment provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0138] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0140] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0142] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones. All modifications within the scope of this invention or its equivalents are included in this invention.

Claims

1. An image super-resolution reconstruction method based on edge guidance and bidirectional rectified residual learning, characterized in that, include: Acquire a low-resolution image of the target; The target low-resolution image is input into a pre-trained image super-resolution reconstruction model, and the reconstructed high-resolution image is obtained based on the output of the image super-resolution reconstruction model. The image super-resolution reconstruction model includes an edge feature extraction module, a low-resolution image processing branch and an edge information image processing branch, a fusion module and an upsampling reconstruction module; the low-resolution image processing branch and the edge information image processing branch are each composed of a shallow feature extraction module and several edge correction modules connected in series. The image super-resolution reconstruction model's processing steps include: using an edge feature extraction module to process low-resolution images. Extracting edge information from an image yields an edge information image. ; to low-resolution images As input to the low-resolution image processing branch, the edge information image is used. As input to the edge information image processing branch, the first shallow features are extracted by the shallow feature extraction module. Second shallow features In each edge correction module, the information of the two branches is fused and interacted. In the last edge correction module, the output of the two branch fusion module Fusion_Block is added to the output of the corresponding shallow feature extraction module. Finally, a fusion module is used for feature fusion. The fusion result is then passed through an upsampling reconstruction module to obtain the reconstructed high-resolution image. The shallow feature extraction module has a single convolutional layer, let This represents a shallow feature extraction operator, where the input in the low-resolution image processing branch is an image. The output is The input in the edge information image processing branch is an edge information image. The output is ; ； ； In an edge correction module, each of the low-resolution image processing branch and the edge information image processing branch has one RCA_Block concatenated with a Fusion_Block. Furthermore, the output of the RCA_Block in the low-resolution image processing branch is simultaneously fed into the Fusion_Block in the edge information image processing branch after edge feature extraction by the edge feature extraction module for feature fusion. Conversely, the output of the RCA_Block in the edge information image processing branch is simultaneously fed into the Fusion_Block in the low-resolution image processing branch for feature fusion. The inputs to the two RCA_Blocks in the first stage are... and The outputs of the two branches of Fusion_Block in each stage are used as the inputs of the two branches of RCA_Block in the next edge correction module. The RCA_Block structure is a combination of several RCA convolutional layers, and short links are used to fuse feature information from different levels to learn more residual information; a common... One RCA convolutional layer, Representing the A number of RCA convolutional layers, RCA_Block is shown in the following formula: ； A channel attention (CA) module is added to the RCA convolutional layer. The CA block generates different channel attentions requiring a constant weight z that reflects the importance of different channels, as well as a mechanism to assign the weights to the channel feature information. The constant weight z is implemented through average pooling: assuming the input feature map is... ,in It is the first One input channel, The input is the total number of channels, which is then subjected to average pooling. The result of the pooling is... It is a one-dimensional tensor with length . As shown in the following formula: ； where H, W are the height and width of the input feature map, is the global average pooling; the weight distribution mechanism is realized through the CNN gate, using the activation function sigmoid as the switch, combined with the input in a multiplicative manner, as shown in the following formula: ； in Represents a sigmoid gate; It performs channel downsampling on the result of global average pooling, and then uses convolution to restore the output channels to their original state. ; It involves upsampling the channels, transforming the output channel into r times the input channel through convolution. Combining the CA block with a residual block consisting of conv-relu-conv results in a channel attention RCA convolutional layer, as shown in the following equation: ； in This is the output of the previous RCA convolutional layer. Represents the channel attention module. Represents the residual block.

2. The method according to claim 1, characterized in that, The edge feature extraction module extracts edge information of the image using either a gradient value algorithm or a neighborhood maximum pixel difference algorithm.

3. The method of claim 1, wherein, The fusion module Fusion_Block consists of one The layer consists of several RCA convolutional layers and one convolutional layer connected in series; first, the two input feature maps are processed... The layers are concatenated, and the input features are processed and fused in several RCA convolutional layers. Finally, the input features are fused into a convolutional layer, as shown in the following formula: ； in, This indicates that the input is concatenated along the channel dimension. Represents several cascaded RCA convolutional layers. This represents the final convolutional layer; The deep features of the low-resolution image are fused with the deep features of the edge image by the fusion module to correct the deep features of the low-resolution image. Similarly, for the edge image branch, the information of the low-resolution image branch is added to correct the deep features of the edge image, forming a two-way correction effect.

4. The method of claim 1, wherein, The upsampling reconstruction module consists of an upsampling layer and a reconstruction layer; the upsampling layer uses a sub-pixel convolutional layer, which is set... The upsampling operator is shown in the following equation: ； wherein the input represents the output of the last feature fusion layer; The reconstruction layer uses a simple convolutional layer to upsample the output. Make final adjustments and optimizations, and set... This process is shown in the following formula: ； Where Y is the output of the reconstruction layer, i.e. the final reconstructed high-resolution image.

5. The method of claim 1, wherein, The training process of the image super-resolution reconstruction model includes: The HR image is degraded to obtain a corresponding LR image, so as to construct a training set , middle is the LR image, is the corresponding HR image, and a subscript indicates the i-th in the LR or HR image ; The image super-resolution reconstruction model is trained using the training set. The ADAM optimization method and backpropagation are used to optimize and adjust the parameters in the model until the preset conditions are met, thus obtaining a trained image super-resolution reconstruction model.

6. The method of claim 5, wherein, The degradation treatment includes: ； wherein, is an LR image, is an HR image, is a convolution operation, is a blur kernel, denotes a downsampling by a factor of 4.

7. The method of claim 5, wherein, During training, the L1 loss of both the high-resolution image and the super-resolution reconstructed image is used as the loss function of the model.

8. An image super-resolution reconstruction device based on edge guidance and bidirectional rectified residual learning, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 7.

Images