A deep residual super-resolution reconstruction method based on feature sharing
By introducing source jump connections to share shallow features in deep residual networks, the problem of insufficient utilization of shallow features is solved, thereby improving image reconstruction quality and training efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-09
- Publication Date
- 2026-03-31
AI Technical Summary
The existing residual block stacking method has the problem of insufficient utilization of shallow features, resulting in the loss of shallow information in the network and affecting the image reconstruction effect.
By introducing multiple source hop connections, shallow features can be directly shared to intermediate layers, making full use of shallow feature information such as image edges and corners, and constructing a deep residual super-resolution reconstruction network based on feature sharing.
It improves the expressive power of deep features, obtains higher quality high-resolution reconstructed images, alleviates the problem of shallow information loss during propagation, and reduces the training difficulty.
Smart Images
Figure CN115908129B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to a depth residual super-resolution reconstruction method based on feature sharing. Background Technology
[0002] For various reasons, the images we obtain are often blurry, low-quality, and low-resolution, while our actual need is for clear, high-quality, and high-resolution images. Single-image super-resolution reconstruction is a method to solve this problem at the software level through post-processing. Single-image super-resolution reconstruction refers to mapping a given degraded single low-resolution image (LR) into a clear, visually natural high-resolution image (HR). Since the rise and rapid development of deep learning, a large number of single-image super-resolution methods have emerged. In the early days, mainly due to limitations in computing power, the network structure was relatively simple and shallow, generally with only a few convolutional layers. Later, on the one hand, technological advancements greatly improved computing power; on the other hand, after Kim et al. proposed a VDSR (Accurate Image Super-Resolution Using Very Deep Convolutional Networks) with 20 convolutional layers and achieved significant results, researchers found that the intuition that network depth is crucial to performance is still correct. Later, Kaiming He proposed ResNet, which constructs residual blocks. By stacking these residual blocks, the network depth can reach hundreds of layers, solving the network degradation problem caused by increasing network depth and achieving significant results. This further illustrates the importance of network depth. Therefore, most subsequent methods focused on designing deeper networks. However, simply stacking residual blocks to increase depth did not yield significant results. To address this issue, researchers improved the stacking method of residual blocks, forming a residual group from several residual blocks. This residual group served as the basic unit of the network, and the addition of long-skip and short-skip connections simplified the training process. These improvements significantly enhanced the reconstruction results. However, these methods still have some shortcomings.
[0003] As network depth increases, the receptive field of convolutional layers also increases, allowing them to acquire more and more information. They selectively reorganize the information from the previous convolutional layer to generate the output of the current layer. However, each convolutional layer has limited feature extraction capabilities, inevitably leading to some information loss. Convolutional layers closer to the output retain less shallow information. Therefore, the influence of shallow features near the input decreases layer by layer in the network. However, shallow features contain rich details that are crucial for reconstructing details. While long-skip and short-skip connections alleviate this problem to some extent, they are primarily used to reduce training difficulty. Shallow information from convolutional layers located in the middle of the connections can only be passed layer by layer, and these shallow features are not fully utilized when the network extracts deeper features.
[0004] Therefore, a single-image super-resolution reconstruction method with feature sharing is proposed. By introducing multiple source-hop connections, shallow features can be directly sent to intermediate layers, that is, the intermediate layers share these features, in order to further make up for the shortcomings of existing methods. Summary of the Invention
[0005] The purpose of this invention is to address the technical problem of insufficient utilization of shallow features in existing residual block stacking methods, and to provide a depth residual super-resolution reconstruction method based on feature sharing. This method, when extracting deep features of an image layer by layer by stacking residual blocks, fully shares shallow feature information such as edges and corners in the input image, thereby obtaining better deep feature representation and achieving a higher quality, higher-resolution reconstructed image.
[0006] A depth residual super-resolution reconstruction method based on feature sharing includes the following steps:
[0007] Step 1: Data collection and selection;
[0008] Step 2: Data preprocessing, including low-resolution image generation, dataset augmentation, dataset enhancement, data standardization, and obtaining the training set;
[0009] Step 3: Select the network optimizer and design the hyperparameters;
[0010] Step 4: Design and build the network, including the design and implementation of the shallow feature extraction part, the deep feature extraction part, the feature map amplification part, and the reconstruction part;
[0011] Step 5: Train and optimize the network to obtain the optimal model;
[0012] Step 6: Reconstruct the target image using the optimal model obtained in Step 5.
[0013] Step 1 includes the following sub-steps:
[0014] Step 1-1: Select and collect high-resolution training sets;
[0015] Step 1-2: Select the test set.
[0016] Step 2 includes the following sub-steps:
[0017] Step 2-1: Generate low-resolution images corresponding to several images in the training set;
[0018] Step 2-2: Expand the training set;
[0019] Steps 2-3: Perform dataset augmentation;
[0020] Steps 2-4: Standardize the data.
[0021] In step 3, select the ADAM optimizer to adjust the network parameters.
[0022] In step 4, the network established is a feature-sharing-based super-resolution reconstruction network, specifically:
[0023] The network input LR is passed to a shallow feature extraction layer to obtain the feature map F. -1 →Shared feature extraction layer in the deep feature extraction part, resulting in feature map F0 →First shared source residual group, resulting in feature map F1 →Several second shared source residual groups →Third shared source residual group, resulting in feature map F G →The last convolutional layer in the deep feature extraction section;
[0024] The last convolutional layer of the deep feature extraction part, followed by the shallow feature extraction layer, and then the first feature fusion, yields the feature map F. DF →Feature map magnification section →Reconstruction section →Reconstructed image HR;
[0025] The following are the structures of the first shared source residual group, the second shared source residual group, and the third shared source residual group:
[0026] There are G shared source residual sets; the G shared source residual sets are connected in series from the input to the output.
[0027] The structure of the first shared source residual group is as follows: the feature map F0 obtained from the shared feature extraction layer of the deep feature extraction part, the first basic residual group → the second feature fusion, to obtain the first shared source residual group, whose output is the feature map F1;
[0028] The structure of several second shared source residual groups is as follows: feature map F0 obtained from the shared feature extraction layer of the deep feature extraction part, several second basic residual groups → several third feature fusion, to obtain several second shared source residual groups;
[0029] The structure of the third shared source residual group is as follows: the feature map F0 obtained from the shared feature extraction layer of the deep feature extraction part, the third basic residual group → the fourth feature fusion, to obtain the third shared source residual group, whose output is the feature map F. G ;
[0030] Each shared source residual set contains a basic residual set, and the G basic residual sets have the same structure.
[0031] Each basic residual group contains B shared source residual blocks, which are connected in series from the input to the output.
[0032] The first, second, and third basic residual groups are as follows:
[0033] Feature map of input from basic residual set → shared feature extraction layer of basic residual set → first shared source residual block;
[0034] The shared feature extraction layer of the basic residual group, the first basic residual block → feature fusion, to obtain the first shared source residual block;
[0035] First shared source residual block → several second shared source residual blocks;
[0036] A shared feature extraction layer for the basic residual group, several second basic residual blocks → feature fusion, to obtain several second shared source residual blocks;
[0037] Several second shared source residual blocks → third shared source residual blocks;
[0038] The shared feature extraction layer of the basic residual group, the third basic residual block → feature fusion, to obtain the third shared source residual block;
[0039] Third shared source residual block → the last convolutional layer of the basic residual set;
[0040] The feature map of the input of the basic residual group is fused into the last convolutional layer of the basic residual group to obtain the output of the basic residual group.
[0041] Taking the first basic residual group as an example, the specific details are as follows:
[0042] The feature map F0, which is the input of the first basic residual group, is then passed through the shared feature extraction layer of the first basic residual group to obtain the feature map F. 1,0 →The first shared source residual block outputs the feature map F. 1,1 ;
[0043] The feature map F obtained from the shared feature extraction layer of the first basic residual group 1,0 The first basic residual block is fused with the fifth feature to obtain the first shared source residual block, whose output is the feature map F.1,1 ;
[0044] The first shared source residual block yields the feature map F. 1,1 → Several second shared source residual blocks;
[0045] The feature map F obtained from the shared feature extraction layer of the first basic residual group 1,0 Several second basic residual blocks are fused with the sixth feature to obtain several second shared source residual blocks.
[0046] Several second shared source residual blocks → third shared source residual blocks, the output of which is feature map F 1,B ;
[0047] The feature map F obtained from the shared feature extraction layer of the first basic residual group 1,0 The third basic residual block is fused with the seventh feature to obtain the third shared source residual block, whose output is the feature map F. 1,B ;
[0048] Third shared source residual block → the last convolutional layer of the first basic residual group;
[0049] The first basic residual group's input feature map F0 is fused into the last convolutional layer of the first basic residual group → the eighth feature fusion, to obtain the output of the first basic residual group.
[0050] The process of constructing a basic residual set using shared source residual blocks is as follows:
[0051] Input of basic residual set → Shared feature extraction layer of basic residual set → First shared source residual block
[0052] The shared feature extraction layer of the basic residual group, the first basic residual block → the ninth feature fusion, to obtain the first shared source residual block;
[0053] First shared source residual block → Second shared source residual block;
[0054] The shared feature extraction layer of the basic residual group, the second basic residual block → the tenth feature fusion, to obtain the second shared source residual block;
[0055] Second shared source residual block → the last convolutional layer of the basic residual set;
[0056] Input to the basic residual set, the last convolutional layer of the basic residual set → eleventh feature fusion → output of the basic residual set.
[0057] In step 4, when using the established feature-sharing-based super-resolution reconstruction network, the following steps are included:
[0058] Step 4-1: Perform shallow feature extraction;
[0059] Shallow feature extraction is achieved by the first convolutional layer in the network, specifically using equation (1);
[0060] F -1 =H SFE (I LR (1)
[0061] Where H SRE (·) represents a two-dimensional convolution operation, whose input is I. LR Its output F -1 These are shallow features, which will be used as input for the deep feature extraction part;
[0062] Step 4-2: Perform deep feature extraction;
[0063] The deep feature layer consists of a shared feature extraction layer, G shared-source residual groups (SSRGs), a long skewing connection, and a convolutional layer.
[0064] Based on the shallow features obtained in step 4-1, the deep features are further obtained, and the deep features are obtained by equation (2);
[0065] F DF =H DFE (F -1 (2)
[0066] Where H DFE (·) The operation of the deep feature extraction part, its output F DF This represents a deep feature.
[0067] Step 4-3: Magnify deep features;
[0068] The deep feature amplification section employs a subpixel convolution module, which is used to amplify deep features to the target image size. Deep feature amplification is achieved using equation (3).
[0069] F UP =H UP (F DF (3)
[0070] Where H UP (·) indicates the operation of the subpixel volume module, F UP Indicates the magnified features;
[0071] Step 4-4: Reconstruct the magnified feature map;
[0072] The magnified feature map is then reconstructed using a two-dimensional convolutional layer. The reconstructed image is obtained from equation (4).
[0073] I SR =H REC (F UP )=H SRN (I LR (4)
[0074] Among them, I SR For the final reconstructed image, H REC (·) represents the operation for the reconstruction part, H SRN (·) represents the computation of a super-resolution reconstruction network based on feature sharing.
[0075] I LR and I HR Let I represent the low-resolution image and its corresponding high-resolution image as input to the feature-sharing-based super-resolution reconstruction network, respectively. SR This represents the super-resolution image output by a super-resolution reconstruction network based on feature sharing.
[0076] In step 4-2, when using the deep feature portion, the following sub-steps are employed:
[0077] The shared-source residual group, as the basic unit of the deep feature extraction part, consists of a basic residual group (RG) and a shared-source skip connection. Each basic residual group is composed of a shared feature extraction layer, B shared-source residual blocks (SSRB), a short skip connection, and a convolutional layer.
[0078] Step 4.2.1: Obtain the shared features of the G shared source residual groups using Equation (5);
[0079] F0=H0(F -1 (5)
[0080] Where H0(·) represents the operation of the shared feature extraction layer in the deep feature extraction part, which is a two-dimensional convolutional layer, and F0 represents the shared features;
[0081] Step 4.2.2: Using the shared features obtained in Step 4.2.1, calculate the output of each shared source residual group in sequence, and use it as the input of the next shared source residual group until the output of the last shared source residual group is obtained; the output of the g-th shared source residual group is obtained by equation (6);
[0082] Fg =w SF *F0+H g (F g-1 (6)
[0083] Where F g-1 and F g These are the input and output of the shared source residual set, respectively; w SF H represents the weight of the shared feature F0, initially set to 0, and then gradually assigned more weight to the shared feature during the learning process; g (·) indicates the output of the basic residual set portion in the current shared source residual set;
[0084] Therefore, the deep feature F in equation (2) DF It can be further expressed as equation (7);
[0085] F DF =H DFE (F -1 ) = F -1 +W LSC F G (7)
[0086] Among them W LSC F represents the set of weights of the last convolutional layer in the deep extraction part. G This is the output of the Gth shared source residual group.
[0087] In step 4.2.2, the basic residual set is used with the following sub-steps:
[0088] The shared source residual block in the basic residual group consists of a shared source hop connection and a basic residual block (RB). The basic residual block is a commonly used module in super-resolution reconstruction, consisting of two 2D convolutional layers, a ReLU layer, and an identity mapping.
[0089] Step 4.2.2.1 Obtain the shared features of B shared source residual blocks in the g-th basic residual group, and use Equation (8) to obtain the shared features;
[0090] F g，0 =H g，0 (F g-1 (8)
[0091] Where H g，0 (·) represents the shared feature extraction layer operation for the g-th basic residual group. This layer is a two-dimensional convolutional layer, F g-1 This represents its input, which is also the output of the previous shared source residual set, F. g，0 For shared features.
[0092] Step 4.2.2.2 Based on the shared feature F obtained in step 4.2.2.1 g，0 The output of each shared source residual block in the g-th residual group is calculated sequentially and used as the input of the next shared source residual block. The output of the b-th shared source residual block in the g-th basic residual group is obtained by equation (9);
[0093] F g，b =w g，SF *F g，0 +H g，b (F g，b-1 (9)
[0094] Among them, F g，b-1 and F g，b H represents the input and output of the b-th shared source residual set in the g-th basic residual set, respectively. g，b (·) represents the operation of the basic residual block in the current shared source residual block, w g，SF In the g-th residual group, B residual units share feature F. g，0 The weights are initially set to 0, and then gradually assigned more weight to shared features during the learning process.
[0095] Therefore, equation (6) can be further specified as equation (10).
[0096]
[0097] Among them W g Let F be the set of weights of the last convolutional layer in the g-th residual group. g，B This is the output of the B-th shared source residual block in the g-th basic residual group.
[0098] Step 5 includes the following steps:
[0099] Step 5-1: Use the training data obtained in Step 2 to train the network built in Step 4;
[0100] Step 5-2: Optimize the network using L1 loss. For a given training dataset... in and Let i and n represent the i-th (i = 1, ..., N) low-resolution image and the corresponding i-th high-resolution image in the dataset, respectively. It contains N LR images and the corresponding HR images. The optimization goal is to make the reconstructed image infinitely close to the original HR image, so that their difference is infinitely small, that is, to minimize the L1 loss. The loss is calculated using Equation (11).
[0101]
[0102] Where Θ represents the parameter set in the network, and the loss function is optimized using gradient descent.
[0103] Step 5-3: During the training process, after each training round, the optimized network is tested once using the test dataset to save the optimal model.
[0104] Compared with the prior art, the present invention has the following technical effects:
[0105] 1) In this invention, the shared source hop connection enables the shallow features extracted by the network, which contain a lot of details, to be fully utilized. The outer shared source hop connection enables this information to be transmitted to each residual group, while the inner shared source hop connection enables the shallow feature information transmitted from the outer layer to be utilized by each residual block in the current residual group. This ensures that each residual block in the entire network can obtain shallow information, which can alleviate the problem of loss of shallow information that is beneficial to reconstruction during the propagation process.
[0106] 2) Shared source jump connections facilitate gradient backpropagation during training, thereby reducing the difficulty of training. Attached Figure Description
[0107] The present invention will be further described below with reference to the accompanying drawings and embodiments:
[0108] Figure 1 This is a flowchart of the present invention;
[0109] Figure 2 This is a schematic diagram of the super-resolution reconstruction network based on feature sharing in this invention;
[0110] Figure 3 This is a schematic diagram illustrating the construction of a basic residual group containing two shared source residual blocks in this invention. Detailed Implementation
[0111] like Figure 1 A depth residual super-resolution reconstruction method based on feature sharing includes the following steps:
[0112] Step 1: Data collection and selection;
[0113] Step 2: Data preprocessing, including low-resolution image generation, dataset augmentation, dataset enhancement, data standardization, and obtaining the training set;
[0114] Step 3: Select the network optimizer and design the hyperparameters;
[0115] Step 4: Design and build the network, including the design and implementation of the shallow feature extraction part, the deep feature extraction part, the feature map amplification part, and the reconstruction part;
[0116] Step 5: Train and optimize the network to obtain the optimal model;
[0117] Step 6: Reconstruct the target image using the optimal model obtained in Step 5;
[0118] Step 1 includes the following sub-steps:
[0119] Step 1-1: Select and collect high-resolution training sets;
[0120] Step 1-2: Select the test set;
[0121] Step 2 includes the following sub-steps:
[0122] Step 2-1: Generate low-resolution images corresponding to several images in the training set;
[0123] Step 2-2: Expand the training set;
[0124] Steps 2-3: Perform dataset augmentation;
[0125] Steps 2-4: Standardize the data.
[0126] like Figure 2 As shown, in step 4, the network established is a feature-sharing-based super-resolution reconstruction network, specifically:
[0127] The network input LR is passed to the shallow feature extraction layer 1 to obtain the feature map F. -1 →Shared feature extraction layer 2 in the deep feature extraction part, obtaining feature map F0 →First shared source residual group 3, obtaining feature map F1 →Several second shared source residual groups 4 →Third shared source residual group 5, obtaining feature map F G →The last convolutional layer 6 in the deep feature extraction section;
[0128] The last convolutional layer 6 of the deep feature extraction part, the shallow feature extraction layer 1 → the first feature fusion 18, yields the feature map F. DF →Feature map magnification section 7 →Reconstruction section 8 →Reconstructed image HR;
[0129] The structures of the first shared source residual group 3, the second shared source residual group 4, and the third shared source residual group 5 are as follows:
[0130] There are G shared source residual sets; the G shared source residual sets are connected in series from the input to the output.
[0131] The structure of the first shared source residual group 3 is as follows: the feature map F0 obtained from the shared feature extraction layer 2 of the deep feature extraction part, the first basic residual group 9 → the second feature fusion 19, to obtain the first shared source residual group 3, whose output is the feature map F1;
[0132] The structure of several second shared source residual groups 4 is as follows: the feature map F0 obtained from the shared feature extraction layer 2 of the deep feature extraction part, several second basic residual groups → several third feature fusion, to obtain several second shared source residual groups;
[0133] The structure of the third shared source residual group 5 is as follows: the feature map f0 obtained from the shared feature extraction layer 2 of the deep feature extraction part, the third basic residual group 10 → the fourth feature fusion 20, to obtain the third shared source residual group 5, whose output is the feature map F. G ;
[0134] Each shared source residual set contains a basic residual set, and the G basic residual sets have the same structure.
[0135] Each basic residual group contains B shared source residual blocks, which are connected in series from the input to the output.
[0136] Among them, the first basic residual group 9, the second basic residual group, and the third basic residual group 10 are:
[0137] Feature map of input from basic residual set → shared feature extraction layer of basic residual set → first shared source residual block;
[0138] The shared feature extraction layer of the basic residual group, the first basic residual block → feature fusion, to obtain the first shared source residual block;
[0139] First shared source residual block → several second shared source residual blocks;
[0140] A shared feature extraction layer for the basic residual group, several second basic residual blocks → feature fusion, to obtain several second shared source residual blocks;
[0141] Several second shared source residual blocks → third shared source residual blocks;
[0142] The shared feature extraction layer of the basic residual group, the third basic residual block → feature fusion, to obtain the third shared source residual block;
[0143] Third shared source residual block → the last convolutional layer of the basic residual set;
[0144] The feature map of the input of the basic residual group is fused into the last convolutional layer of the basic residual group to obtain the output of the basic residual group.
[0145] Taking the first basic residual group 9 as an example, the specific explanation is as follows:
[0146] The feature map F0, which is the input of the first basic residual group, is then passed through the shared feature extraction layer 11 of the first basic residual group to obtain the feature map F. 1，0→The first shared source residual block 12, its output is the feature map F 1，1 ;
[0147] The feature map F obtained from the shared feature extraction layer 11 of the first basic residual group 1，0 The first basic residual block 16 is fused with the fifth feature fusion block 21 to obtain the first shared source residual block 12, whose output is the feature map F. 1，1 ;
[0148] The first shared source residual block 12 yields the feature map F. 1，1 →Several second shared source residual blocks 13;
[0149] The first basic residual group's shared feature extraction layer 11 yields the feature map F. 1，0 Several second basic residual blocks → sixth feature fusion, to obtain several second shared source residual blocks 13;
[0150] Several second shared source residual blocks 13 → third shared source residual blocks 14, the output of which is feature map F 1，B ;
[0151] The feature map F obtained from the shared feature extraction layer 11 of the first basic residual group 1，0 The third basic residual block 17 is fused with the seventh feature fusion 22 to obtain the third shared source residual block 14, whose output is the feature map F. 1，B ;
[0152] Third shared source residual block 14 → Last convolutional layer 15 of the first basic residual group;
[0153] The first basic residual group's input feature map F0, the last convolutional layer 15 of the first basic residual group → the eighth feature fusion 23, to obtain the output of the first basic residual group;
[0154] like Figure 3 The diagram shown is a simplified construction diagram of a basic residual set, illustrating the process of constructing a basic residual set using two shared source residual blocks:
[0155] Input of basic residual set 24 → Shared feature extraction layer of basic residual set 25 → First shared source residual block 26
[0156] The shared feature extraction layer 25 of the basic residual group, the first basic residual block 30 → the ninth feature fusion 33, to obtain the first shared source residual block 26;
[0157] First shared source residual block 26 → Second shared source residual block 27;
[0158] The shared feature extraction layer 25 of the basic residual group, the second basic residual block 31 → the tenth feature fusion 34, are used to obtain the second shared source residual block 27.
[0159] Second shared source residual block 27 → Last convolutional layer 28 of basic residual group;
[0160] Input to the basic residual group 24, the last convolutional layer of the basic residual group 28 → eleventh feature fusion 32 → output of the basic residual group 29.
[0161] In step 4, when using the established feature-sharing-based super-resolution reconstruction network, the following steps are included:
[0162] Step 4-1: Perform shallow feature extraction;
[0163] Shallow feature extraction is implemented by the first convolutional layer 1 in the network, specifically using equation (1);
[0164] F -1 =H SFE (I LR (1)
[0165] Where H SRE (·) represents a two-dimensional convolution operation, whose input is I. LR Its output F -1 These are shallow features, which will be used as input for the deep feature extraction part;
[0166] Step 4-2: Perform deep feature extraction;
[0167] The deep feature layer consists of a shared feature extraction layer 2, G shared-source residual groups (SSRGs) with structures as shown in 3, 4, and 5, a long skewing connection, and a convolutional layer 6.
[0168] Based on the shallow features obtained in step 4-1, the deep features are further obtained, and the deep features are obtained by equation (2);
[0169] F DF =H DFE (F -1 (2)
[0170] Where H DFE (·) The operation of the deep feature extraction part, its output F DF This represents a deep feature.
[0171] Step 4-3: Magnify deep features;
[0172] The deep feature amplification section employs a subpixel convolution module 7, which is used to amplify deep features to the target image size. Deep feature amplification is achieved using equation (3).
[0173] F UP =H UP (F DF (3)
[0174] Where H UP (·) indicates the operation of the subpixel volume module, F UP Indicates the magnified features;
[0175] Step 4-4: Reconstruct the magnified feature map;
[0176] The magnified feature map is reconstructed using a two-dimensional convolutional layer (8). The reconstructed image is obtained from equation (4).
[0177] I SR =H REC (F UP )=H SRN (I LR (4)
[0178] Among them, I SR For the final reconstructed image, H REC (·) represents the operation for the reconstruction part, H SRN (·) represents the computation of a super-resolution reconstruction network based on feature sharing.
[0179] I LR and I HR Let I represent the low-resolution image and its corresponding high-resolution image as input to the feature-sharing-based super-resolution reconstruction network, respectively. SR This represents the super-resolution image output by a super-resolution reconstruction network based on feature sharing.
[0180] In step 4-2, when using the deep feature portion, the following sub-steps are employed:
[0181] The shared source residual group, as the basic unit of the deep feature extraction part, contains a basic residual group (RG), as shown in 9 and 10, and a shared source jump connection. Each basic residual group consists of a shared feature extraction layer 11, B shared-source residual blocks (SSRB), as shown in 12, 13, and 14, a short jump connection, and a convolutional layer 15.
[0182] Step 4.2.1: Obtain the shared features of the G shared source residual groups using Equation (5);
[0183] F0=H0(F -1(5)
[0184] Where H0(·) represents the operation of the shared feature extraction layer in the deep feature extraction part, which is a two-dimensional convolutional layer, and F0 represents the shared features;
[0185] Step 4.2.2: Using the shared features obtained in Step 4.2.1, calculate the output of each shared source residual group in sequence, and use it as the input of the next shared source residual group until the output of the last shared source residual group is obtained; the output of the g-th shared source residual group is obtained by equation (6);
[0186] F g =w SF *F0+H g (F g-1 (6)
[0187] Where F g-1 and F g These are the input and output of the shared source residual set, respectively; w SF H represents the weight of the shared feature F0, initially set to 0, and then gradually assigned more weight to the shared feature during the learning process; g (·) indicates the output of the basic residual set portion in the current shared source residual set;
[0188] Therefore, the deep feature F in equation (2) DF It can be further expressed as equation (7);
[0189] F DF =H DFE (F -1 ) = F -1 +W LSC F G (7)
[0190] Among them W LSC F represents the set of weights of the last convolutional layer in the deep extraction part. G This is the output of the Gth shared source residual group.
[0191] In step 4.2.2, the basic residual set is used with the following sub-steps:
[0192] The shared source residual block in the basic residual group, such as 12, consists of a shared source hop connection and a basic residual block (RB), as shown in 16. The basic residual block is a commonly used module in super-resolution reconstruction, consisting of two 2D convolutional layers, a ReLU layer, and an identity mapping.
[0193] Step 4.2.2.1 Obtain the shared features of B shared source residual blocks in the g-th basic residual group, and use Equation (8) to obtain the shared features;
[0194] F g，0 =H g，0 (F g-1 (8)
[0195] Where H g，0 (·) represents the shared feature extraction layer operation for the g-th basic residual group. This layer is a two-dimensional convolutional layer, F g-1 This represents its input, which is also the output of the previous shared source residual set, F. g，0 For shared features.
[0196] Step 4.2.2.2 Based on the shared feature F obtained in step 4.2.2.1 g，0 The output of each shared source residual block in the g-th residual group is calculated sequentially and used as the input of the next shared source residual block. The output of the b-th shared source residual block in the g-th basic residual group is obtained by equation (9);
[0197] F g，b =w g，SF *F g，0 +H g，b (F g，b-1 (9)
[0198] Among them, F g，b-1 and F g，b H represents the input and output of the b-th shared source residual set in the g-th basic residual set, respectively. g，b (·) represents the operation on the basic residual set in the current shared source residual block, w g，SF In the g-th residual group, B residual units share feature F. g，0 The weights are initially set to 0, and then gradually assigned more weight to shared features during the learning process.
[0199] Therefore, equation (6) can be further specified as equation (10).
[0200]
[0201] Among them W g Let F be the set of weights of the last convolutional layer in the g-th residual group. g，B This is the output of the B-th shared source residual block in the g-th basic residual group.
[0202] Step 5 includes the following steps:
[0203] Step 5-1: Use the training data obtained in Step 2 to train the network built in Step 4;
[0204] Step 5-2: Optimize the network using L1 loss. For a given training dataset... in and Let i and n represent the i-th (i = 1, ..., N) low-resolution image and the corresponding i-th high-resolution image in the dataset, respectively. It contains N LR images and the corresponding HR images. The optimization goal is to make the reconstructed image infinitely close to the original HR image, so that their difference is infinitely small, that is, to minimize the L1 loss. The loss is calculated using Equation (11).
[0205]
[0206] Where Θ represents the parameter set in the network, and the loss function is optimized using gradient descent.
[0207] Step 5-3: During the training process, after each training round, the optimized network is tested once using the test dataset to save the optimal model.
[0208] Example: This example uses Peak Signal-to-Noise Ratio (PSNR) and Structured Pixels (SSIM), commonly used evaluation metrics in super-resolution reconstruction, to evaluate the model's performance. Table 1 compares the reconstruction results with the classic method RCAN, where the total number of training iterations for both models is 1000 for different magnifications.
[0209] Table 1 compares the experimental results with those of rcan.
[0210]
[0211] As can be seen from Table 1, after training the same number of iterations, for the two test sets Set5 and Set14, the reconstruction results of the method proposed in this invention are on par with RCAN at a magnification of 2, and the reconstruction results are higher than RCAN at a magnification of 3.
[0212] In summary, the feature-sharing-based deep residual super-resolution reconstruction method proposed in this invention utilizes shared source residual blocks and shared source residual groups constructed by shared source jump connections. With only a small increase in the number of parameters and computational cost, it can indeed make fuller use of the information in the shallow layer that is beneficial to reconstruction, extract more detailed deep features, and ultimately reconstruct a better image.
Claims
1. A deep residual super-resolution reconstruction method based on feature sharing, characterized in that, It comprises the following steps: Step 1: collection and selection of data sets; Step 2: data preprocessing, including generation of low-resolution images, data set expansion, data set enhancement, data standardization, and obtaining a training set; Step 3: selection of network optimizer and design of hyperparameters; Step 4: design and construction of the network, including design and implementation of the shallow feature extraction part, the deep feature extraction part, the feature map amplification part, and the reconstruction part; Step 5: training and optimization of the network to obtain an optimal model; Step 6: reconstruction of the target image using the optimal model obtained in step 5 to reconstruct the target image; In step 1, the following sub-steps are included: Step 1-1: selection and collection of high-resolution image training sets; Step 1-2: selection of test sets; In step 2, the following sub-steps are included: Step 2-1: generation of low-resolution images corresponding to several images in the training set; Step 2-2: expansion of the training set; Step 2-3: enhancement of the data set; Step 2-4: standardization of the data; In step 4, the established network is a feature-sharing-based super-resolution reconstruction network, specifically: Network input LR→ shallow feature extraction layer (1), feature map → shared feature extraction layer (2) of deep feature extraction part, feature map → first shared source residual group (3), feature map → several second shared source residual groups (4)→ third shared source residual group (5), feature map → last convolutional layer (6) of deep feature extraction part; The last convolutional layer (6) of the deep feature extraction part, the shallow feature extraction layer (1) → the first feature fusion (18), and the feature map → The feature map enlargement part (7) → the reconstruction part (8) → the reconstructed image HR; The structure of the first shared source residual group (3), the second shared source residual group (4), and the third shared source residual group (5) is as follows: There are G shared source residual groups in total; the G shared source residual groups are sequentially connected from the input end to the output end; The structure of the first shared source residual group (3) is as follows: the feature map obtained by the shared feature extraction layer (2) of the deep feature extraction part , the first basic residual group (9) → the second feature fusion (19), to obtain the first shared source residual group (3), and the output is a feature map ; The structure of the second shared source residual group (4) is: the feature map obtained by the shared feature extraction layer (2) of the deep feature extraction part , the second basic residual group → the third feature fusion, to obtain the second shared source residual group; The structure of the third shared source residual group (5) is as follows: the feature map obtained by the shared feature extraction layer (2) of the deep feature extraction part , the third basic residual group (10)→the fourth feature fusion (20), to obtain the third shared source residual group (5), and the output is a feature map ; Each shared source residual group contains a basic residual group, and the G basic residual groups have the same structure; Each basic residual group contains B shared source residual blocks, which are sequentially connected from the input end to the output end; the first basic residual group (9), the second basic residual group, and the third basic residual group (10) are as follows: The input feature map of the first basic residual group → the shared feature extraction layer (11) of the first basic residual group → the first shared source residual block (12); The shared feature extraction layer (11) of the first basic residual group, the first basic residual block (16) → the fifth feature fusion (21), obtaining the first shared source residual block (12); The first shared source residual block (12) → the second shared source residual block (13); The shared feature extraction layer (11) of the first basic residual group, the second basic residual block → the sixth feature fusion, obtaining the second shared source residual block (13); The second shared source residual block (13) → the third shared source residual block (14); The shared feature extraction layer (11) of the first basic residual group, the third basic residual block (17) → the seventh feature fusion (22), obtaining the third shared source residual block (14); The third shared source residual block (14) → the last convolutional layer (15) of the first basic residual group; The input feature map of the first basic residual group, the last convolutional layer (15) of the first basic residual group → the eighth feature fusion (23), obtaining the output of the first basic residual group.
2. The method of claim 1, wherein, The process of constructing a basic residual group with two shared source residual blocks is as follows: The input (24) of the basic residual group → the shared feature extraction layer (25) of the basic residual group → the first shared source residual block (26); shared feature extraction layer (25) of the basic residual group, the first basic residual block (30)→the ninth feature fusion (33), to obtain the first shared source residual block (26); the first shared source residual block (26)→the second shared source residual block (27); shared feature extraction layer (25) of the basic residual group, the second basic residual block (31)→the tenth feature fusion (34), to obtain the second shared source residual block (27); the second shared source residual block (27)→the last convolution layer (28) of the basic residual group; the input (24) of the basic residual group, the last convolution layer (28) of the basic residual group→the eleventh feature fusion (32)→the output (29) of the basic residual group.
3. The method of claim 1, wherein, In step 4, when using the established feature sharing based super-resolution reconstruction network, the following steps are included: Step 4-1: shallow layer feature extraction is performed; The shallow layer feature extraction is realized by the first convolution layer (1) in the network, and formula (1) is used specifically; ； wherein is a two-dimensional convolution operation, whose input is , whose output is a shallow feature, which will be the input of the deep feature extraction part; Step 4-2: deep layer feature extraction is performed; The deep layer feature part is composed of a shared feature extraction layer (2), G shared source residual groups (SSRG), a long skip connection, and a convolution layer (6); According to the shallow layer feature obtained in step 4-1, the deep layer feature is further obtained, and the deep layer feature is obtained by formula (2); ； wherein the operation of the deep feature extraction portion, which outputs is a deep feature; Step 4-3: the deep layer feature is enlarged; The sub-pixel convolution module (7) is used for the deep layer feature enlargement part, which is used to enlarge the deep layer feature to the target image size; formula (3) is used to realize the deep layer feature enlargement; ； wherein represents the operation of the sub-pixel convolution module, represents the amplified feature; Step 4-4: the enlarged feature map is reconstructed; The enlarged feature map is reconstructed, and a two-dimensional convolution layer (8) is used to realize the reconstruction; the reconstructed image is obtained by formula (4); ； wherein, is the final reconstructed image, is the operation of reconstructing the part, is the operation of the feature sharing based super-resolution reconstruction network; and denote a low-resolution image and its corresponding high-resolution image, respectively, input to the feature sharing based super-resolution reconstruction network, denote a super-resolution image output by the feature sharing based super-resolution reconstruction network.
4. The method of claim 3, wherein, In step 4-2, the deep layer feature part is used in the following sub-steps: The shared source residual group is the basic unit of the deep layer feature extraction part, which includes a basic residual group (RG) and a shared source skip connection; each basic residual group is composed of a shared feature extraction layer (11), B shared source residual blocks (SSRB), a short skip connection, and a convolution layer (15); Step 4.2.1: obtain the shared feature of the G shared source residual groups, and obtain the shared feature by formula (5); ； wherein represents an operation of a shared feature extraction layer of the deep feature extraction section, the layer being a two-dimensional convolution layer, represents a shared feature; Step 4.2.2: use the shared feature obtained in step 4.2.1 to calculate the output of each shared source residual group in turn, and take it as the input of the next shared source residual group, until the output of the last shared source residual group is obtained; the output of the gth shared source residual group is obtained by formula (6); ； wherein and are input and output of the shared source residual group, respectively; represents the weight of the shared feature with an initial value of 0, and then gradually gives more weight to the shared feature in the process of learning; represents the output of the basic residual group part in the current shared source residual group; Thus, the deep features in equation (2) can be further represented as equation (7); Thus, the deep features in equation (2) can be further represented as equation (7); ； wherein denotes the set of weights of the last convolutional layer (6) in the deep extraction part, is the output of the Gth shared source residual group.
5. The method of claim 4, wherein, In step 4.2.2, the basic residual group is used in the following sub-steps: The shared source residual block in the basic residual group is composed of a shared source skip connection and a basic residual block (RB) ; the basic residual block is a module commonly used in super-resolution reconstruction, and is composed of two two-dimensional convolution layers, a ReLU layer and an identity mapping; Step 4.2.2.1: Obtain the shared features of the B shared source residual blocks in the gth basic residual group, and obtain the shared features by formula (8) ; ； wherein is the operation of the shared feature extraction layer of the gth basic residual group, which is a two-dimensional convolution layer, denotes its input, which is also the output of the previous shared source residual group, is the shared feature; Step 4.2.2.2: calculating the output of each shared source residual block in the gth residual group according to the shared feature obtained in step 4.2.2.1 , and the output of the bth shared source residual block in the gth basic residual group is obtained by equation (9). ； in, and Let these represent the input and output of the b-th shared source residual set within the g-th basic residual set, respectively. For the computation of the basic residual blocks in the current shared source residual block, In the g-th residual group, B residual units share the same feature. The weights are initially set to 0, and then more weight is gradually given to shared features during the learning process; Therefore, formula (6) can be further specified as formula (10) : ； wherein is a set of weights of the last convolutional layer in the gth residual group, is an output of the Bth shared source residual block in the gth basic residual group.
6. The method of claim 1, wherein, Step 5 includes the following steps: Step 5-1: using the training data obtained in step 2 to train the network built in step 4; Step 5-2: Using loss to optimize the network; for a given training dataset , where and denote the th low-resolution image and the corresponding th high-resolution image in the dataset, respectively, where ; containing N LR images and corresponding HR images, the goal of the optimization is to make the reconstructed images infinitely close to the original HR images, making their difference infinitely small, i.e. to minimize loss, which is calculated using equation (11); ； wherein denotes the set of parameters in the network, and the optimization method of the loss function is gradient descent. Step 5-3: during the training process, after training each round, the optimized network is tested once using the test data set to save an optimal model.
Citation Information
Patent Citations
Image decompression effect method based on enhanced separable convolutional network
CN115249274A
Image processing method and apparatus
WO2022068682A1