Image super-resolution reconstruction method based on MGD-DUMRN network
By using multi-scale feature fusion and momentum gradient descent algorithm of MGD-DUMRN network, the problems of transparency and detail capture in image super-resolution reconstruction in existing technologies are solved, and efficient and clear image reconstruction results are achieved.
Patent Information
- Application Number
- CN202411074980.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
Existing image super-resolution reconstruction methods lack transparency and interpretability, and are difficult to effectively capture fine details in images.
An image super-resolution reconstruction method based on the MGD-DUMRN network is adopted. Through multi-scale feature fusion and momentum gradient descent algorithm, feature extraction, image interpolation, feature reconstruction and image reconstruction units are constructed. The units are trained by combining loss function and evaluation function to generate high-resolution images.
It improves the interpretability and robustness of image reconstruction, enabling the generation of visually clear and natural high-resolution images in resource-constrained environments, adapting to different image content and conditions, and reducing computational resource requirements.
Smart Images

Figure CN119006284A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and in particular relates to an image super-resolution reconstruction method. Background Art
[0002] Image super-resolution reconstruction is one of the classic inverse problems in the field of low-level computer vision. It aims to restore natural, clear and detailed high-resolution (HR) corresponding images from low-resolution (LR) images, and has a wide range of practical applications. With the development of deep learning, model-based deep neural networks have made significant progress in image super-resolution reconstruction tasks. However, existing advanced super-resolution reconstruction methods are usually designed as "black boxes" and lack transparency and interpretability. The emergence of deep unfolding methods, compared with traditional model-based and learning-based methods, not only maintains computational efficiency, but also significantly improves the interpretability of deep learning.
[0003] The design concept of multi-scale feature fusion has a profound impact on the development of computer vision. Multi-scale refers to sampling signals at different granularities. Different features can be observed at different scales, which is suitable for different tasks. The feature fusion method effectively integrates contextual information to provide a more comprehensive and rich visual background, thereby enhancing the robustness and accuracy of the model. However, multi-scale fusion requires a suitable fusion strategy to improve the effect of image processing. Therefore, when building a network, we should focus on the ability to learn image features and contextual information. Summary of the invention
[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an image super-resolution reconstruction method based on the MGD-DUMRN network that can capture fine detail information.
[0005] The technical solution adopted to solve the above technical problems is: an image super-resolution reconstruction method based on MGD-DUMRN network, comprising the following steps:
[0006] Step 1. Dataset preprocessing
[0007] Obtain an HD image dataset, divide it into a training set X1, a validation set X2, and a test set X3 in proportion, resize the training set X1 to obtain image datasets R1 and R2, randomly crop each image in the training set X1, image dataset R1, and image dataset R2 three times according to the pyramid comparison, and crop them into image blocks with pixel sizes of R×R, M×M, and C×C, R≠M≠C, and perform normalization to obtain a new training set X11. At the same time, perform the same resizing, cropping, and normalization on the validation set X2 and the test set X3 to obtain a new validation set X21 and a new test set X31.
[0008] Step 2. Build the MGD-DUMRN network
[0009] The MGD-DUMRN network consists of a feature extraction unit, an image interpolation processing unit, a feature reconstruction module based on the momentum gradient descent algorithm, and an image reconstructor unit;
[0010] The feature extraction unit is used to map the low-resolution image to a feature space to generate a feature map;
[0011] The image interpolation processing unit is used to perform interpolation processing on the feature map;
[0012] The feature reconstruction module is composed of 6 feature reconstruction submodules with the same structure that are iteratively constructed in sequence, and each submodule uses a momentum gradient descent algorithm to perform feature reconstruction;
[0013] The feature reconstruction submodule includes a multi-scale feature fusion unit and a residual structure; the multi-scale feature fusion unit is used to perform feature fusion of the original image and images of different magnifications to generate feature priors; the residual structure uses a residual block instead of a step size to achieve adaptive extended gradients;
[0014] The image reconstructor unit is used to convert the depth feature map output by the feature reconstruction module into the final image output;
[0015] Step 3. Train the MGD-DUMRN network
[0016] 1) Determine the objective function
[0017] The objective function includes the loss function Loss and the evaluation functions PSNR and SSIM. The loss function Loss is determined as follows:
[0018]
[0019] Where N is the number of training patch pairs, y i is a low-resolution image sample, x i is a real high-resolution image sample, f is the MGD-DUMRN function; i∈[1,N];
[0020] The evaluation function PSNR is determined by the following formula:
[0021]
[0022]
[0023] Where MSE is the mean square error, I(i,j) is the pixel value of the reference image at coordinate (i,j), K(i,j) is the pixel value of the image to be compared at coordinate (i,j), m is the height of the image, n is the width of the image, and MAX isI is the maximum pixel value of the reference image;
[0024] The evaluation function SSIM is determined by the following formula:
[0025]
[0026] In the formula, μ a , μ b are the mean values of local image blocks a and b, respectively, a , σ b are the standard deviations of image local blocks a and b, respectively, a ·σ b is the covariance of the local image block a and the local image block b, c1 and c2 are both constants;
[0027] 2) Training the MGD-DUMRN network
[0028] The new training set X11 is input into the MGD-DUMRN network for training. During the training process, the learning rate of the MGD-DUMRN network is γ, and the optimizer uses the Adam optimization method, and iterates the training until convergence;
[0029] Step 4. Save the weights file
[0030] During the training of the MGD-DUMRN network, the weights are continuously updated using the deep learning framework and the weight files are saved;
[0031] Step 5. Verify the MGD-DUMRN network
[0032] Input the new validation set X21 into the MGD-DUMRN network for validation;
[0033] Step 6. Test the MGD-DUMRN network
[0034] Input the new test X31 into the MGD-DUMRN network for testing, load the saved weight file, and obtain the super-resolution reconstruction result of the image.
[0035] As a preferred technical solution, in step 2, each submodule performs feature reconstruction of the momentum gradient descent algorithm according to the following formula;
[0036] x t+1 =x t -A t [(x t -y)-S t (x t )]-B t [(x t-1 -y)-S t-1 (xt-1 )]+C t (x t-1 -x t-2 )
[0037] In the formula, x t+1 is the feature reconstruction result of the t+1th time, x t is the t-th feature reconstruction result, y = x 0 is the input image after linear interpolation, A t and B t is the step size, C t is the momentum parameter of the tth time, λ is the regularization parameter, is the current estimate x at iteration t t The gradient of the regularization term φ of is the value of the last estimate x at the t-1th iteration t-1 The gradient of the regularization term φ;
[0038] As a preferred technical solution, in step 2, the multi-scale feature fusion unit is composed of the following formula:
[0039] S(X SR )=S(X)↑4
[0040] S(X)=G3(X)+e2+e3
[0041] e2=G1(X)↓4-G3(X↓4)
[0042] e3=S(X2)↓2-G1(X)↓4
[0043] S(X2)=G2(X 2× )+e1
[0044] e1=G1(X)↓2-G2(X↓2)
[0045] In the formula, S(X SR ) is the feature prior, G1(φ), G2(φ), and G3(φ) are deep features learned from different feature scales, X is the input image of the current submodule, G1(X)↓4 is the result of downsampling the input image X to 1 / 4 after being processed by G1, X↓4 is the result of reducing the image X to 1 / 4 of its original size by downsampling, G1(X)↓2 is the result of downsampling the image X to 1 / 2 of its original size after being processed by G1, X↓2 is the result of reducing the image X to half of its original size by downsampling, X 2× It is the feature map obtained by inputting another magnification image.
[0046] As a preferred technical solution, the structures of deep learning modules G1, G2, and G3 are composed of a 3×3 convolution layer 1, a ReLU activation function unit 1, a ResBlock layer, a 3×3 convolution layer 2, and a ReLU activation function unit 2 connected in sequence. The number of channels in the deep learning module G1 is 64, the number of channels in the deep learning module G2 is 128, and the number of channels in the deep learning module G3 is 256.
[0047] The beneficial effects of the present invention are as follows:
[0048] The present invention utilizes multi-scale feature fusion and momentum gradient descent algorithm to effectively capture high-frequency details in images. Through step-by-step iterative optimization, it can generate visually clearer and more natural high-resolution images and improve image quality. It combines multi-scale information and regularization terms to provide robust super-resolution reconstruction results even in the presence of noise or imperfect data.
[0049] Compared with the traditional "black box" deep learning method, the present invention provides better model interpretability for the deep unfolding network framework, adapts to different image contents and conditions, reduces the demand for computing resources, enables effective image reconstruction even in resource-constrained environments, and has higher generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a schematic diagram of the process of the present invention.
[0051] Figure 2 It is a structural diagram of the multi-scale feature fusion unit.
[0052] Figure 3 It is a schematic diagram of feature reconstruction of the feature reconstruction module of the present invention based on the momentum gradient descent algorithm.
[0053] Figure 4 It is a schematic diagram of the super-resolution result of the present invention in an example of the present invention. DETAILED DESCRIPTION
[0054] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples, but the present invention is not limited to the following embodiments.
[0055] exist Figure 1 In the present invention, an image super-resolution reconstruction method based on the MGD-DUMRN network comprises the following steps:
[0056] Step 1. Dataset preprocessing
[0057] 4999 HD images were randomly divided into training set X1, validation set X2 and test set X3 in a ratio of 7:1.5:1.5. The training set X1 was resized to obtain image datasets R1 and R2. Each image in the training set X1, image dataset R1 and image dataset R2 was randomly cropped three times according to the pyramid comparison, cropped into image blocks with pixel sizes of 192×192, 96×96 and 48×48, and normalized to obtain a new training set X11. At the same time, the validation set X2 and the test set X3 were resized, cropped and normalized in the same way to obtain a new validation set X21 and a new test set X31.
[0058] Step 2. Build the MGD-DUMRN network
[0059] The MGD-DUMRN network consists of a feature extraction unit, an image interpolation processing unit, a feature reconstruction module based on the momentum gradient descent algorithm, and an image reconstructor unit.
[0060] A feature extraction unit, used for mapping the original low-resolution image into a feature space to generate a feature map;
[0061] An image interpolation processing unit, used for adjusting the size of an input low-resolution image to a required high-resolution size;
[0062] The feature reconstruction module is composed of six feature reconstruction submodules with the same structure, which are iteratively constructed in sequence. The feature reconstruction submodule includes a multi-scale feature fusion unit and a residual structure.
[0063] The multi-scale feature fusion unit fuses the interpolated feature map with images of different magnifications according to the following formula to generate feature priors, such as Figure 2 ;
[0064] S(X SR )=S(X)↑4
[0065] S(X)=G3(X)+e2+e3
[0066] e2=G1(X)↓4-G3(X↓4)
[0067] e3=S(X2)↓2-G1(X)↓4
[0068] S(X2)=G2(X 2× )+e1
[0069] e1=G1(X)↓2-G2(X↓2)
[0070] In the formula, S(X SR) is the feature prior, G1(φ), G2(·), and G3(·) are deep features learned from different feature scales, X is the input image of the current submodule, G1(X)↓4 is the result of downsampling the input image X to 1 / 4 after being processed by G1, X↓4 is the result of reducing the image X to 1 / 4 of its original size by downsampling, G1(X)↓2 is the result of downsampling the input image X to 1 / 2 of its original size after being processed by G1, X↓2 is the result of reducing the input image X to half of its original size by downsampling, X 2× It is the feature map obtained by inputting another magnification image;
[0071] Among them, the structures of deep learning modules G1, G2, and G3 are all composed of 3×3 convolution layer 1, ReLU activation function unit 1, ResBlock layer, 3×3 convolution layer 2, and ReLU activation function unit 2 connected in sequence. The number of channels in the deep learning module G1 is set to 64, the number of channels in the deep learning module G2 is set to 128, and the number of channels in the deep learning module G3 is set to 256;
[0072] The residual structure uses residual blocks instead of step size to achieve adaptive expansion of gradients.
[0073] like Figure 3 ,Each submodule uses the momentum gradient descent algorithm to reconstruct features according to the following formula;
[0074] x t+1 =x t -A t [(x t -y)-S t (x t )]-B t [(x t-1 -y)-S t-1 (x t-1 )]+C t (x t-1 -x t-2 )
[0075] In the formula, x t+1 is the feature reconstruction result of the t+1th time, x t is the t-th feature reconstruction result, y = x 0 is the input image after linear interpolation, A t and B t is the step size, C t is the momentum parameter of the tth time, λ is the regularization parameter, is the current estimate x at iteration t t The gradient of the regularization term φ of is the value of the last estimate x at the t-1th iteration t-1 The gradient of the regularization term φ;
[0076] The image reconstructor unit is used to convert the deep feature map output by the feature reconstruction module into the final image output.
[0077] Step 3. Train the MGD-DUMRN network
[0078] 1) Determine the objective function
[0079] The objective function includes the loss function Loss and the evaluation functions PSNR and SSIM. The loss function Loss is determined as follows:
[0080]
[0081] Where N is the number of training patch pairs, y i is a low-resolution image sample, x i is a real high-resolution image sample, f is the MGD-DUMRN function; i∈[1,N];
[0082] The evaluation function PSNR is determined by the following formula:
[0083]
[0084]
[0085] Where MSE is the mean square error, I(i,j) is the pixel value of the reference image at coordinate (i,j), K(i,j) is the pixel value of the image to be compared at coordinate (i,j), m is the height of the image, n is the width of the image, and MAX is I is the maximum pixel value of the reference image;
[0086] The evaluation function SSIM is determined by the following formula:
[0087]
[0088] In the formula, μ a , μ b are the mean values of local image blocks a and b, respectively, a , σ b are the standard deviations of image local blocks a and b, respectively, a ·σ b is the covariance of the local image block a and the local image block b, c1 and c2 are both constants;
[0089] 2) Training the MGD-DUMRN network
[0090] The new training set X11 is input into the MGD-DUMRN network for training. During the training process, the learning rate of the MGD-DUMRN network is γ = 10 -4 ,The optimizer uses the Adam optimization method and iterates the training until convergence;
[0091] Step 4. Save the weights file
[0092] During the training of the MGD-DUMRN network, the weights are continuously updated using the deep learning framework and the weight files are saved;
[0093] Step 5. Verify the MGD-DUMRN network
[0094] Input the new validation set X21 into the MGD-DUMRN network for validation;
[0095] Step 6. Test the MGD-DUMRN network
[0096] Input the new test X31 into the MGD-DUMRN network for testing, load the saved weight file, and obtain the super-resolution reconstruction result of the image, as shown in Figure 4 .
Claims
1. An image super-resolution reconstruction method based on MGD-DUMRN network, characterized in that: The following steps are involved: Step 1. Dataset preprocessing Obtain an HD image dataset, divide it into a training set X1, a validation set X2, and a test set X3 in proportion, resize the training set X1 to obtain image datasets R1 and R2, randomly crop each image in the training set X1, image dataset R1, and image dataset R2 three times according to the pyramid comparison, and crop them into image blocks with pixel sizes of R×R, M×M, and C×C, R≠M≠C, and perform normalization to obtain a new training set X11. At the same time, perform the same resizing, cropping, and normalization on the validation set X2 and the test set X3 to obtain a new validation set X21 and a new test set X31. Step 2. Build the MGD-DUMRN network The MGD-DUMRN network consists of a feature extraction unit, an image interpolation processing unit, a feature reconstruction module based on the momentum gradient descent algorithm, and an image reconstructor unit; The feature extraction unit is used to map the low-resolution image to a feature space to generate a feature map; The image interpolation processing unit is used to perform interpolation processing on the feature map; The feature reconstruction module is composed of 6 feature reconstruction submodules with the same structure that are iteratively constructed in sequence, and each submodule uses a momentum gradient descent algorithm to perform feature reconstruction; The feature reconstruction submodule includes a multi-scale feature fusion unit and a residual structure; the multi-scale feature fusion unit is used to perform feature fusion of the original image and images of different magnifications to generate feature priors; the residual structure uses a residual block instead of a step size to achieve adaptive extended gradients; The image reconstructor unit is used to convert the depth feature map output by the feature reconstruction module into the final image output; Step 3. Train the MGD-DUMRN network 1) Determine the objective function The objective function includes the loss function Loss and the evaluation functions PSNR and SSIM. The loss function Loss is determined as follows: Where N is the number of training patch pairs, y i is a low-resolution image sample, x i is a real high-resolution image sample, f is the MGD-DUMRN function; i∈[1,N]; The evaluation function PSNR is determined by the following formula: Where MSE is the mean square error, I(i,j) is the pixel value of the reference image at coordinate (i,j), K(i,j) is the pixel value of the image to be compared at coordinate (i,j), m is the height of the image, n is the width of the image, and MAX is I is the maximum pixel value of the reference image; The evaluation function SSIM is determined by the following formula: In the formula, μ a , μ b are the mean values of local image blocks a and b, respectively, a , σ b are the standard deviations of image local blocks a and b, respectively, a ·σ b is the covariance of the local image block a and the local image block b, c1 and c2 are both constants; 2) Training the MGD-DUMRN network The new training set X11 is input into the MGD-DUMRN network for training. During the training process, the learning rate of the MGD-DUMRN network is γ, and the optimizer uses the Adam optimization method, and iterates the training until convergence; Step 4. Save the weights file During the training of the MGD-DUMRN network, the weights are continuously updated using the deep learning framework and the weight files are saved; Step 5. Verify the MGD-DUMRN network Input the new validation set X21 into the MGD-DUMRN network for validation; Step 6. Test the MGD-DUMRN network Input the new test X31 into the MGD-DUMRN network for testing, load the saved weight file, and obtain the super-resolution reconstruction result of the image.
2. The image super-resolution reconstruction method based on the MGD-DUMRN network according to claim 1, characterized in that: In step 2, each submodule performs feature reconstruction using the momentum gradient descent algorithm according to the following formula; x t+1 =x t -A t [(x t -y)-S t (x t )]-B t [(x t-1 -y)-S t-1 (x t-1 )]+C t (x t-1 -x t-2 ) In the formula, x t+1 is the feature reconstruction result of the t+1th time, x t is the t-th feature reconstruction result, y = x 0 is the input image after linear interpolation, A t and B t is the step size, C t is the momentum parameter of the tth time, λ is the regularization parameter, is the current estimate x at iteration t t The gradient of the regularization term φ of is the value of the last estimate x at the t-1th iteration t-1 The gradient of the regularization term φ.
3. The image super-resolution reconstruction method based on the MGD-DUMRN network according to claim 1, characterized in that: In step 2, the multi-scale feature fusion unit is composed of the following formula: S(X SR )=S(X)↑4 S(X)=G3(X)+e2+e3 e2=G1(X)↓4-G3(X↓4) e3=S(X2)↓2-G1(X)↓4 S(X2)=G2(X 2× )+e1 e1=G1(X)↓2-G2(X↓2) In the formula, S(X SR ) is the feature prior, G1(·), G2(·), and G3(·) are deep features learned from different feature scales, X is the input image of the current submodule, G1(X)↓4 is the result of downsampling the input image X to 1 / 4 after being processed by G1, X↓4 is the result of reducing the image X to 1 / 4 of its original size by downsampling, G1(X)↓2 is the result of downsampling the image X to 1 / 2 of its original size after being processed by G1, X↓2 is the result of reducing the image X to half of its original size by downsampling, and X 2× It is the feature map obtained by inputting another magnification image.
4. The image super-resolution reconstruction method based on the MGD-DUMRN network according to claim 3 is characterized in that: The structures of the deep learning modules G1, G2, and G3 are all composed of a 3×3 convolution layer 1, a ReLU activation function unit 1, a ResBlock layer, a 3×3 convolution layer 2, and a ReLU activation function unit 2 connected in sequence. The number of channels in the deep learning module G1 is 64, the number of channels in the deep learning module G2 is 128, and the number of channels in the deep learning module G3 is 256.
