A layered image super-resolution method, system, computer device, and application

Abstract

This invention belongs to the field of image processing technology and discloses a layered image super-resolution method, system, computer equipment, and application. It uses the DIV2K image dataset as the training set and layers the training samples based on a guided filter. Two deep residual convolutional networks, DLRCN and DGRCN, are designed for training on detail images and smoothed images, respectively. The DLRCN is trained using a training set of detail image samples, and the DGRCN is trained using a training set of smoothed image samples. The accuracy and reliability of the proposed model are verified by training the DLRCN on a test set of detail images and the DGRCN on a test set of smoothed images. This invention performs super-resolution and fusion on the detail and smoothing layers separately, improving image resolution and enhancing details.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a layered image super-resolution method, system, computer device and application. Background Technology

[0002] Currently, images from various fields, including medical images, remote sensing images, astronomical images, SRA radar images, and military night vision images, are subject to interference from the equipment itself and external signals during imaging and transmission. This results in images with significant noise, causing image details to be buried in noise or reducing image resolution, severely impacting image quality and hindering subsequent image processing tasks such as image segmentation and recognition. Image super-resolution reconstruction technology is a technique that uses software processing methods to process low-resolution images of the same scene to obtain high-resolution images without changing existing hardware imaging equipment and technology. This technology can economically and efficiently improve image resolution and visual quality, and has become a research hotspot in the field of image processing.

[0003] In recent years, deep learning-based image super-resolution technology has been widely studied in academia. Dong et al. from the Chinese University of Hong Kong were the first to introduce convolutional neural networks into image super-resolution, constructing the SRCNN network model. This network consists of only three layers: a feature extraction layer, a nonlinear mapping layer, and a reconstruction layer. While this model achieved better results than traditional algorithms, it suffered from slow convergence and the need for retraining at different magnification levels. To address these issues, Dong et al. improved upon the SRCNN model, proposing the Fast-SRCNN (FSRCNN) model. FSRCNN first introduces deconvolution layers, reducing computation and input errors, and allows for different magnification levels by changing the deconvolution layers. Simultaneously, it employs a shrinking strategy to reduce feature dimensions, uses smaller convolutional kernels, and deepens the network to further reduce computation. FSRCNN achieves nearly 40 times faster training speed and superior reconstruction results compared to SRCNN. Luo et al. improved the SRCNN network model to perform super-resolution reconstruction of satellite video sequences, achieving excellent results. Ducournau et al. used an SRCNN network model with finely tuned intermediate layer parameters to perform super-resolution reconstruction of satellite-acquired Sea Surface Temperature (SST) maps for subsequent data processing. Shi et al. proposed an ESPCN network model based on pixel rearrangement, whose core is sub-pixel convolution. This model arranges the sub-pixel features obtained after convolution to fill in the missing sub-pixel information in the original low-resolution image. Kim et al. from the Computer Vision Lab at Seoul National University first introduced the idea of ​​residual learning into image super-resolution reconstruction methods. They constructed a 20-layer VDSR model using small convolutional kernels, learning the contextual information of the image by increasing the receptive field. Lai et al. from the University of California, Merced proposed a LapSRN model using a pyramid structure for single-image super-resolution reconstruction. This model can progressively reconstruct high-resolution images at different magnification scales, achieving not only super-resolution reconstruction at magnification factors of 2 and 4, but also satisfactory results at a magnification factor of 8. Existing deep learning-based image super-resolution methods directly train network models using low-resolution and high-resolution image samples, resulting in smooth but not sharp edges and blurred image details in the final super-resolution image.

[0004] Based on the above analysis, the existing technologies have the following problems and shortcomings: Current deep learning-based image super-resolution methods directly train the network model using low-resolution and high-resolution image samples, resulting in smooth but not sharp edges and blurred image details in the final super-resolution image. The key to solving this problem is to accurately extract image details and recover lost details through training the image detail layer, thereby enabling subsequent image processing tasks such as image segmentation and image recognition to proceed smoothly. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a layered image super-resolution method, system, computer device, and application.

[0006] This invention is implemented as follows: a layered image super-resolution method, the layered image super-resolution method comprising:

[0007] Using the DIV2K image dataset as the training set, the training samples were stratified based on the guided filter. The LR sample images were divided into detail images and smooth images respectively using the formula, and the HR sample images were divided into detail images and smooth images respectively, and a one-to-one correspondence was established. Two sample training sets were formed, one for detail images and one for smooth images.

[0008] Two deep residual convolutional networks, DLRCN and DGRCN, are designed for training on detailed images and smooth images, respectively.

[0009] The Deep Local Residual Convolutional Network (DLRCN) was trained using a detailed image sample training set, and the Deep Global Residual Convolutional Network (DGRCN) was trained using a smooth image sample training set.

[0010] The Deep Local Residual Convolutional Network (DLRCN) was trained using a detailed image test set, and the Deep Global Residual Convolutional Network (DGRCN) was trained using a smooth image test set to verify the accuracy and reliability of the proposed model.

[0011] Furthermore, the hierarchical image super-resolution method establishes a training set: using the image dataset DIV2K as the training set, 400 images are selected from DIV2K to construct the LR-HR training sample library.

[0012] Furthermore, the image layering method of the layered image super-resolution utilizes the structural changes of the guiding image to perform guided filtering on the input image. The final output image retains the overall features of the input image while capturing the details of the changes in the guiding image. The algorithm of the guiding filter is that the guiding image I and the filtered output image q are in a locally linear relationship, within a window W centered on pixel k. kIn this context, the output image q is a linear transformation of the guiding image I:

[0013] q i =a k I i +b k

[0014] In the formula: (a k ,b k ) is a constant. This linear model assumes that q has an edge if and only if the guiding image I has an edge, ▽q=a▽I. In order to find the linear coefficients (a k ,b k The input image p needs to be constrained; the output image q is obtained by subtracting the noise n from the input image p.

[0015] q i =p i -n i ;

[0016] To find the linear coefficients (a) k ,b k The optimal solution for q, which minimizes the difference between q and p, is equivalent to minimizing the window W. k The cost function in:

[0017]

[0018] Where λ is a regularization parameter less than 1, the formula is a linear regression model, and the optimal solution is:

[0019] b k =μ k -a k μ k ;

[0020] From the properties of the guided filter, we know that, according to equation q i =a k I i +b k The resulting output image q of the guided filter preserves the edges of the guided image I. Therefore, using equation I... enhance =K(Iq)+q performs detail enhancement on the output image q of the guided filter:

[0021] I enhance =K(Iq)+q;

[0022] Among them, I enhance This represents a detail-enhanced image, where K represents the detail magnification factor, which is used in this paper according to Equation I. enhance =K(Iq)+q divides image I into layers, specifically detail layers I.detail and smoothing layer I smooth The formula is as follows:

[0023] I detail =I enhance -I;

[0024] I smooth =2I-I enhance ;

[0025] I = I detail +I smooth ;

[0026] After being layered, the image is divided into two parts: the detail layer and the smoothing layer. That is, an image is composed of a detail image and a smoothing image. The detail image represents the high-frequency part of the image, and the smoothing image represents the low-frequency part of the image.

[0027] Furthermore, the model established by the layered image super-resolution method can divide an image into two parts, a detail image and a smooth image, through a guided filter. Based on the different characteristics of the detail image and the smooth image, different network structures are used to perform super-resolution reconstruction on the detail image and the smooth image respectively. The super-resolution results of these parts are then merged to obtain the super-resolution result of the entire image.

[0028] Deep Local Residual Convolutional Network (DLRCN) is used for super-resolution reconstruction of detail images. It is a local residual convolutional network, and its structure consists of three parts:

[0029] a) Feature extraction layer Conv+PReLU: 64 convolutional kernels with a kernel size of 3×3×c are used to generate 64 feature maps. PReLU is used to perform non-linear transformation on the feature maps to form a non-linear feature map. The stride is set to 1, c is the number of channels, the grayscale image has 1 channel, and the color image has 3 channels.

[0030] b) Local Residual Nonlinear Mapping Layer: The local residual nonlinear mapping layer consists of M residual convolutional blocks connected in series. Each residual block consists of Conv+BN+PReLU+Conv. The nonlinear mapping layer consists of M layers of residual blocks. Each layer contains 64 convolutional kernels of size 3×3 to filter 64 feature maps. Batch normalization (BN) is added between the convolution and PReLU to solve the problem of covariate transformation within the network. The stride is set to 1.

[0031] c) Deconvolutional layer: The network ends with a deconvolutional layer, which uses a set of deconvolutional filters to upsample the output residual image; the deconvolutional layer uses a 5×5 convolutional kernel with a stride of the magnification factor m.

[0032] Furthermore, the Depth Global Residual Convolutional Network (DGRCN) established by the hierarchical image super-resolution method is used for smooth image super-resolution reconstruction. As a global residual convolutional network, its structure also consists of three parts:

[0033] a) Feature extraction layer Conv+PReLU: 64 convolutional kernels with a kernel size of 3×3×c are used to generate 64 feature maps. PReLU is then used to perform a non-linear transformation on the feature maps to form a non-linear feature map. Here, the stride is set to 1, c is the number of channels, grayscale images have 1 channel, and color images have 3 channels.

[0034] b) Nonlinear mapping layer: The nonlinear mapping layer consists of M convolutional blocks (ConvBlock) connected in series. Each ConvBlock consists of Conv+BN+PReLU. Each nonlinear mapping layer contains 64 convolutional kernels of size 3×3 to filter 64 feature maps. Batch normalization (BN) is added between the convolution and PReLU, with a stride of 1.

[0035] c) Deconvolutional layer: The network ends with a deconvolutional layer. This layer uses a set of deconvolutional filters to upsample the output residual image. The deconvolutional layer uses a 5×5 convolutional kernel with a stride of the magnification factor m.

[0036] Furthermore, each layer of the Depth Local Residual Convolutional Network (DLRCN) and the Depth Global Residual Convolutional Network (DGRCN) uses PReLU as the activation function, and its output is:

[0037] f(x) = max(x,0) + amin(0,x);

[0038] The loss function used for training is not learned end-to-end from low-resolution image to high-resolution image. The high-resolution image y is composed of the low-resolution image x and the residual image v. The residual image v is essentially the detail image lost by the low-resolution image.

[0039] y = x + v;

[0040] Residual learning (RL) is used to train the residual mapping R(y)≈v. The mean squared error between the expected residual image and the predicted residual image is calculated by y=x+R(y). The loss function is defined as follows:

[0041]

[0042] In the formula, w and b are the parameters to be trained in the network. Given a pair of high-resolution and low-resolution images, to compute the minimum value of l(w,b), the proposed network model uses a stochastic gradient descent iterative optimization algorithm to update the parameter matrix w and parameter vector b, defined as follows:

[0043]

[0044] In the formula, λ represents the learning rate, l represents the network layer number where the parameters are located, and l(w,b) represents the target loss function. The expected network model is obtained by using the forward propagation and backward propagation algorithms of the neural network and the iterative rules.

[0045] Furthermore, the model training of the layered image super-resolution method adopts the idea of ​​local residual learning and skip connections to form an enhanced residual block structure. First, high-frequency features of the LR input image are extracted through convolutional layers. After every two convolutional layers, the feature image extracted by the previous convolutional layer is added, that is, the input of all identity branches in the residual block remains the same. A recursive structure is introduced into the residual block, so that the weights are shared between the residual blocks and the model is more compact. The receptive field of the convolutional neural network with a depth of d is (2d+1)×(2d+1). 3×3 small convolutional blocks are selected in each layer of the network. The depth of the deep local residual convolutional network is 20, and the depth of the deep global residual convolutional network is 16. 400 images are selected from DIV2K to construct LR-HR training image pairs. Using the image layering method proposed in this paper, the LR-HR training image pairs are divided into LR-HR detail image pairs and LR-HR smooth image pairs. LR-HR detail image pairs are used to train DLRCN, and LR-HR smooth image pairs are used to train DLRCN.

[0046] Model testing of the layered image super-resolution method: The test set uses four widely used benchmark image test sets Set5, Set14, Urban100 and BSD100, containing 10, 10, 20 and 20 images respectively. The test images are divided into detail images and smooth images, which are input into the trained DLRCN and DGRCN respectively. The two output images are merged to obtain the super-resolution image.

[0047] The DLRCN network for detail image super-resolution has a depth of 20 layers, and the DGRCN network for smoothing image super-resolution has a depth of 16 layers. Each convolutional layer uses 64 filters of size 3×3, and each deconvolutional layer uses filters of size 5×5. The stride is equal to the magnification factor m, the image batch size is 128, and the momentum is 0.9. The SGD optimization method with a weight decay coefficient of 0.0001 and a decay rate of 0.9 is used, and batch normalization is employed to transform the initial data distribution into a Gaussian distribution. Iterative training stops if the training error remains constant at a certain value for six consecutive epochs.

[0048] Another object of the present invention is to provide a computer device comprising a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the following steps:

[0049] Using the DIV2K image dataset as the training set, the training samples were stratified based on the guided filter. The LR sample images were divided into detail images and smooth images respectively using the formula, and the HR sample images were divided into detail images and smooth images respectively, and a one-to-one correspondence was established. Two sample training sets were formed, one for detail images and one for smooth images.

[0050] Two deep residual convolutional networks, DLRCN and DGRCN, are designed for training on detailed images and smooth images, respectively.

[0051] The Deep Local Residual Convolutional Network (DLRCN) was trained using a detailed image sample training set, and the Deep Global Residual Convolutional Network (DGRCN) was trained using a smooth image sample training set.

[0052] The Deep Local Residual Convolutional Network (DLRCN) was trained using a detailed image test set, and the Deep Global Residual Convolutional Network (DGRCN) was trained using a smooth image test set to verify the accuracy and reliability of the proposed model.

[0053] Another object of the present invention is to provide a layered image super-resolution system for implementing the layered image super-resolution method, the layered image super-resolution system comprising:

[0054] The training set creation module is used to use the image dataset DIV2K as the training set.

[0055] The image layering module is used to layer training samples based on guided filters;

[0056] The model building module is used to design two deep residual convolutional networks, DLRCN and DGRCN, for training on detailed images and smooth images, respectively.

[0057] The model training module is used to train the Deep Local Residual Convolutional Network (DLRCN) using a detailed image sample training set and the Deep Global Residual Convolutional Network (DGRCN) using a smooth image sample training set.

[0058] The model testing module is used to train the Deep Local Residual Convolutional Network (DLRCN) with a detailed image test sample set and the Deep Global Residual Convolutional Network (DGRCN) with a smooth image test sample set, respectively, to verify the accuracy and reliability of the proposed model.

[0059] Another objective of this invention is to provide an image processing terminal equipped with the aforementioned layered image super-resolution system; the image processing terminal is a medical image terminal, a remote sensing image terminal, an astronomical image terminal, an SRA radar image terminal, or a military night vision image terminal.

[0060] Combining all the above technical solutions, the advantages and positive effects of this invention are as follows: The image layering of this invention uses a novel edge-preserving filter: the guided image filter effectively smooths background details and preserves edge variations in the scene, while avoiding guided inversion effects. By utilizing the structural changes of the guided image, the input image is guided filtered, and the final output image retains both the overall features of the input image and fully captures the changing details of the guided image. Therefore, on the one hand, the guided image filter, like the bilateral filter, effectively preserves edges and smooths noise; on the other hand, under the influence of the guided image, this filter can make the output image have more edge information than the input image.

[0061] This invention establishes a model where a guided filter divides an image into two parts: a detail image and a smooth image. To improve the super-resolution effect, based on the different characteristics of the detail and smooth images, different network structures are used to perform super-resolution reconstruction on the detail and smooth images separately. Then, the super-resolution results of these parts are merged to obtain the super-resolution result of the entire image. As network depth increases, deep learning encounters serious problems such as gradient vanishing and gradient explosion, causing the network to fail to converge. Traditional solutions involve initializing and regularizing the data. While this solves the gradient problem, further problems arise with increasing depth: network performance degradation and increased error rate, meaning both training and testing accuracy decrease. Residual learning not only solves the network performance degradation problem but also the gradient vanishing problem, further improving network performance.

[0062] This invention introduces a recursive structure into residual blocks, enabling weight sharing among residual blocks and resulting in a more compact model. In this method, a batch normalization operation is introduced before each nonlinearization layer of the proposed deep learning network's nonlinear mapping layer. Then, normalization, scaling, and shift transformations are combined to address the issue of covariate transformation within the network. Each activation function only needs two parameters added during batch normalization, which are updated through backpropagation. This mechanism accelerates convergence during training, improves model accuracy, and reduces the sensitivity of model initialization. Residual networks exhibit excellent performance in low- to high-level computer vision tasks. The proposed method employs local residual learning and skip connections to construct an enhanced residual block structure. First, high-frequency features of the LR input image are extracted through convolutional layers. Then, after every two convolutional layers, the feature image extracted by the previous convolutional layer is added, ensuring that the inputs of all identity branches in the residual block remain the same. This method not only transmits more image information to deeper layers of the network, but its identity branches also facilitate backpropagation of gradients during training, avoiding overfitting. Increasing the number of network layers can improve network performance; however, as the number of layers increases, the number of network parameters also increases, leading to increased computational and storage requirements. To control model parameters, this invention introduces a recursive structure into the residual blocks, allowing weights to be shared among residual blocks and making the model more compact. A convolutional neural network of depth *d* has a receptive field of (2d+1)×(2d+1). Increasing the receptive field can fully exploit image contextual information, which helps in the recovery of lost information in the image, but this increases the computational burden on the network, requiring more computational resources and efficiency. Based on this, this invention uses 3×3 small convolutional blocks in each layer of the network, achieving efficient super-resolution by increasing the network depth without reducing the network's computational speed. The depth of the proposed depthwise local residual convolutional network is 20, and the depthwise global residual convolutional network is 16.

[0063] This invention employs a 20-layer DLRCN network for detail image super-resolution and a 16-layer DGRCN network for smoothing image super-resolution. Each convolutional layer uses 64 3×3 filters, and each deconvolutional layer uses a 5×5 filter. The stride is equal to the magnification factor n. The image batch size is 128, and the momentum is 0.9. An SGD optimization method with a weight decay coefficient of 0.0001 and a decay rate of 0.9 is used, and batch normalization is employed to transform the initial data distribution into a Gaussian distribution, accelerating convergence. Iterative training stops if the training error remains constant at a certain value for six consecutive epochs. Experiments show that the model is ultimately trained iteratively for 10 epochs. During training, the learning rate is initialized to 1e-1, and when the training error stops decreasing, it is fixed at 1e-4 for both training and testing.

[0064] This invention divides an image into a detail layer and a smoothing layer, and uses different depth neural network systems to perform super-resolution and fusion on the detail layer and the smoothing layer respectively, thereby improving image resolution and enhancing details. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart of the layered image super-resolution method provided in an embodiment of the present invention.

[0067] Figure 2 This is a schematic diagram of the structure of the layered image super-resolution system provided in an embodiment of the present invention;

[0068] Figure 2 The module consists of: 1. Training set creation module; 2. Image layering module; 3. Model creation module; 4. Model training module; 5. Model testing module.

[0069] Figure 3 This is a flowchart illustrating the implementation of the layered image super-resolution method provided in this embodiment of the invention.

[0070] Figure 4 This is a block diagram illustrating the algorithm principle of the guided filter provided in this embodiment of the invention.

[0071] Figure 5 This is an image layering result diagram provided in an embodiment of the present invention;

[0072] Figure 5 (a) Original image; (b) Detail image; (c) Smoothed image.

[0073] Figure 6 This is a diagram of the Deep Local Residual Convolutional Network (DLRCN) structure for super-resolution of detail layers provided in an embodiment of the present invention.

[0074] Figure 7 This is a diagram of the Deep Global Residual Convolutional Network (DGRCN) structure for super-resolution of smoothing layers provided in an embodiment of the present invention.

[0075] Figure 8 This is a diagram showing the butterfly processing results for method comparison provided in an embodiment of the present invention.

[0076] Figure 9This is a diagram showing the results of character processing for method comparison provided in an embodiment of the present invention. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0078] To address the problems existing in the prior art, the present invention provides a layered image super-resolution method, system, computer device, and application. The present invention will be described in detail below with reference to the accompanying drawings.

[0079] like Figure 1 As shown, the layered image super-resolution method provided by the present invention includes the following steps:

[0080] S101: Using the image dataset DIV2K as the training set, the training samples are stratified based on the guided filter. The LR sample images are divided into detail images and smooth images respectively using the formula, and the HR sample images are divided into detail images and smooth images respectively, and a one-to-one correspondence is established; thus forming two sample training sets, one for detail images and one for smooth images.

[0081] S102: Design two deep residual convolutional networks, DLRCN and DGRCN, for training on detailed images and smooth images, respectively;

[0082] S103: Train the Deep Local Residual Convolutional Network (DLRCN) using a detailed image sample training set, and train the Deep Global Residual Convolutional Network (DGRCN) using a smooth image sample training set.

[0083] S104: The Deep Local Residual Convolutional Network (DLRCN) was trained using a detailed image test sample set, and the Deep Global Residual Convolutional Network (DGRCN) was trained using a smooth image test sample set to verify the accuracy and reliability of the proposed model.

[0084] This invention establishes a training set: the DIV2K image dataset is used as the training set, which contains 800 training images covering almost all natural scenes: buildings, forests, lakes, animals, mountains, people, etc. 400 images are selected from DIV2K to construct the LR-HR training sample library.

[0085] The layered image super-resolution method provided by this invention can also be implemented using other steps by those skilled in the art. Figure 1 The layered image super-resolution method provided by this invention is merely one specific embodiment.

[0086] like Figure 2 As shown, the layered image super-resolution system provided by the present invention includes:

[0087] Training set building module 1 is used to use the image dataset DIV2K as the training set.

[0088] Image layering module 2 is used to layer training samples based on guided filters.

[0089] Model building module 3 is used to design two deep residual convolutional networks, DLRCN and DGRCN, for training on detail images and smooth images, respectively.

[0090] Model training module 4 is used to train the Deep Local Residual Convolutional Network (DLRCN) using a detailed image sample training set and the Deep Global Residual Convolutional Network (DGRCN) using a smooth image sample training set.

[0091] Model testing module 5 is used to train the Deep Local Residual Convolutional Network (DLRCN) with a detailed image test sample set and the Deep Global Residual Convolutional Network (DGRCN) with a smooth image test sample set, respectively, to verify the accuracy and reliability of the proposed model.

[0092] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0093] like Figure 3 As shown, the layered image super-resolution method provided by the present invention specifically includes the following steps:

[0094] (1) Image Layering: The guided image filter is a novel edge-preserving filter that effectively smooths background details and preserves edge variations in the scene without producing a guided inversion effect. By utilizing the structural changes of the guided image, the input image is guided filtered, and the final output image retains both the overall features of the input image and fully captures the details of the guided image's changes. Therefore, on the one hand, like the bilateral filter, the guided image filter effectively preserves edges and smooths noise; on the other hand, under the influence of the guided image, this filter can make the output image contain more edge information than the input image. The algorithm principle block diagram of the guided filter is shown below. Figure 4 As shown.

[0095] The key to guided filters is the assumption that the guided image I and the filtered output image q are locally linear. Within a window W centered at pixel k... k In this context, the output image q is a linear transformation of the guiding image I:

[0096] q i =a k I i +b k

[0097] In the formula: (a k ,b k The linear model assumes that q has an edge if and only if the guiding image I has an edge, because ▽q=a▽I. To find the linear coefficients (a… k ,b k The input image p to be filtered needs to be constrained. Assume the output image q is obtained by subtracting noise n from the input image p:

[0098] q i =p i -n i (2)

[0099] To find the linear coefficients (a) k ,b k The optimal solution for q, which minimizes the difference between q and p, is equivalent to minimizing the window W. k The cost function in:

[0100]

[0101] Where λ is a regularization parameter less than 1, the formula is a linear regression model, and its optimal solution is:

[0102] b k =μ k -a k μ k (4)

[0103] Based on the properties of the guided filter, the output image q obtained by equation (1) preserves the edges of the guided image I. Therefore, equation (5) can be used to enhance the details of the output image q of the guided filter:

[0104] I enhance =K(Iq)+q (5)

[0105] Among them, I enhance The image I represents a detail-enhanced image, and K represents the detail magnification factor. In this paper, k = 2 is chosen. Therefore, according to equation (5), image I can be divided into layers, namely detail layer I. detail and smoothing layer I smooth The formula is as follows:

[0106] I detail =I enhance -I (6)

[0107] I smooth =2I-I enhance (7)

[0108] I = I detail +I smooth (8)

[0109] After being layered, the image is divided into two parts: the detail layer and the smoothing layer. That is, an image is composed of a detail image and a smoothing image. The detail image represents the high-frequency part of the image, and the smoothing image represents the low-frequency part of the image. Figure 5 The image layering results are shown. The first row is the natural image, and the second row is the lung CT image.

[0110] (2) Model building: An image can be divided into two parts, detail image and smooth image, by using a guided filter. In order to improve the super-resolution effect of the image, according to the different characteristics of detail image and smooth image, this invention uses different network structures to perform super-resolution reconstruction on detail image and smooth image respectively, and then merges the super-resolution results of these parts to obtain the super-resolution result of the whole image.

[0111] The network architecture for detail image super-resolution in this invention integrates the characteristics and advantages of both VeryDeep Convolution Networks (VDSR) and Enhanced Deep Super-Resolution Networks (EDSR), establishing a structure as follows: Figure 6 The Deep Local Residual Convolutional Network (DLRCN) shown is used for super-resolution reconstruction of detail images. It is called a Local Residual Convolutional Network, and its structure consists of three parts:

[0112] a) Feature extraction layer Conv+PReLU: 64 convolutional kernels with a kernel size of 3×3×c are used to generate 64 feature maps. PReLU is then used to perform a non-linear transformation on the feature maps, thus forming a non-linear feature map. Here, the stride is set to 1, and c is the number of channels (1 channel for grayscale images and 3 channels for color images).

[0113] b) Local Residual Nonlinear Mapping Layer: The local residual nonlinear mapping layer consists of M residual convolutional blocks (ResidualBlocks) connected in series. The structure of the ResidualBlock is as follows: Figure 6 As shown, each ResidualBlock consists of Conv+BN+PReLU+Conv: the nonlinear mapping layer consists of M ResidualBlock layers, each containing 64 3×3 convolutional kernels to filter 64 feature maps. Batch normalization (BN) is added between the convolutions and PReLU to address the covariate transformation problem within the network. The stride is still set to 1.

[0114] c) Deconvolutional Layer (Deconv): The network ends with a deconvolutional layer, which uses a set of deconvolutional filters to upsample the output residual image. Unlike traditional interpolation methods, deconvolution is an upsampling kernel that can learn image features and can be considered the inverse operation of convolution. The deconvolutional layer uses a 5×5 convolutional kernel with a stride of the magnification factor m.

[0115] This invention provides a network structure for smoothing image super-resolution that combines the characteristics and advantages of both the DnCNN and VDSR network models, establishing a structure as follows: Figure 7 The Deep Global Residual Convolutional Network (DGRCN) shown is used for smoothing image super-resolution reconstruction. It is called a global residual convolutional network, and its structure consists of three parts:

[0116] a) Feature extraction layer Conv+PReLU: 64 convolutional kernels with a kernel size of 3×3×c are used to generate 64 feature maps. PReLU is then used to perform a non-linear transformation on the feature maps, thus forming a non-linear feature map. Here, the stride is set to 1, and c is the number of channels (1 channel for grayscale images and 3 channels for color images).

[0117] b) Nonlinear mapping layer: The nonlinear mapping layer consists of M concatenated convolutional blocks (ConvBlocks). The structure of ConvBlock is as follows: Figure 7 As shown, each ConvBlock consists of Conv+BN+PReLU: each non-linear mapping layer contains 64 3×3 convolutional kernels to filter 64 feature maps, and batch normalization (BN) is added between the convolutions and PReLU to address the covariate transformation problem within the network. Here, the stride is still set to 1.

[0118] c) Deconvolutional Layer (Deconv): The network ends with a deconvolutional layer, which uses a set of deconvolutional filters to upsample the output residual image. Unlike traditional interpolation methods, deconvolution is an upsampling kernel that can learn image features and can be considered the inverse operation of convolution. The deconvolutional layer uses a 5×5 convolutional kernel with a stride of the magnification factor m.

[0119] Both Deep Local Residual Convolutional Networks (DLRCN) and Deep Global Residual Convolutional Networks (DGRCN) use PReLU (Parametric Rectified LinearUnit) as the activation function in each layer, and their output is:

[0120] f(x)=max(x,0)+amin(0,x) (9)

[0121] Compared to the ReLU activation function, the PReLU activation function achieves higher accuracy with only a small increase in computation, and can avoid the dead features phenomenon caused by the ReLU activation function.

[0122] As network depth increases, deep learning encounters serious problems such as vanishing and exploding gradients, causing the network to fail to converge. Traditional solutions involve initializing and regularizing the data. While this addresses the gradient problem, further depth introduces other issues: performance degradation and increased error rates, meaning both training and testing accuracy decline. Residual learning, however, not only solves the performance degradation problem but also the vanishing gradient problem, ultimately improving network performance.

[0123] Therefore, the loss function used for training in this invention is not learned end-to-end from low-resolution image to high-resolution image, but rather learns the residual between low-resolution and high-resolution images, which is different from previous super-resolution reconstruction methods. Assume that the high-resolution image y is composed of the low-resolution image x and the residual image v, as shown in Equation (9). Here, the residual image v is essentially the detail image lost by the low-resolution image.

[0124] y = x + v (10)

[0125] Next, residual learning (RL) is used to train the residual mapping R(y)≈v, and then the mean squared error between the expected residual image and the predicted residual image is calculated using y=x+R(y). The loss function is defined as follows:

[0126]

[0127] In the formula, w and b are the parameters to be trained in the network. It consists of a high-resolution image and a low-resolution image pair. To calculate the minimum value of l(w,b), the network model proposed in this invention uses a stochastic gradient descent iterative optimization algorithm to update the parameter matrix w and the parameter vector b, defined as follows:

[0128]

[0129] In equation (12), λ represents the learning rate, l represents the network layer number where the parameter is located, and l(w,b) represents the target loss function. Using the forward propagation and backward propagation algorithm of the neural network and the iterative rule of equation (12), the expected network model can be trained.

[0130] Recently, residual networks have demonstrated excellent performance in computer vision tasks ranging from low-level to high-level. The method proposed in this invention employs the idea of ​​local residual learning and skip connections to construct an enhanced residual block structure. First, high-frequency features of the LR input image are extracted through convolutional layers. Then, after every two convolutional layers, the feature image extracted by the previous convolutional layer is added, meaning that the inputs of all identity branches in the residual block remain the same. This method not only transmits more image information to deeper layers of the network, but its identity branches also facilitate backpropagation of gradients during training, avoiding overfitting. While increasing the number of network layers can improve network performance, the number of network parameters also increases with the number of layers, leading to increased computational and storage requirements. To control model parameters, this invention introduces a recursive structure into the residual blocks, allowing weights to be shared among residual blocks and making the model more compact.

[0131] (3) Model Training: Stochastic Gradient Descent (SGD) is widely used in CNN training, but its performance is significantly affected by the shifting of internal covariates. While effective and simple, this algorithm is prone to internal covariate shifting during network model training, altering the distribution of nonlinear inputs and reducing training efficiency. Therefore, Batch Normalization (BN) was proposed to address this internal covariate shifting, overcoming the difficulties in training deep neural networks and effectively preventing gradient vanishing. BN adds normalization, scaling, and shifting operations before each nonlinear processing layer to mitigate internal covariate shifting, resulting in faster convergence, better performance, and less influence of initialization variables on the network. In this method, batch normalization is introduced before each nonlinear mapping layer in the proposed deep learning network, combined with normalization, scaling, and shifting operations to address the problem of internal covariate shifting. Each activation function only needs two parameters added during batch normalization, which are updated through backpropagation. This mechanism can accelerate the convergence speed during training, improve the accuracy of the model, and reduce the sensitivity of model initialization.

[0132] Residual networks exhibit excellent performance across low- to high-level computer vision tasks. The proposed method employs local residual learning (LR) and skip connections to construct an enhanced residual block structure. First, high-frequency features of the LR input image are extracted through convolutional layers. Then, after every two convolutional layers, the feature image extracted by the previous layer is added, ensuring that the inputs to all identity branches within the residual block remain identical. This method not only transmits more image information to deeper network layers, but its identity branches also facilitate backpropagation of gradients during training, preventing overfitting. While increasing the number of network layers can improve performance, it also increases the number of network parameters, leading to increased computational and storage requirements. To control model parameters, this invention introduces a recursive structure into the residual blocks, allowing weights to be shared among the blocks and resulting in a more compact model. A convolutional neural network of depth *d* has a receptive field of (2d+1)×(2d+1). Increasing the receptive field can fully extract contextual information from the image, which helps in the recovery of lost information. However, this increases the computational burden on the network, requiring more computational resources and efficiency. Therefore, this invention uses 3×3 small convolutional blocks in each layer of the network. Without reducing the network's computational speed, it achieves efficient super-resolution by increasing the network depth. The depth of the proposed depthwise local residual convolutional network is 20, and the depthwise global residual convolutional network is 16.

[0133] This invention uses the DIV2K image dataset as the training set, which contains 800 training images and 100 test images. The DIV2K dataset includes almost all natural scenes: buildings, forests, lakes, animals, mountains, people, etc. First, 400 images are selected from DIV2K to construct LR-HR training image pairs. Then, using the image layering method proposed in this paper, the LR-HR training image pairs are divided into LR-HR detail image pairs and LR-HR smooth image pairs. Thus, the LR-HR detail image pairs are used to train DLRCN, while the LR-HR smooth image pairs are used for DGRCN training.

[0134] (3) Model Testing: The test set uses four widely used benchmark image test sets: Set5, Set14, Urban100, and BSD100, which contain 10, 10, 20, and 20 images, respectively. The test set also uses the same method, dividing the test images into detail images and smoothed images, which are then input into the trained DLRCN and DGRCN, respectively. The two output images are then merged to obtain the super-resolution image.

[0135] This invention employs a 20-layer DLRCN network for detail image super-resolution and a 16-layer DGRCN network for smoothing image super-resolution. Each convolutional layer uses 64 3×3 filters, and each deconvolutional layer uses a 5×5 filter. The stride is equal to the magnification factor m. The image batch size is 128, and the momentum is 0.9. An SGD optimization method with a weight decay coefficient of 0.0001 and a decay rate of 0.9 is used, and batch normalization is employed to transform the initial data distribution into a Gaussian distribution, accelerating convergence. Iterative training stops if the training error remains constant at a certain value for six consecutive epochs. Experiments show that the model is ultimately trained iteratively for 10 epochs. During training, the learning rate is initialized to 1e-1, and when the training error stops decreasing, it is fixed at 1e-4 for both training and testing.

[0136] The technical effects of the present invention will be described in detail below with reference to experiments.

[0137] To verify the effectiveness and accuracy of the deep learning-based hierarchical image super-resolution method proposed in this invention, the method of this invention is compared and analyzed with four other super-resolution methods: Bicubic, LapSRN, VDSR, and DRCN. The results are as follows: Figure 8 and Figure 9 As shown, this invention compares two metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The method of this invention achieves the maximum value in both metrics, reaching the best super-resolution result.

[0138] In the description of this invention, unless otherwise stated, "a plurality of" means two or more; the terms "upper," "lower," "left," "right," "inner," "outer," "front end," "rear end," "head," "tail," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0139] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0140] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A layered image super-resolution method, characterized in that, The layered image super-resolution method includes: Using the DIV2K image dataset as the training set, the training samples were stratified based on the guided filter. The LR sample images were divided into detail images and smooth images respectively using the formula, and the HR sample images were divided into detail images and smooth images respectively, and a one-to-one correspondence was established. Two sample training sets were formed, one for detail images and one for smooth images. Two deep residual convolutional networks, DLRCN and DGRCN, are designed for training on detailed images and smooth images, respectively. The Deep Local Residual Convolutional Network (DLRCN) was trained using a detailed image sample training set, and the Deep Global Residual Convolutional Network (DGRCN) was trained using a smooth image sample training set. The Deep Local Residual Convolutional Network (DLRCN) was trained using a detailed image test sample set, and the Deep Global Residual Convolutional Network (DGRCN) was trained using a smooth image test sample set to verify the accuracy and reliability of the proposed model. The hierarchical image super-resolution method establishes a training set: using the image dataset DIV2K as the training set, 400 images are selected from DIV2K to construct the LR-HR training sample library; The layered image super-resolution method described above utilizes the structural changes of the guiding image to perform guided filtering on the input image. The final output image retains the overall features of the input image while capturing the details of the changes in the guiding image. The algorithm for the guiding filter is a locally linear model between the guiding image I and the filtered output image q, within a window W centered on pixel k. k In this context, the output image q is a linear transformation of the guiding image I: In the formula: (a k ,b k ) is a constant. This linear model assumes that q has an edge if and only if the guiding image I has an edge. To find the linear coefficients (a) k ,b k The input image p needs to be constrained; the output image q is obtained by subtracting the noise n from the input image p. q i =p i -n i ; To find the linear coefficients (a) k ,b k The optimal solution for q, which minimizes the difference between q and p, is equivalent to minimizing the window W. k The cost function in: Where λ is a regularization parameter less than 1, the formula is a linear regression model, and the optimal solution is: From the properties of the guided filter, we know that, according to equation q i =a k I i +b k The resulting output image q of the guided filter preserves the edges of the guided image I. Therefore, using equation I... enhance =K(Iq)+q performs detail enhancement on the output image q of the guided filter: I enhance =K(I-q)+q; Among them, I enhance This represents a detail-enhanced image, where K represents the detail magnification factor, according to Equation I. enhance =K(Iq)+q divides image I into layers, specifically detail layers I. detail and smoothing layer I smooth The formula is as follows: I detail =I enhance -I; AND smooth =2I-I enhance ; I=I detail +I smooth ; After being layered, an image is divided into two parts: a detail layer and a smoothing layer. That is, an image is composed of a detail image and a smoothing image. The detail image represents the high-frequency part of the image, and the smoothing image represents the low-frequency part of the image. The hierarchical image super-resolution method establishes a model that can divide an image into two parts, a detail image and a smooth image, through a guided filter. Based on the different characteristics of the detail image and the smooth image, different network structures are used to perform super-resolution reconstruction on the detail image and the smooth image respectively. The super-resolution results of these parts are then merged to obtain the super-resolution result of the entire image. Deep Local Residual Convolutional Networks (DLRCNs) are used for super-resolution reconstruction of detailed images. As a local residual convolutional network, the DLRCN network structure consists of three parts: a) Feature extraction layer Conv+PReLU: 64 convolutional kernels with a kernel size of 3×3×c are used to generate 64 feature maps. PReLU is used to perform non-linear transformation on the feature maps to form a non-linear feature map. The stride is set to 1, c is the number of channels, the grayscale image has 1 channel, and the color image has 3 channels. b) Local Residual Nonlinear Mapping Layer: The local residual nonlinear mapping layer consists of M residual convolutional blocks connected in series. Each residual block consists of Conv+BN+PReLU+Conv. The nonlinear mapping layer consists of M layers of residual blocks. Each layer contains 64 convolutional kernels of size 3×3 to filter 64 feature maps. Batch normalization (BN) is added between the convolution and PReLU to solve the problem of covariate transformation within the network. The stride is set to 1. c) Deconvolutional layer: The network ends with a deconvolutional layer, which uses a set of deconvolutional filters to upsample the output residual image; the deconvolutional layer uses a 5×5 convolutional kernel with a stride of the magnification factor m; The Depth Global Residual Convolutional Network (DGRCN) established by the hierarchical image super-resolution method is used for smooth image super-resolution reconstruction. The DGRCN network structure also consists of three parts: a) Feature extraction layer Conv+PReLU: 64 convolutional kernels with a kernel size of 3×3×c are used to generate 64 feature maps. PReLU is used to perform non-linear transformation on the feature maps to form a non-linear feature map. The stride is set to 1, c is the number of channels, the grayscale image has 1 channel, and the color image has 3 channels. b) Nonlinear mapping layer: The nonlinear mapping layer consists of M convolutional blocks ConvBlock connected in series. Each ConvBlock consists of Conv+BN+PReLU. Each nonlinear mapping layer contains 64 convolutional kernels of size 3×3 to filter 64 feature maps. Batch normalization (BN) is added between the convolution and PReLU, with a stride of 1. c) Deconvolutional layer: The network ends with a deconvolutional layer. This layer uses a set of deconvolutional filters to upsample the output residual image. The deconvolutional layer uses a 5×5 convolutional kernel with a stride of the magnification factor m.

2. The layered image super-resolution method as described in claim 1, characterized in that, The Depth Local Residual Convolutional Network (DLRCN) and the Depth Global Residual Convolutional Network (DGRCN) both use PReLU as the activation function in each layer, and their output is: f(x) = max(x,0) + amin(0,x); The loss function used for training is not learned end-to-end from low-resolution image to high-resolution image. The high-resolution image y is composed of the low-resolution image x and the residual image v. The residual image v is essentially the detail image lost by the low-resolution image. y = x + v; Residual learning (RL) is used to train the residual mapping R(y)≈v. The mean squared error between the expected residual image and the predicted residual image is calculated by y=x+R(y). The loss function is defined as follows: In the formula, w and b are the parameters to be trained in the network. Given a pair of high-resolution and low-resolution images, to compute the minimum value of l(w,b), the proposed network model uses a stochastic gradient descent iterative optimization algorithm to update the parameter matrix w and parameter vector b, defined as follows: In the formula, λ represents the learning rate, l represents the network layer number where the parameters are located, and l(w,b) represents the target loss function. The expected network model is obtained by using the forward propagation and backward propagation algorithms of the neural network and the iterative rules.

3. The layered image super-resolution method as described in claim 1, characterized in that, The hierarchical image super-resolution method employs local residual learning (LR) and skip connections in its model training to form an enhanced residual block structure. First, high-frequency features of the LR input image are extracted through convolutional layers. After every two convolutional layers, the feature image extracted by the previous layer is added, ensuring that the inputs of all identity branches in the residual block remain identical. A recursive structure is introduced into the residual block, allowing weights to be shared between residual blocks and making the model more compact. The receptive field of a convolutional neural network with depth d is (2d+1)×(2d+1). 3×3 small convolutional blocks are selected in each layer of the network. The depth of the local residual convolutional network is 20, and the depth of the global residual convolutional network is 16. 400 images are selected from DIV2K to construct LR-HR training image pairs. Using the image layering method of the hierarchical image super-resolution method, the LR-HR training image pairs are divided into LR-HR detail image pairs and LR-HR smooth image pairs. The LR-HR detail image pairs are used to train the DLRCN, while the DLRCN is trained using the LR-HR smooth image pairs. Model testing of the layered image super-resolution method: The test set uses four widely used benchmark image test sets Set5, Set14, Urban100 and BSD100, containing 10, 10, 20 and 20 images respectively. The test images are divided into detail images and smooth images, which are input into the trained DLRCN and DGRCN respectively. The two output images are merged to obtain the super-resolution image. The DLRCN network for detail image super-resolution has a depth of 20 layers, and the DGRCN network for smoothing image super-resolution has a depth of 16 layers. Each convolutional layer uses 64 filters of size 3×3, and each deconvolutional layer uses filters of size 5×5. The stride is equal to the magnification factor m, the image batch size is 128, and the momentum is 0.

9. The SGD optimization method with a weight decay coefficient of 0.0001 and a decay rate of 0.9 is used, and batch normalization is employed to transform the initial data distribution into a Gaussian distribution. Iterative training stops if the training error remains constant at a certain value for six consecutive epochs.

4. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the layered image super-resolution method according to any one of claims 1-3, comprising the following steps: Using the DIV2K image dataset as the training set, the training samples were stratified based on the guided filter. The LR sample images were divided into detail images and smooth images respectively using the formula, and the HR sample images were divided into detail images and smooth images respectively, and a one-to-one correspondence was established. Two sample training sets were formed, one for detail images and one for smooth images. Two deep residual convolutional networks, DLRCN and DGRCN, are designed for training on detailed images and smooth images, respectively. The Deep Local Residual Convolutional Network (DLRCN) was trained using a detailed image sample training set, and the Deep Global Residual Convolutional Network (DGRCN) was trained using a smooth image sample training set. The Deep Local Residual Convolutional Network (DLRCN) was trained using a detailed image test set, and the Deep Global Residual Convolutional Network (DGRCN) was trained using a smooth image test set to verify the accuracy and reliability of the proposed model.

5. A layered image super-resolution system implementing the layered image super-resolution method according to any one of claims 1 to 3, characterized in that, The layered image super-resolution system includes: The training set creation module is used to use the image dataset DIV2K as the training set. The image layering module is used to layer training samples based on guided filters; The model building module is used to design two deep residual convolutional networks, DLRCN and DGRCN, for training on detailed images and smooth images, respectively. The model training module is used to train the Deep Local Residual Convolutional Network (DLRCN) using a detailed image sample training set and the Deep Global Residual Convolutional Network (DGRCN) using a smooth image sample training set. The model testing module is used to train the Deep Local Residual Convolutional Network (DLRCN) with a detailed image test sample set and the Deep Global Residual Convolutional Network (DGRCN) with a smooth image test sample set, respectively, to verify the accuracy and reliability of the proposed model.

6. An image processing terminal, characterized in that, The image processing terminal is equipped with the layered image super-resolution system as described in claim 5; the image processing terminal is one of a medical image terminal, a remote sensing image terminal, an astronomical image terminal, an SRA radar image terminal, and a military night vision image terminal.

Images