Arbitrary scale image super-resolution method based on meta-learning and structure preservation

Abstract

The present invention provides a method for super-resolution of images of any scale based on meta-learning and structure preservation, comprising: constructing and training a super-resolution network model, preprocessing the acquired high-resolution image data to obtain low-resolution picture data; processing the low-resolution picture data to obtain gradient map data; performing feature extraction and upsampling on the low-resolution picture to obtain a sampled picture on a super-resolution branch network, performing feature extraction and upsampling on the gradient map to obtain a sampled picture on a gradient map branch network; inputting the sampled picture on the super-resolution branch network and the sampled picture on the gradient map branch network into a mixing module for weighted operation to reconstruct a super-resolution image; setting a loss function, iteratively training the super-resolution network model until the model converges, and obtaining a trained super-resolution network model of any scale; the super-resolution network model of the present invention can generate super-resolution SR images with good visual effects at any scale.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method for super-resolution of images of any scale based on meta-learning and structure preservation. Background Art

[0002] Image super-resolution (SR) methods are methods for generating high-resolution images from low-resolution images. As a classic computer vision problem, this problem has been widely studied in recent years. This direction is not only of theoretical research significance, but also has very wide applications in security monitoring, medical imaging, satellite and aerial imaging. With the development of deep learning, many super-resolution methods have been proposed, such as ESPCNN (Efficient Sub-pixel Convolutional Neural Network), EDSR (Enhanced Deep Residual Networks), and RDN (Residual Dense Network). Most of them are optimized by mean square error (MSE), which is used to measure the pixel distance between high-resolution images and high-pixel images. Due to the setting of the optimization target, the goal of this type of method is to make the image have a higher peak signal-to-noise ratio (PSNR), but this type of method has the problem of blurred image structure itself. As a result, some works aimed at restoring photo-realistic images have emerged, which use generative adversarial networks (GANs), such as Super Resolution Generative Adversarial Networks (SRGAN), EnhanceNet, and Natural Super Resolution (NatSR).

[0003] In addition, in real-world scenarios, it is very common for users to view images using custom magnifications, just like ordinary image viewers, which can zoom in and view images at will by sliding the scroll wheel. However, in traditional single image super-resolution (SISR) tasks, one magnification often corresponds to one model. Among the existing super-resolution methods, most methods must design a specific upsampling module for each scale factor. Each upsampling module can only use a fixed integer scale factor (×2, ×3, ×4, ...) to enlarge the image, so the super-resolution network model is only applicable to integer scale factors. If you want to achieve the effect of arbitrary magnification, it means that you need to train a specific model for each different magnification, but it is obviously unscientific to store so many models, and the computational efficiency will be very low. These shortcomings limit the use of SISR in real-world scenarios, so it is of great significance to solve a single SISR model that can be magnified at any scale.

[0004] The Chinese patent with publication number CN111583109A provides an image super-resolution method based on a generative adversarial network. The invention constructs an image super-resolution model based on a generative network model and a discriminative network model, uses a loss function to train the generative network model and the discriminative network model, obtains a test data set and inputs it into the trained generative network model, generates a super-resolution image, calculates the peak signal-to-noise ratio between the generated super-resolution image and the real high-resolution image, and evaluates the reconstruction quality of the image. This method uses a generative adversarial network to improve the performance of the network in reconstructing super-resolution images by optimizing the network structure, solving the image super-resolution problem. Although the high-resolution images generated by this method have good photo realism, they lack sharp edges and fine textures. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned existing image super-resolution technology, the present invention provides a method for arbitrary-scale image super-resolution based on meta-learning and structure preservation, which performs super-reconstruction on low-resolution images and retains more edge details of objects in the image when performing arbitrary-scale super-resolution processing on the image.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] The method for super-resolution of arbitrary-scale images based on meta-learning and structure preservation includes the following steps:

[0008] S1: construct a super-resolution network model, which includes a super-resolution branch network, a gradient map branch network, a hybrid module and an image reconstruction module;

[0009] The super-resolution branch network includes an SR branch network feature extraction module, an SR branch network arbitrary scale upsampling module, and an SR branch network output module. The output end of the SR branch network feature extraction module is connected to the SR branch network arbitrary scale upsampling module, and the output end of the SR branch network arbitrary scale upsampling module is connected to the SR branch network output module.

[0010] The gradient map branch network includes a gradient map branch network feature extraction module, a gradient map branch network arbitrary scale upsampling module, and a gradient map branch network output module. The output end of the gradient map branch network feature extraction module is connected to the gradient map branch network arbitrary scale upsampling module, and the gradient map branch network arbitrary scale upsampling module is connected to the gradient map branch network output module.

[0011] S2: Acquire high-resolution image data to be processed;

[0012] S3: preprocess the high-resolution image data to obtain low-resolution image data;

[0013] S4: Process the low-resolution image data to obtain gradient map data;

[0014] S5: The SR branch network feature extraction module extracts features from the low-resolution image data to obtain a feature map of the low-resolution image; the gradient map branch network feature extraction module extracts features from the gradient map data to obtain a feature map of the gradient map;

[0015] Using the constructed meta-learning network, we can obtain a sampling module at any scale;

[0016] The arbitrary scale upsampling module is added to the super-resolution branch network as the arbitrary scale upsampling module of the SR branch network, and the arbitrary scale upsampling module is added to the gradient map branch network as the arbitrary scale upsampling module of the gradient map branch network;

[0017] The meta-learning network includes a first fully connected layer, an activation layer, and a second fully connected layer. The output end of the first fully connected layer is connected to the activation layer, and the output end of the activation layer is connected to the second fully connected layer.

[0018] S6: The super-resolution branch network feature map is input into the SR branch network arbitrary scale upsampling module for upsampling, and a super-resolution branch network upsampled image is obtained; the gradient map branch network feature map is input into the gradient map branch network arbitrary scale upsampling module for upsampling, and a gradient map branch network upsampled image is obtained;

[0019] S7: The sampled image on the super-resolution branch network and the sampled image on the gradient map branch network are input into the mixing module for weighted operation, and the weighted operation result is input into the image reconstruction module for image reconstruction to obtain a super-resolution image;

[0020] S8: Set the loss function and iteratively train the super-resolution network model;

[0021] S9: Repeat S5-S8 until the model converges to obtain a trained super-resolution network model of any scale;

[0022] S10: Input the low-resolution image into the trained arbitrary-scale super-resolution network model to obtain an arbitrary-scale super-resolution image with structure preserved.

[0023] Wherein, in step S2, obtaining the high-resolution image data HR to be processed includes: selecting pictures in the existing public high-resolution image data set DIV2K data set as the high-resolution image data HR to be processed.

[0024] Preferably, in step S1, the construction process of the super-resolution network model includes: the backbone network of the super-resolution branch network is an enhanced super-resolution generative adversarial network ESRGAN; the backbone network of the gradient map branch network is constructed based on the dense residual block RRDB module; the super-resolution network model optimizer adopts an adaptive moment estimation ADAM optimizer; the last convolutional reconstruction layer of the original super-resolution branch network and the last convolutional reconstruction layer of the original gradient map branch network are removed; and the feature map extracted by the feature extraction module of some SR branch networks in the super-resolution branch network is input into the gradient map feature extraction module of the gradient map branch network.

[0025] Preferably, the feature maps extracted by some SR branch network feature extraction modules in the super-resolution branch network are input into the gradient map feature extraction module of the gradient map branch network, and the feature maps extracted by some SR branch network feature extraction modules include: starting from the first SR branch network feature extraction module, the feature maps extracted by the 5th, 10th, 15th, and 20th SR branch network feature extraction modules in the super-resolution branch network.

[0026] Preferably, in step S3, the step of preprocessing the high-resolution image data includes: downsampling all pictures in the high-resolution image data using a bicubic interpolation downsampling method to obtain downsampled image data; converting the downsampled image data into a tensor data structure and normalizing it; performing random data enhancement on the normalized downsampled image data to obtain a processed low-resolution LR picture, wherein the random data enhancement operation includes image mirroring and image rotation.

[0027] Preferably, in step S4, the step of processing the low-resolution image data to obtain gradient map data includes: performing calculation processing by a Sobel operator to obtain a gradient map.

[0028] Preferably, the step of obtaining a gradient map by performing calculation processing using a Sobel operator includes: the Sobel operator obtains a gradient map by calculating the difference between adjacent pixels on the low-resolution image, wherein the horizontal gradient value I of the pixel point (i, j) on the low-resolution image is x The vertical gradient value I y Respectively expressed as:

[0029] I x =I LR (i+1,j)-I LR (i-1,j),

[0030] I y =I LR (i,j+1)-I LR (i,j-1),

[0031] In the formula, i and j are pixels on the low-resolution image, I LR For low-resolution images, I LR (i,j) is the low-resolution image I LR The pixel value of the previous pixel (i, j);

[0032] make Then the low-resolution image I LR The gradient map is Where M(·) represents the Sobel operator and ||·||2 represents the 2-norm.

[0033] Preferably, in step S5, the super-resolution branch network continuously passes through multiple SR feature extraction modules to extract features one by one, thereby obtaining a SR branch network feature map;

[0034] The gradient map branch network continuously passes through multiple gradient map feature extraction modules to extract features one by one, and obtains a gradient map branch network feature map;

[0035] The steps of inputting the matrix of offset vector combinations into the constructed meta-learning network to obtain the sampling module of any scale include: coordinate mapping and weight prediction;

[0036] The coordinate mapping process is:

[0037] Assuming that the magnification of the low-resolution image is r times, the mapping relationship between the pixel coordinates on the low-resolution image and the super-resolution image to be generated is:

[0038]

[0039] Where (i, j) is the pixel coordinate on the low-resolution image, (i′, j′) is the pixel coordinate on the super-resolution image to be generated. is the floor function;

[0040] Among them, the weight prediction process is:

[0041] The offset vector v corresponding to the pixel point (i′, j′) ij As the prior information guiding the weight prediction, when the magnification is r, the calculation formula of the offset vector corresponding to the pixel point (i′, j′) is:

[0042]

[0043] In the formula, v ij is the offset vector corresponding to the pixel point (i′, j′);

[0044] The above offset vector v ij Combined into a matrix V of size HW×3, where H is the length of the super-resolution image to be generated, W is the width of the super-resolution image to be generated, and the matrix V expression is as follows:

[0045]

[0046] Input V into the constructed meta-learning network for weight prediction to obtain the sampling module W′ at any scale, which is expressed as follows:

[0047]

[0048] In the formula, represents the meta-learning network, θ is the meta-learning network Parameters.

[0049] Preferably, in steps S8 and S9, the step of setting a loss function and iteratively training the super-resolution network model includes: each time the super-resolution network model is iteratively trained, the loss function is set, and the iterative training is repeated until the super-resolution network model converges to obtain a trained super-resolution network model of any scale;

[0050] The loss function includes a super-resolution branch network loss function and a gradient map branch network loss function. The super-resolution branch network loss function performs iterative training on the super-resolution branch network, and the gradient map branch network loss function performs iterative training on the gradient map branch network.

[0051] Preferably, the super-resolution branch network loss function is based on the consideration of improving PSNR, and a super-resolution branch network pixel-by-pixel comparison loss function is set. The expression formula is as follows:

[0052]

[0053] In the formula, pix represents pixel, SR represents super-resolution image, and I LR Represents a low-resolution image, I HR represents a high-resolution image, sr(·) represents the output of the super-resolution branch network, and ||·||1 represents the 1-norm;

[0054] In the super-resolution branch network loss function, based on the consideration of improving the feature richness, a VGG (Visual Geometry Group) network model φ is set up, features are extracted through the pre-trained VGG network model φ, and the loss is further calculated for the features. A super-resolution branch network perception loss function is set, and its calculation formula is as follows:

[0055]

[0056] In the formula, is the perceptual loss function of the super-resolution branch network, per represents perception, φ i is the output of the i-th layer of the pre-trained VGG network model;

[0057] In the super-resolution branch network loss function, the network is built based on ESRGAN, so the following loss function is set, and its calculation formula is as follows:

[0058]

[0059]

[0060] In the formula, is the super-resolution branch network discriminator loss function, is the adversarial loss function of the super-resolution branch network, Adv represents adversarial, and D I (·) is the super-resolution image discriminator;

[0061] In the super-resolution branch network loss function, it is necessary to extract the gradient map and use different loss functions to calculate the loss. The calculation formula is as follows:

[0062]

[0063]

[0064]

[0065] In the formula, represents the pixel loss function of the super-resolution branch network gradient map, represents the super-resolution branch network gradient map discriminator loss function, represents the super-resolution branch network gradient map adversarial loss function, GM represents the gradient map;

[0066] The gradient map branch network loss function is the gradient map branch network pixel-by-pixel comparison loss function The loss function is expressed as follows:

[0067]

[0068] Preferably, the loss function expression of the super-resolution network model is as follows:

[0069]

[0070] In the formula, α SR represents the coefficient of the super-resolution branch network perception loss function, β SR Represents the coefficient of the pixel-by-pixel comparison loss function of the super-resolution branch network, γ SR Represents the coefficient of the super-resolution branch network adversarial loss function, Represents the coefficients of the pixel loss function of the super-resolution branch network gradient map, Represents the coefficient of the super-resolution branch network gradient map against the loss function, α G L represents the coefficient of the pixel-by-pixel comparison loss function of the gradient map branch network.

[0071] Among them, setting parameter α SR =1.0,β SR =0.01,γ SR =0.005, α G =0.5.

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] The present invention provides a method for super-resolution of images of any scale based on meta-learning and structure preservation. By removing the original convolutional reconstruction layers of the super-resolution branch network and the gradient map branch network, replacing the original upsampling module with an arbitrary scale upsampling module, and iteratively training the super-resolution network model in combination with the corresponding loss function, super-resolution reconstruction of images of any scale is achieved. In the case of a custom magnification scale, an image with more complete details is reconstructed, and an image with good photo realism is obtained, which solves the problem of sharp edges and fine textures after image reconstruction, and provides strong support for technical fields such as satellite images, video surveillance, and medical imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a schematic diagram of the process of the method of the present invention;

[0075] Figure 2 This is a schematic diagram of the application of a super-resolution network model in one embodiment of the present invention;

[0076] Figure 3 Schematic diagram of the structure of a meta-learning network in one embodiment of the present invention. DETAILED DESCRIPTION

[0077] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0078] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0079] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0080] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0081] Example 1

[0082] like Figure 1 , 2 As shown, the method for super-resolution of arbitrary-scale images based on meta-learning and structure preservation includes the following steps:

[0083] S1: construct a super-resolution network model, which includes a super-resolution branch network, a gradient map branch network, a hybrid module and an image reconstruction module;

[0084] The super-resolution branch network includes an SR branch network feature extraction module, an SR branch network arbitrary scale upsampling module, and an SR branch network output module. The output end of the SR branch network feature extraction module is connected to the SR branch network arbitrary scale upsampling module, and the output end of the SR branch network arbitrary scale upsampling module is connected to the SR branch network output module.

[0085] The gradient map branch network includes a gradient map branch network feature extraction module, a gradient map branch network arbitrary scale upsampling module, and a gradient map branch network output module. The output end of the gradient map branch network feature extraction module is connected to the gradient map branch network arbitrary scale upsampling module, and the gradient map branch network arbitrary scale upsampling module is connected to the gradient map branch network output module.

[0086] S2: Acquire high-resolution image data to be processed;

[0087] S3: preprocess the high-resolution image data to obtain low-resolution image data;

[0088] S4: Process the low-resolution image data to obtain gradient map data;

[0089] S5: The SR branch network feature extraction module extracts features from the low-resolution image data to obtain a feature map of the low-resolution image; the gradient map branch network feature extraction module extracts features from the gradient map data to obtain a feature map of the gradient map;

[0090] Using the constructed meta-learning network, we can obtain a sampling module at any scale;

[0091] The arbitrary scale upsampling module is added to the super-resolution branch network as the arbitrary scale upsampling module of the SR branch network, and the arbitrary scale upsampling module is added to the gradient map branch network as the arbitrary scale upsampling module of the gradient map branch network;

[0092] The meta-learning network includes a first fully connected layer, an activation layer, and a second fully connected layer. The output end of the first fully connected layer is connected to the activation layer, and the output end of the activation layer is connected to the second fully connected layer.

[0093] S6: The super-resolution branch network feature map is input into the SR branch network arbitrary scale upsampling module for upsampling, and a super-resolution branch network upsampled image is obtained; the gradient map branch network feature map is input into the gradient map branch network arbitrary scale upsampling module for upsampling, and a gradient map branch network upsampled image is obtained;

[0094] S7: The sampled image on the super-resolution branch network and the sampled image on the gradient map branch network are input into the mixing module for weighted operation, and the weighted operation result is input into the image reconstruction module for image reconstruction to obtain a super-resolution image;

[0095] S8: Set the loss function and iteratively train the super-resolution network model;

[0096] S9: Repeat S5-S8 until the model converges to obtain a trained super-resolution network model of any scale;

[0097] S10: Input the low-resolution image into the trained arbitrary-scale super-resolution network model to obtain an arbitrary-scale super-resolution image with structure preserved.

[0098] In step S2 of the present solution, obtaining the high-resolution image data to be processed includes: selecting images in the existing public high-resolution image dataset DIV2K dataset as the high-resolution image data to be processed.

[0099] Example 2

[0100] More specifically, in step S1, the construction process of the super-resolution network model includes: the backbone network of the super-resolution branch network is the enhanced super-resolution generative adversarial network ESRGAN; the backbone network of the gradient map branch network is constructed based on the dense residual block RRDB module; the super-resolution network model optimizer adopts the adaptive moment estimation ADAM optimizer; the last convolutional reconstruction layer of the original super-resolution branch network and the last convolutional reconstruction layer of the original gradient map branch network are removed; the feature map extracted by the feature extraction module of some SR branch networks in the super-resolution branch network is input into the gradient map feature extraction module of the gradient map branch network.

[0101] In the specific implementation process, ESRGAN is used to make the images processed by the super-resolution branch network have more realistic and natural textures; the backbone network of the gradient map branch network is built based on the dense residual block RRDB module to ensure the image visual quality of the gradient map branch network; the optimizer parameters are set to β1 = 0.9, β2 = 0.999, ε = 1×10 -8 , and the model learning rate is set to 0.01, which optimizes the computational efficiency of the super-resolution network model.

[0102] More specifically, the feature maps extracted by some SR branch network feature extraction modules in the super-resolution branch network are input into the gradient map feature extraction module of the gradient map branch network, and the feature maps extracted by some SR branch network feature extraction modules include: starting from the first SR branch network feature extraction module, the feature maps extracted by the 5th, 10th, 15th, and 20th SR branch network feature extraction modules in the super-resolution branch network.

[0103] More specifically, in step S3, the step of preprocessing the high-resolution image data includes: downsampling all pictures in the high-resolution image data using a bicubic interpolation downsampling method to obtain downsampled image data; converting the downsampled image data into a tensor data structure and normalizing it; performing random data enhancement on the normalized downsampled image data to obtain a processed low-resolution LR picture, wherein the random data enhancement operation includes image mirroring and image rotation.

[0104] More specifically, in step S4, the step of processing the low-resolution image data to obtain the gradient map data includes: performing calculation processing by a Sobel operator to obtain the gradient map;

[0105] Among them, the Sobel operator has the advantage of high computational efficiency.

[0106] More specifically, the step of obtaining a gradient map by performing calculation processing using a Sobel operator includes: the Sobel operator obtains a gradient map by calculating the difference between adjacent pixels on a low-resolution image, wherein the horizontal gradient value I of a pixel point (i, j) on the low-resolution image is x The vertical gradient value I y Respectively expressed as:

[0107] I x =I LR (i+1,j)-I LR (i-1,j),

[0108] I y =I LR (i,j+1)-I LR (i,j-1),

[0109] In the formula, i and j are pixels on the low-resolution image, I LR For low-resolution images, I LR (i,j) is the low-resolution image I LR The pixel value of the previous pixel (i, j);

[0110] make Then the low-resolution image I LR The gradient map is Where M(·) represents the Sobel operator and ||·||2 represents the 2-norm.

[0111] More specifically, in step S5, the super-resolution branch network continuously passes through multiple SR feature extraction modules, extracts features one by one, and obtains a SR branch network feature map;

[0112] The gradient map branch network continuously passes through multiple gradient map feature extraction modules to extract features one by one, and obtains a gradient map branch network feature map;

[0113] The steps of inputting the matrix of offset vector combinations into the constructed meta-learning network to obtain the sampling module of any scale include: coordinate mapping and weight prediction;

[0114] The coordinate mapping process is:

[0115] Assuming that the magnification of the low-resolution image is r times, the mapping relationship between the pixel coordinates on the low-resolution image and the super-resolution image to be generated is:

[0116]

[0117] Where (i, j) is the pixel coordinate on the low-resolution image, (i′, j′) is the pixel coordinate on the super-resolution image to be generated. is the floor function;

[0118] Among them, the weight prediction process is:

[0119] The offset vector v corresponding to the pixel point (i′, j′) ij As the prior information guiding the weight prediction, when the magnification is r, the calculation formula of the offset vector corresponding to the pixel point (i′, j′) is:

[0120]

[0121] In the formula, v ij is the offset vector corresponding to the pixel point (i′, j′);

[0122] The above offset vector v ij Combined into a matrix V of size HW×3, where H is the length of the super-resolution image to be generated, W is the width of the super-resolution image to be generated, and the matrix V expression is as follows:

[0123]

[0124] Input V into the constructed meta-learning network for weight prediction to obtain the sampling module W′ at any scale, which is expressed as follows:

[0125]

[0126] In the formula, represents the meta-learning network, θ is the meta-learning network Parameters.

[0127] Example 3

[0128] In steps S8 and S9, the step of setting a loss function and iteratively training the super-resolution network model includes: each time the super-resolution network model is iteratively trained, the loss function is set, and the iterative training is repeated until the super-resolution network model converges to obtain a trained super-resolution network model of any scale;

[0129] The loss function includes a super-resolution branch network loss function and a gradient map branch network loss function. The super-resolution branch network loss function performs iterative training on the super-resolution branch network, and the gradient map branch network loss function performs iterative training on the gradient map branch network.

[0130] More specifically, the super-resolution branch network loss function is based on the consideration of improving PSNR, and a super-resolution branch network pixel-by-pixel comparison loss function is set. The expression formula is as follows:

[0131]

[0132] In the formula, pix represents pixel, SR represents super-resolution image, and I LR Represents a low-resolution image, I HR represents a high-resolution image, sr(·) represents the output of the super-resolution branch network, and ||·||1 represents the 1-norm;

[0133] In the super-resolution branch network loss function, based on the consideration of improving the feature richness, a VGG network model φ is set up, features are extracted through the pre-trained VGG network model φ, and the loss is further calculated for the features. A super-resolution branch network perception loss function is set, and its calculation formula is as follows:

[0134]

[0135] In the formula, is the perceptual loss function of the super-resolution branch network, per represents perception, φ i is the output of the i-th layer of the pre-trained VGG network model;

[0136] In the super-resolution branch network loss function, the network is built based on ESRGAN, so the following loss function is set, and its calculation formula is as follows:

[0137]

[0138]

[0139] In the formula, is the super-resolution branch network discriminator loss function, is the adversarial loss function of the super-resolution branch network, Adv represents adversarial, and D I (·) is the super-resolution image discriminator;

[0140] In the super-resolution branch network loss function, it is necessary to extract the gradient map and use different loss functions to calculate the loss. The calculation formula is as follows:

[0141]

[0142]

[0143]

[0144] In the formula, represents the pixel loss function of the super-resolution branch network gradient map, represents the super-resolution branch network gradient map discriminator loss function, represents the super-resolution branch network gradient map adversarial loss function, GM represents the gradient map;

[0145] The gradient map branch network loss function is the gradient map branch network pixel-by-pixel comparison loss function The loss function is expressed as follows:

[0146]

[0147] More specifically, the loss function expression of the super-resolution network model is as follows:

[0148]

[0149] In the formula, α SR represents the coefficient of the super-resolution branch network perception loss function, β SR Represents the coefficient of the pixel-by-pixel comparison loss function of the super-resolution branch network, γ SR Represents the coefficient of the super-resolution branch network adversarial loss function, Represents the coefficients of the pixel loss function of the super-resolution branch network gradient map, Represents the coefficient of the super-resolution branch network gradient map against the loss function, α G L represents the coefficient of the pixel-by-pixel comparison loss function of the gradient map branch network;

[0150] Among them, in order to achieve better results in the super-resolution network model loss function, the parameter is set to α SR =1.0,β SR =0.01,γ SR =0.005, α G =0.5.

[0151] Further, such as Figure 3 As shown, the step of inputting the offset vector combination matrix into the meta-learning network for weight prediction in the present invention includes:

[0152] The offset vector matrix is ​​input into the meta-learning network to obtain an arbitrary scale upsampling module;

[0153] Among them, the meta-learning network includes the SR branch network fully connected layer, the activation layer, and the second fully connected layer. The offset vector matrix passes through the SR branch network fully connected layer, the activation layer, and the second fully connected layer of the meta-learning network in sequence to obtain an arbitrary scale upsampling module.

Claims

1. A method for super-resolution of images of any scale based on meta-learning and structure preservation, characterized in that: The following steps are involved: S1: constructing a super-resolution network model, wherein the super-resolution network model includes a super-resolution branch network, a gradient map branch network, a hybrid module and an image reconstruction module; The super-resolution branch network includes an SR branch network feature extraction module, an SR branch network arbitrary scale upsampling module, and an SR branch network output module. The output end of the SR branch network feature extraction module is connected to the SR branch network arbitrary scale upsampling module, and the output end of the SR branch network arbitrary scale upsampling module is connected to the SR branch network output module. The gradient graph branch network comprises a gradient graph branch network feature extraction module, a gradient graph branch network arbitrary scale upsampling module, and a gradient graph branch network output module, wherein the output end of the gradient graph branch network feature extraction module is connected to the gradient graph branch network arbitrary scale upsampling module, and the gradient graph branch network arbitrary scale upsampling module is connected to the gradient graph branch network output module; The construction process of the super-resolution network model includes: the backbone network of the super-resolution branch network is the enhanced super-resolution generative adversarial network ESRGAN; the backbone network of the gradient map branch network is constructed based on the dense residual block RRDB module; the super-resolution network model optimizer adopts the adaptive moment estimation ADAM optimizer; the last convolutional reconstruction layer of the original super-resolution branch network and the last convolutional reconstruction layer of the original gradient map branch network are removed; the feature maps extracted by some SR branch network feature extraction modules in the super-resolution branch network are input into the gradient map feature extraction module of the gradient map branch network; S2: Acquire high-resolution image data to be processed; S3: preprocess the high-resolution image data to obtain low-resolution image data; S4: Process the low-resolution image data to obtain gradient map data; S5: The SR branch network feature extraction module extracts features from the low-resolution image data to obtain a feature map of the low-resolution image; the gradient map branch network feature extraction module extracts features from the gradient map data to obtain a feature map of the gradient map; Using the constructed meta-learning network, we can obtain a sampling module at any scale; The arbitrary scale upsampling module is added to the super-resolution branch network as the arbitrary scale upsampling module of the SR branch network, and the arbitrary scale upsampling module is added to the gradient map branch network as the arbitrary scale upsampling module of the gradient map branch network; The meta-learning network includes a first fully connected layer, an activation layer, and a second fully connected layer, wherein the output end of the first fully connected layer is connected to the activation layer, and the output end of the activation layer is connected to the second fully connected layer; S6: The super-resolution branch network feature map is input into the SR branch network arbitrary scale upsampling module for upsampling, and a super-resolution branch network upsampled image is obtained; the gradient map branch network feature map is input into the gradient map branch network arbitrary scale upsampling module for upsampling, and a gradient map branch network upsampled image is obtained; S7: The sampled image on the super-resolution branch network and the sampled image on the gradient map branch network are input into the mixing module for weighted operation, and the weighted operation result is input into the image reconstruction module for image reconstruction to obtain a super-resolution image; S8: Set the loss function and iteratively train the super-resolution network model; S9: Repeat S5-S8 until the model converges to obtain a trained super-resolution network model of any scale; In the steps S8 and S9, the step of setting a loss function and iteratively training the super-resolution network model includes: each time the super-resolution network model is iteratively trained, the set loss function is used, and the iterative training is repeated until the super-resolution network model converges to obtain a trained super-resolution network model of any scale; The loss function includes a super-resolution branch network loss function and a gradient map branch network loss function, the super-resolution branch network loss function performs super-resolution branch network iterative training on the super-resolution branch network, and the gradient map branch network loss function performs gradient map branch network iterative training on the gradient map branch network; The super-resolution branch network loss function is based on the consideration of improving PSNR, and a super-resolution branch network pixel-by-pixel comparison loss function is set. The expression formula is as follows: In the formula, pix represents pixel, SR represents super-resolution image, and I LR Represents a low-resolution image, I HR represents a high-resolution image, sr(·) represents the output of the super-resolution branch network, and ||·||1 represents the 1-norm; In the super-resolution branch network loss function, based on the consideration of improving the feature richness, a VGG network model φ is set up, features are extracted through the pre-trained VGG network model φ, and the loss is further calculated for the features. A super-resolution branch network perception loss function is set, and its calculation formula is as follows: In the formula, is the perceptual loss function of the super-resolution branch network, per represents perception, φ i is the output of the i-th layer of the pre-trained VGG network model; In the super-resolution branch network loss function, the network is built based on ESRGAN, so the following loss function is set, and its calculation formula is as follows: In the formula, is the super-resolution branch network discriminator loss function, is the adversarial loss function of the super-resolution branch network, Adv represents adversarial, and D I (·) is the super-resolution image discriminator; In the super-resolution branch network loss function, it is necessary to extract the gradient map and use different loss functions to calculate the loss. The calculation formula is as follows: In the formula, = represents the pixel loss function of the super-resolution branch network gradient map, represents the super-resolution branch network gradient map discriminator loss function, represents the super-resolution branch network gradient map adversarial loss function, GM represents the gradient map; The gradient map branch network loss function is the gradient map branch network pixel-by-pixel comparison loss function The loss function is expressed as follows: The loss function expression of the super-resolution network model is as follows: In the formula, α SR represents the coefficient of the super-resolution branch network perception loss function, β SR Represents the coefficient of the pixel-by-pixel comparison loss function of the super-resolution branch network, γ SR Represents the coefficient of the super-resolution branch network adversarial loss function, Represents the coefficients of the pixel loss function of the super-resolution branch network gradient map, Represents the coefficient of the super-resolution branch network gradient map against the loss function, α G L represents the coefficient of the pixel-by-pixel comparison loss function of the gradient map branch network; S10: Input the low-resolution image into the trained arbitrary-scale super-resolution network model to obtain an arbitrary-scale super-resolution image with structure preserved.

2. The method for arbitrary-scale image super-resolution based on meta-learning and structure preservation according to claim 1, characterized in that: The feature maps extracted by some SR branch network feature extraction modules in the super-resolution branch network are input into the gradient map feature extraction module of the gradient map branch network, and the feature maps extracted by some SR branch network feature extraction modules include: starting from the first SR branch network feature extraction module, the feature maps extracted by the 5th, 10th, 15th, and 20th SR branch network feature extraction modules in the super-resolution branch network.

3. The method for arbitrary-scale image super-resolution based on meta-learning and structure preservation according to claim 1, characterized in that: In step S3, the step of preprocessing the high-resolution image data includes: downsampling all pictures in the high-resolution image data using a bicubic interpolation downsampling method to obtain downsampled image data; converting the downsampled image data into a tensor data structure and normalizing it; performing random data enhancement on the normalized downsampled image data to obtain a processed low-resolution picture, wherein the random data enhancement operation includes image mirroring and image rotation.

4. The method for arbitrary-scale image super-resolution based on meta-learning and structure preservation according to claim 1, characterized in that: In step S4, the step of processing the low-resolution image data to obtain gradient map data includes: performing calculation processing through a Sobel operator to obtain a gradient map.

5. The method for arbitrary-scale image super-resolution based on meta-learning and structure preservation according to claim 4, characterized in that: The step of obtaining a gradient map by calculating and processing with a Sobel operator includes: the Sobel operator obtains a gradient map by calculating the difference between adjacent pixels on the low-resolution image, wherein the horizontal gradient value I of the pixel point (i, j) on the low-resolution image x The vertical gradient value I y Respectively expressed as: I x =I LR (i+1,j)-I LR (i-1,j), I y =I LR (i,j+1)-I LR (i,j-1), Where i and j are pixels on a low-resolution image, I LR For low-resolution images, I LR (i,j) is the low-resolution image I LR The pixel value of the previous pixel (i, j); make Then the low-resolution image I LR The gradient map is Where M(·) represents the Sobel operator and ||·||2 represents the 2-norm.

6. The method for arbitrary-scale image super-resolution based on meta-learning and structure preservation according to claim 1, characterized in that: In step S5, the super-resolution branch network continuously passes through multiple SR feature extraction modules to extract features one by one, and obtains a SR branch network feature map; The gradient map branch network continuously passes through multiple gradient map feature extraction modules to extract features one by one, and obtains a gradient map branch network feature map; The step of obtaining a sampling module at any scale by using the constructed meta-learning network includes: coordinate mapping and weight prediction; The coordinate mapping process is: Assuming that the magnification of the low-resolution image is r times, the mapping relationship between the pixel coordinates on the low-resolution image and the super-resolution image to be generated is: Where (i, j) is the pixel coordinate on the low-resolution image, (i′, j′) is the pixel coordinate on the super-resolution image to be generated. is the floor function; Among them, the weight prediction process is: The offset vector v corresponding to the pixel point (i′, j′) ij As the prior information guiding the weight prediction, when the magnification is r, the calculation formula of the offset vector corresponding to the pixel point (i′, j′) is: In the formula, v ij is the offset vector corresponding to the pixel point (i′, j′); The above offset vector v ij Combined into a matrix V of size HW×3, where H is the length of the super-resolution image to be generated, W is the width of the super-resolution image to be generated, and the matrix V expression is as follows: Input V into the constructed meta-learning network for weight prediction to obtain the sampling module W′ at any scale, which is expressed as follows: In the formula, represents the meta-learning network, θ is the meta-learning network Parameters.

Images