Hyperspectral image super-resolution reconstruction method based on deep image prior

Abstract

The present invention provides a hyperspectral image super-resolution reconstruction method based on deep image priors. By adding an automatic processing network that is not based on training to the existing main body learning network, the input of the network is adjusted according to the main structure of the network by designing an input processing module, so that the input resonates with the network structure, thereby making full use of the image prior information, using the intrinsic characteristics of DCNN to first capture the image, and then restore the image. The network structure proposed by the present invention makes full use of the spatial information of the hyperspectral image and the correlation between the spectral bands to learn the image features, and the input processing module in the automatic processing network of the present invention can automatically adjust the input structure, which can greatly expand the application of the DIP algorithm, make the DIP method more suitable for hyperspectral super-resolution tasks, and further improve the quality of the reconstructed image.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and image processing, and in particular relates to a hyperspectral image super-resolution reconstruction method based on depth image prior. Background Art

[0002] Hyperspectral images have rich spectral and spatial information and are advantageous in capturing details of target objects. They have a wide range of applications, such as observations in land, atmosphere, ocean, and other fields. However, due to the limitation of incident energy, in the design of optical remote sensing systems, if other factors remain unchanged and a high signal-to-noise ratio (SNR) is guaranteed, the spatial resolution decreases as the spectral features increase. The hyperspectral image super-resolution reconstruction technology aims to reconstruct a high-resolution hyperspectral image from a low-resolution hyperspectral image.

[0003] The existing technology proposes learning methods for super-resolution of hyperspectral images, including classic interpolation-based methods, fusion-based methods and the latest deep learning-based methods. Interpolation-based methods directly use neighborhood pixel information and cannot bring more information, so they cannot produce satisfactory results. Fusion methods based on Bayesian reasoning and sparse representation fuse the observed low-resolution hyperspectral images with auxiliary images (such as panchromatic images, RGB images and multispectral images) to improve the quality of low-resolution images. Good image registration is required in these methods, which seriously restricts the application of fusion-based methods in the real world.

[0004] With the rapid development of convolutional neural networks (CNNs), hyperspectral image super-resolution methods based on deep learning have recently flourished. Due to the powerful nonlinear mapping ability of convolutional neural networks, it has gradually become an important technology in the field of hyperspectral image processing and has achieved remarkable success. However, unlike traditional grayscale / RGB images, it is difficult to build a unified deep neural network due to the internal characteristics of hyperspectral images, the differences in imaging conditions and the number of spectral bands between different datasets. In addition, the lack of training images has hindered the research progress of hyperspectral image processing, and the super-resolution task inevitably faces the following challenges: 1) It is difficult to obtain prior information of hyperspectral images, which usually contain hundreds of continuous spectral bands. Therefore, in order to avoid the overfitting problem, it is necessary to learn the real prior image from a large number of sample images. However, compared with natural images, the number of available hyperspectral image datasets is small. 2) Considering the different imaging conditions and the different number of hyperspectral bands, it is still a difficult problem to build a unified deep neural network.

[0005] Ulyanov et al. proposed a non-training-based automatic image processing network in the paper "D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Deep image prior. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018." (hereinafter referred to as Deep image prior DIP algorithm). This method proves that most of the image statistics are captured by the structure of the generator CNN, which is independent of the learning process. This means that both the learning ability and the internal structure of CNN are conducive to the processing of image data. In addition, the structure itself can be used to obtain and process image statistics without learning external images.

[0006] However, directly applying this algorithm to HSISR will also cause some problems:

[0007] 1) The proposed structure is based on the traditional grayscale / RGB image super-resolution task, so this method cannot achieve satisfactory results when processing high-dimensional hyperspectral images;

[0008] 2) Experiments show that this method is affected by the network structure and the input image, and it is necessary to repeatedly adjust the network structure and network input to make the two match. Summary of the invention

[0009] In order to solve the above problems existing in the prior art, the present invention provides a hyperspectral image super-resolution reconstruction method based on depth image prior. The technical problem to be solved by the present invention is achieved by the following technical solutions:

[0010] The present invention provides a hyperspectral image super-resolution reconstruction method based on depth image prior, comprising:

[0011] Step 1: Obtain the image to be super-resolved and the subject learning network;

[0012] Step 2: Crop the edges of the image to be super-resolved to obtain the central area image;

[0013] Step 3: Evenly divide the first position area of ​​the central area image to obtain non-overlapping test image samples and extract them in the second position area to obtain sub-image blocks to be super-resolved;

[0014] Wherein, the first location area and the second location area do not overlap;

[0015] Step 4: Connect the output of the constructed automatic processing network to the input of the main learning network to form a super-resolution reconstruction network;

[0016] Among them, the automatic processing network includes: an input processing module and a main body recovery module. The input processing module has a first convolution layer, three second convolution layers and an upsampling layer with the same number of pixels as the input image, and the main body recovery module has a second convolution layer, an upsampling layer, an attention module and four second convolution layers, and the main processing network includes a trunk, a staggered jump channel and a combination channel. The trunk is composed of the output of the input processing module connected to the input of the main body recovery module until the output of the main body recovery module. The combination channel is composed of the output of the first convolution layer of the input processing module directly connected to the input of the main body recovery module. The staggered jump channel is composed of the first convolution layer of the input processing module directly connected to the output of the last second convolution layer of the main body recovery module.

[0017] Step 5: Input white noise of the same size as the sub-image block to be super-resolved into the super-resolution reconstruction network, and adjust the parameters of the super-resolution reconstruction network through back propagation to reduce the loss function until the optimal number of iterations, so that the super-resolution reconstruction network can learn the characteristics of the sub-image block to be super-resolved according to the correlation between the spatial information of the sub-image block to be super-resolved and the spectral band, and obtain the reconstructed hyperspectral image.

[0018] The first position area is the left half area of ​​the central area image, and the second position area is the right half area of ​​the central area image. The size of the central area image is 1096×715×102, the size of the test image sample and the sub-image block to be super-resolved is 223×223, the size of the first convolution layer is 223×223, and the size of the second convolution layer is 3×3.

[0019] Among them, each convolution layer in the automatic processing network is connected to an activation layer, and the activation function of the activation layer is:

[0020]

[0021] Among them, x represents the parameters after the previous convolution layer, which represents the slope in the negative direction;

[0022] The loss function of the automatic processing network is expressed as:

[0023] E(x;x0)=||d(f θ (IPM(Z)))-x0||1

[0024] Among them, x0 represents the low-quality image to be super-resolved, x represents the output image, and d(f θ(IPM(Z))) represents the input white noise Z which is input into the super-resolution reconstruction network and then downsampled after being output by the main learning network.

[0025] Optionally, the first convolution layer in the input processing module is used to compress the input sample image;

[0026] The compressed data transmitted by the combined channel and the data sampled by the upsampling layer are input into the main recovery module after dimension combination.

[0027] The compressed data transmitted by the combined channel is represented as:

[0028] The sampling data of the upsampling layer is expressed as:

[0029] The output of the automatic processing network is expressed as: Z * =F IPM (Z) = Z p +H cp (Z ca )

[0030] Among them, Z p Represents the compressed data after compression by the first convolutional layer in the input processing module.

[0031] Optionally, the optimal number of iterations is obtained through multiple experiments, the experimental process includes:

[0032] Set the number of experimental iterations according to the parameter scale of the super-resolution reconstruction network;

[0033] Obtain high-definition sub-images to form an experimental set;

[0034] Using a white noise input super-resolution reconstruction network of the same size as the experimental set to train the super-resolution reconstruction network until the number of experimental iterations is reached, recording the parameter values ​​of the super-resolution reconstruction network at each iteration and storing the reconstructed image output by the super-resolution reconstruction network every first number of iterations;

[0035] Compare each stored image with the high-definition sub-image to determine the reconstructed image with the highest definition;

[0036] According to the experimental set, determine the parameter values ​​and experimental iteration times corresponding to the reconstructed image with the highest clarity output by the super-resolution reconstruction network;

[0037] The experimental iteration number corresponding to the reconstructed image with the highest clarity is determined as the optimal iteration number for training the super-resolution reconstruction network.

[0038] Optionally, step 5 includes:

[0039] Step 51: inputting white noise of the same size as the sub-image block to be super-resolved into a super-resolution reconstruction network to obtain an output image;

[0040] Step 52: Calculate the loss function of the downsampled output image and the supramolecular image block to be processed;

[0041] Step 53: adjusting the parameters of the super-resolution reconstruction network by back-propagation to train the super-resolution reconstruction network until the optimal number of iterations is reached to obtain a reconstructed sub-block;

[0042] Step 54: Integrate the reconstructed sub-blocks to obtain a reconstructed hyperspectral image.

[0043] The present invention provides a hyperspectral image super-resolution reconstruction method based on deep image prior, by adding an automatic processing network that is not based on training to the existing main learning network, and by designing the automatic processing network to fully adjust the network input according to the main structure of the network, so that the input resonates with the network structure, thereby making full use of the image prior information, using the intrinsic characteristics of DCNN to first capture the image, and then restore the image. The network structure proposed by the present invention makes full use of the spatial information of the hyperspectral image and the correlation between the spectral bands to learn the image features, and the automatic processing network of the present invention can automatically adjust the input structure, which can greatly expand the application of the DIP algorithm, make the DIP method more suitable for hyperspectral super-resolution tasks, and further improve the quality of the reconstructed image.

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of a method for super-resolution reconstruction of a hyperspectral image based on a depth image prior provided by an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the internal structure of an automatic processing network provided by an embodiment of the present invention;

[0047] Figure 3 This is a comparison diagram of the 4-fold reconstructed image on the PAVIA CENTRE database using the method of the present invention and the existing super-resolution reconstruction method. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0049] like Figure 1 As shown, the present invention provides a hyperspectral image super-resolution reconstruction method based on depth image prior, comprising:

[0050] Step 1: Obtain the image to be super-resolved and the subject learning network;

[0051] Step 2: Crop the edges of the image to be super-resolved to obtain the central area image;

[0052] It is worth mentioning that the low-quality images to be super-resolved often have dozens or hundreds of bands, and the overall operation requires a lot of computing space. Usually, the image to be super-resolved is first cut into blocks, and the corresponding low-quality image blocks are processed separately.

[0053] Step 3: Evenly divide the first position area of ​​the central area image to obtain non-overlapping test image samples and extract them in the second position area to obtain sub-image blocks to be super-resolved;

[0054] Among them, there is no overlap between the first position area and the second position area; the first position area is the left half area in the central area image, and the second position area is the right half area in the central area image. The size of the central area image is 1096×715×102, and the size of the test image sample and the low-quality image block to be super-resolved is 223×223.

[0055] It is worth noting that: in a small data set like PAVIA, there is only one image. The present invention cuts the image to obtain a sub-image, namely the central area image. A part of it is used as a test, and the remaining sub-images are the same size as the training set of VDSR. The central area of ​​the PAVIA image is cropped to obtain a sub-image of size 1096×715×102, and the sub-image is further divided into data to be super-resolved and test data. Among them, the left part of the image is extracted as test data, and there are 4 non-overlapping image blocks in the test data with a pixel size of 223×223. In addition, the test data is used to verify the effect of the network, and the number of iterations required for the network to iterate to form the best image can be determined based on this, and the super-resolved data is subjected to the same number of super-resolved iterations as mentioned above to form the corresponding super-resolved image sub-blocks.

[0056] Step 4: Connect the output of the constructed automatic processing network to the input of the main learning network to form a super-resolution reconstruction network;

[0057] Among them, the automatic processing network includes: an input processing module and a main body recovery module. The input processing module has a first convolution layer, three second convolution layers and an upsampling layer with the same number of pixels as the input image, and the main body recovery module has a second convolution layer, an upsampling layer, an attention module and four second convolution layers, and the main processing network includes a trunk, a staggered jump channel and a combination channel. The trunk is composed of the output of the input processing module connected to the input of the main body recovery module until the output of the main body recovery module. The combination channel is composed of the output of the first convolution layer of the input processing module directly connected to the input of the main body recovery module. The staggered jump channel is composed of the first convolution layer of the input processing module directly connected to the output of the last second convolution layer of the main body recovery module.

[0058] refer to Figure 2 , assuming that the input size is A×A white noise image, a hyperspectral deep learning neural network is established. The automatic processing network consists of an input processing module and a main recovery module. The input processing module contains a first convolution layer A×A with the same size as the number of image pixels and three 3×3 convolution layers. The size of the first convolution layer is 223×223, the size of the second convolution layer is 3×3, and a bilinear interpolation upsampling layer. Each convolution layer contains a PReLU activation layer, where the PReLU activation is defined as:

[0059]

[0060] Among them, x represents the parameters after the previous convolution layer, which represents the slope in the negative direction. In addition, it also includes a connection operation, which compresses the input image to 1 layer of channels and connects it to the end of the input processing module. The formula is defined as follows:

[0061] Where H is a series of convolution operations with extended width (increased number of channels), It is the input of the channel after convolution, input to the upsampling module, and output Z up , is the output after convolution, and the two are finally connected together in the channel dimension and input into the main recovery module, where

[0062] For the main recovery module, a 3×3 convolutional layer is first used for shallow feature extraction. Secondly, the attention channel mechanism is used to make full use of the spatial information of the hyperspectral image and the correlation between the spectral bands. In addition, there are 4 3×3 convolutional layers, each of which contains a PReLU activation layer after the convolution, plus staggered jump connections, to achieve the final function of adjusting the input and generate an optimized input Z that matches the subsequent main network. * .

[0063] The first convolutional layer in the input processing module is used to compress the input noisy image;

[0064] The compressed data transmitted by the combined channel and the data sampled by the upsampling layer are input into the main recovery module after dimension combination.

[0065] The compressed data transmitted by the combined channel is represented as:

[0066] The sampling data of the upsampling layer is expressed as:

[0067] The output of the automatic processing network is expressed as: Z * =F IPM (Z) = Z p +H cp (Z ca )

[0068] Among them, Z p Represents the compressed data after compression by the first convolutional layer in the input processing module.

[0069] It is worth noting that the spatial information dimension of the image to be super-resolved is 1096 times 715 times 102 channels. In the automatic processing network, it will be adjusted in three different dimensions of pixel level, feature block, and channel layer to fully make the feature map output by the input noise image after the automatic processing network fully adapt to the subsequent main learning network, so that the quality of the reconstructed image is greatly improved. The first convolution layer will assign a trainable weight to each pixel point, and will freely adjust each pixel point value during the iteration. This is an adjustment at the pixel level. Secondly, the feature map undergoes two upsampling structures to increase the dimension and is finally compressed by the last three convolution layers. After the process of stretching and compression, the overall feature map is readjusted with a size of 3×3 feature blocks as the processing unit. Finally, the attention module part plays a role in allocating weights in the channel dimension and adjusting each channel during the iteration. In normal network training or iteration, such a combination of three structures is rare or can be said to be non-existent, but since this algorithm is an algorithm that automatically generates super-resolved images through iterations and outputs high-definition images from noisy images, it is different from conventional methods. The present invention requires a more free processing transformation of the output image in advance, so that the meaningless noise image pixels can generate meaningful high-definition images after being mapped by the network. In addition, the automatic processing network gives the output image a greater degree of freedom to adapt to the subsequent subject learning network, making the choice of the subsequent subject super-resolution network structure more free.

[0070] Step 5: Input white noise of the same size as the sub-image block to be super-resolved into the super-resolution reconstruction network, and adjust the parameters of the super-resolution reconstruction network through back propagation to reduce the loss function until the optimal number of iterations, so that the super-resolution reconstruction network learns the characteristics of the image according to the correlation between the spatial information of the sub-image block to be super-resolved and the spectral band, and obtains the reconstructed hyperspectral image.

[0071] It is worth noting that the present invention is a super-resolution method that is not based on training and does not require real images. When random noise is first input into the network, the network does not undergo iterative parameter updates, so the network parameters are still initial. Of course, the network cannot output meaningful images. The present invention uses the L1 loss function to measure the difference between the reconstructed output image and the low-quality image x0 to be super-resolved. After back propagation, the generated image will gradually approach x0. In this process, the network will resist noise and iterate faster to an image that looks natural. In this process, a higher quality image can be reconstructed.

[0072] The loss function used in the present invention is the L1 loss function, which is as follows:

[0073] E(x;x0)=||d(f θ (IPM(Z)))-x0||1

[0074] Among them, x0 represents the low-quality image to be super-resolved, x represents the output image, and d(f θ (IPM(Z))) represents the input white noise Z which is input into the super-resolution reconstruction network and then downsampled after being output by the main learning network.

[0075] As an optional implementation manner of the present invention, the number of iterations is obtained through multiple experiments, and the experimental process includes:

[0076] Set the number of experimental iterations according to the parameter scale of the super-resolution reconstruction network;

[0077] Obtain high-definition sub-images to form an experimental set;

[0078] Using a white noise input super-resolution reconstruction network of the same size as the experimental set to train the super-resolution reconstruction network until the number of experimental iterations is reached, recording the parameter values ​​of the super-resolution reconstruction network at each iteration and storing the reconstructed image output by the super-resolution reconstruction network every first number of iterations;

[0079] Compare each stored image with the high-definition sub-image to determine the reconstructed image with the highest definition;

[0080] According to the experimental set, determine the parameter values ​​and experimental iteration times corresponding to the reconstructed image with the highest clarity output by the super-resolution reconstruction network;

[0081] The experimental iteration number corresponding to the reconstructed image with the highest clarity is determined as the optimal iteration number for training the super-resolution reconstruction network.

[0082] It is worth noting that the present invention determines the optimal number of iterations as the number of iterations required for super-resolution of the image to be super-resolved, performs super-resolution on each molecular image block to be super-resolved, and finally integrates them.

[0083] It is worth noting that: by selecting a suitable high-definition sub-image, the method of the present invention is not easy to overfit after multiple iterations, and the reconstructed image quality is not easy to degrade, which is related to the number of iterations. Under experimental conditions, 15,000 iterations are preset to observe the trend of each indicator. In the process of determining the optimal number of iterations through the experimental set, the present invention will store its pictures every 50 times, and select the picture record corresponding to the best indicator. The image quality represented by each indicator has an overall downward trend after exceeding the optimal number of iterations, and gradually approaches the corresponding low-quality image in the later stage, so it is necessary to stop the iteration when the optimal number of iterations is reached, and record the current number of iterations according to the best indicator obtained in the experimental set in the experiment. The number of iterations is close to the optimal stop iteration number of the network in practical applications. The present invention can set a number that is 1000 greater than the stop iteration number to conduct experiments. Finally, the reconstructed image with the best performance is selected by observation.

[0084] In actual scenarios, 15,000 iterations is a suitable number. Since the network of the present invention is not easy to fit, it has strong robustness in practical applications. Even after 20,000 iterations, all indicators are still good enough. After multiple iterations, the reconstructed image will gradually become clear. The images generated in each step are saved, and the best results are selected according to the clarity, and finally a series of clear sub-images are obtained.

[0085] As an optional implementation of the present invention, step 5 includes:

[0086] Step 51: inputting white noise having the same size as the supramolecular image block into a super-resolution reconstruction network to obtain an output image;

[0087] Step 52: Calculate the loss function of the downsampled output image and the supramolecular image block to be processed;

[0088] Step 53: adjusting the parameters of the super-resolution reconstruction network by back-propagation to train the super-resolution reconstruction network until the optimal number of iterations is reached to obtain a reconstructed sub-block;

[0089] Step 54: Integrate the reconstructed sub-blocks to obtain a reconstructed hyperspectral image.

[0090] It is worth noting that the present invention needs to integrate the super-resolved sub-images to form an image, and integrate each super-resolved sub-block into the final super-resolved hyperspectral image.

[0091] It is worth noting that: due to the complex structure of the image data to be super-resolved, it usually takes up a lot of memory. The commonly used processing method is to cut it into blocks and super-resolve them separately to reduce the pressure of the network to process data, and finally integrate them together to obtain the super-resolved image.

[0092] The present invention provides a method for super-resolution reconstruction of an image to be super-resolved based on deep image priors. By adding an automatic processing network that is not based on training to an existing main body learning network, the input of the network is adjusted according to the main structure of the network by designing an input processing module, so that the input resonates with the network structure, thereby making full use of image prior information, using the intrinsic characteristics of DCNN to first capture the image, and then restore the image. The network structure proposed by the present invention makes full use of the spatial information of the hyperspectral image and the correlation between the spectral bands to learn the image features, and the input processing module in the hyperspectral automatic processing network of the present invention can automatically adjust the input structure, which can greatly expand the application of the deep image prior algorithm, making the deep image prior DIP method more suitable for hyperspectral super-resolution tasks, and can further improve the quality of the reconstructed image.

[0093] The reconstruction effect of the reconstruction method of the present invention is verified by experiments below.

[0094] In order to solve the difficulties of unclear reconstructed images and low indicators caused by insufficient training data in small data sets, the present invention proposes the reconstruction method of the present invention and improves the application of the method. Therefore, the main comparison method of the present invention is the deep image prior method, which mainly shows that it is a method that does not require training data and reference to ground truth. The algorithm performance of the present invention is leading in non-training-based methods, and can also be compared in learning-based methods. In most practical applications, many SR methods are almost impossible to apply under complex conditions. This work is mainly aimed at scenarios where training data is difficult to obtain and does not require ground-truth references, and is more practical in application. The effect of the present invention can be further illustrated by the following simulation experiments.

[0095] 1. Simulation conditions

[0096] The present invention is based on a central processing unit of Inter(R)Core(TM)i7-4790 3.60GHz CPU, NVIDIATitan XP GPU, and Ubuntu 16.04 operating system. The relevant experimental environment is as follows:

[0097] Python = 3.7

[0098] Pytorch = 1.2.0

[0099] numpy

[0100] skimage

[0101] imageio

[0102] matplotlib

[0103] torch

[0104] scipy

[0105] argparse

[0106] The database uses the Pavia centre dataset, which is widely used in the field of hyperspectral super-resolution. The methods compared in the experiment are as follows:

[0107] The first is a deep prior neural network, denoted as DIP in the experiment, which uses the structure itself to capture and process image statistics without learning details shared between external images. The second is a super-resolution reconstruction algorithm based on residual learning and deepening the network structure, denoted as VDSR in the experiment. VDSR proposes an operation of padding 0 after convolution to keep the image size consistent.

[0108] 2. Simulation content

[0109] Experiment 1, Reconstructed Image Quality Evaluation

[0110] According to the specific implementation method of the present invention and the existing natural image super-resolution reconstruction algorithm DIP based on deep image prior, the Pavia centre dataset and Chikusei dataset, which are widely used in the field of hyperspectral super-resolution, are used. The left part of the image in the dataset is extracted as the test data. There are 4 non-overlapping images to be super-resolved in the test data, with a pixel size of 223×223. The Pavia centre dataset is a public HSI dataset, which is the central area of ​​Pavia and was taken by a reflective optical system imaging spectrometer sensor in 2001. It has 102 spectral bands and 1096×1096 pixels. The Chikusei dataset is the urban area of ​​Chikusei, Japan, taken on July 29, 2014. It has 2517×2335 pixels, 128 spectral bands, and a spectral range of 363 to 1018nm. The experiment uses standard evaluation indicators: 5 commonly used quantitative image quality indicators (PQIs) are used to evaluate the performance of the method, including spectral angle mapping (SAM), relative global size error (ERGAS), root mean square error (RMSE), structural similarity (SSIM) and peak signal-to-noise ratio (PSNR). The present invention shows the average value of all bands of PSNR and SSIM, called MPSNR, MSSIM. The other three indicators are widely used in the field of HS fusion. The optimal values ​​of the above five indicators are 0, 0, 0, 1 and +∞, respectively. . The experimental results are shown in Table 1. Bicubic represents the result of image enlargement by bicubic interpolation, and scale represents different magnifications, which are 2x, 3x and 4x methods, respectively. The reconstruction method of the present invention is significantly superior to the current super-resolution reconstruction algorithm not based on training in all performance indicators, and it also has excellent results in many training-based methods, indicating that the image resolution method reconstructed by the method of the present invention is more suitable for high-spectral super-resolution tasks where it is difficult to obtain data sets and unify networks, further verifying the advancement of the present invention.

[0111] Table 1 Performance comparison of different algorithms on PAVIA CENTRE dataset

[0112] Pavia dataset Scale SAM↓ RMSE↓ ERGAS↓ MPSNR↑ SSIM↑ Bicubic 4 6.152 0.044 6.895 27.556 0.694 VDSR 4 6.700 0.0419 6.699 27.882 0.724 DIP 4 6.266 0.0410 6.484 28.106 0.736 Ours 4 5.983 0.040 6.253 28.659 0.761 Bicubic 8 7.746 0.062 4.830 24.595 0.472 VDSR 8 8.076 0.061 4.685 24.848 0.501 DIP 8 7.928 0.062 4.736 24.725 0.496 Ours 8 7.618 0.060 4.619 24.887 0.503

[0113] Table 2 Performance comparison of different algorithms on CHIKUSEI dataset

[0114] CHIKUSEI Scale SAM↓ RMSE↓ ERGAS↓ MPSNR↑ SSIM↑ Bicubic 4 3.524 0.016 6.756 37.528 0.889 VDSR 4 3.664 0.014 6.870 37.775 0.906 DIP 4 3.559 0.014 6.209 38.192 0.901 Ours 4 3.112 0.013 6.007 39.302 0.926 Bicubic 8 5.028 0.023 4.851 34.495 0.821 VDSR 8 5.177 0.021 4.905 34.566 0.830 DIP 8 5.338 0.021 4.678 34.669 0.812 Ours 8 5.024 0.020 4.586 34.841 0.839

[0115] As can be seen from Table 1 and Table 2, the image automatic processing network implemented based on the reconstruction method of the present invention has better performance indicators. At the same time, the reconstruction method of the present invention has the advantage that it does not need to be trained based on samples and ignores the differences in imaging conditions between different data sets. This work does not require true value images and focuses on scenes where training data is difficult to obtain, which has practical application value.

[0116] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A hyperspectral image super-resolution reconstruction method based on deep image prior, characterized in that: include: Step 1: Obtain the image to be super-resolved and the subject learning network; Step 2: Crop the edges of the image to be super-resolved to obtain a central area image; Step 3: uniformly divide the first position area of ​​the central area image to obtain non-overlapping test image samples and extract the second position area to obtain the sub-image block to be super-resolved; Wherein, the first location area and the second location area do not overlap; Step 4: Connecting the output of the constructed automatic processing network to the input of the main learning network to form a super-resolution reconstruction network; Wherein, the automatic processing network includes: an input processing module and a main body recovery module. The input processing module includes a first convolution layer, three second convolution layers and an upsampling layer with the same number of pixels as the input image, according to the order of input first and output later. The main body recovery module includes a second convolution layer, an upsampling layer, an attention module and four second convolution layers, according to the order of input first and output later. The automatic processing network includes a main road, a staggered jump channel and a combination channel. The main road is composed of the output of the input processing module connecting the input of the main body recovery module to the output of the main body recovery module. The combination channel is composed of the output of the first convolution layer of the input processing module directly connected to the input of the main body recovery module. The staggered jump channel is composed of the first convolution layer of the input processing module directly connected to the output of the last second convolution layer of the main body recovery module. Step 5: Input white noise of the same size as the sub-image block to be super-resolved into the super-resolution reconstruction network, and adjust the parameters of the super-resolution reconstruction network by back-propagation to reduce the loss function until the optimal number of iterations, so that the super-resolution reconstruction network learns the characteristics of the sub-image block to be super-resolved according to the correlation between the spatial information of the sub-image block to be super-resolved and the spectral band, and obtains the reconstructed hyperspectral image.

2. The method for super-resolution reconstruction of hyperspectral images based on depth image prior according to claim 1, characterized in that: The first position area is a left half area in the central area image, and the second position area is a right half area in the central area image.

3. The hyperspectral image super-resolution reconstruction method based on deep image prior according to claim 1 is characterized in that: The size of the central area image is 1096×715×102, the size of the test image sample and the sub-image block to be super-resolved is 223×223, the size of the first convolutional layer is 223×223, and the size of the second convolutional layer is 3×3.

4. The method for super-resolution reconstruction of hyperspectral images based on depth image prior according to claim 1, characterized in that: In the automatic processing network, each convolutional layer is connected to an activation layer, and the activation function of the activation layer is: Among them, x represents the parameters after the previous convolution layer, which represents the slope in the negative direction; The loss function of the automatic processing network is expressed as: E(x;x0)=||d(f θ (IPM(Z)))-x0||1 Among them, x0 represents the low-quality image to be super-resolved, x represents the output image, and d(f θ (IPM(Z))) represents the input white noise Z which is input into the super-resolution reconstruction network and then downsampled after being output by the main learning network.

5. The method for super-resolution reconstruction of hyperspectral images based on depth image prior according to claim 1, characterized in that: The first convolution layer in the input processing module is used to compress the input sample image; The compressed data transmitted by the combined channel and the data sampled by the upsampling layer are input into the main body recovery module after dimension combination. The compressed data transmitted by the combined channel is represented as: The sampling data of the upsampling layer is expressed as: The output of the automatic processing network is expressed as: Z * =F IPM (Z) = Z p +H cp (Z ca ) Among them, Z p Represents the compressed data after compression by the first convolutional layer in the input processing module.

6. The method for super-resolution reconstruction of hyperspectral images based on depth image prior according to claim 1, characterized in that: The optimal number of iterations is obtained through multiple experiments, and the experimental process includes: Setting the number of experimental iterations according to the parameter scale of the super-resolution reconstruction network; Obtain high-definition sub-images to form an experimental set; Using white noise of the same size as the experimental set to input the super-resolution reconstruction network to train the super-resolution reconstruction network until the number of experimental iterations is reached, recording the parameter values ​​of the super-resolution reconstruction network at each iteration and storing the reconstructed image output by the super-resolution reconstruction network every first number of iterations; Compare each stored image with the high-definition sub-image to determine the reconstructed image with the highest definition; According to the experimental set, determining the parameter values ​​and the number of experimental iterations corresponding to the reconstructed image with the highest clarity output by the super-resolution reconstruction network; The number of experimental iterations corresponding to the reconstructed image with the highest clarity is determined as the optimal number of iterations for training the super-resolution reconstruction network.

7. The method for super-resolution reconstruction of hyperspectral images based on depth image prior according to claim 6, characterized in that: The step 5 comprises: Step 51: inputting white noise having the same size as the sub-image block to be super-resolved into a super-resolution reconstruction network to obtain an output image; Step 52: Calculate the loss function of the downsampled output image and the supramolecular image block to be processed; Step 53: adjusting the parameters of the super-resolution reconstruction network by back-propagation to train the super-resolution reconstruction network until the optimal number of iterations is reached to obtain a reconstructed sub-block; Step 54: Integrate the reconstructed sub-blocks to obtain a reconstructed hyperspectral image.

Images