Multi-spectral remote sensing image depth spectrum super-division method considering land types

Abstract

The invention relates to a multi-spectral remote sensing image depth spectrum super-division method considering land types, which comprises the following steps: constructing a one-dimensional convolution-up-sampling fusion spectrum super-division network model, and adopting a one-dimensional up-sampling convolution network architecture which comprises a spectrum convolution layer, a linear spectrum space full-connection layer and a spectrum feature up-sampling layer; establishing a multispectral-hyperspectral non-monocular network set by land classes, and for multispectral-hyperspectral pixel pairs of different land classes, inputting a network model for training to obtain multispectral-hyperspectral non-monocular weights of different land classes; weighted adversarial pixel spectrum prediction and image reconstruction are carried out, spectrum prediction is carried out pixel by pixel by using a spectrum non-single-shot network set to obtain spectrum prediction results of different land classes, and according to a spectrum coefficient weighted average method and spectrum angle minimum adversarial comparison, an optimal pixel spectrum prediction result is selected, and prediction is carried out pixel by pixel to obtain a reconstructed hyperspectral image. According to the invention, a reconstruction scheme of the multispectral image is designed, and the spectral resolution of the multispectral remote sensing image is enhanced.

Description

Technical Field

[0001] The present invention relates to the research field of spectral super-resolution of multispectral remote sensing images, and in particular to a method for deep spectral super-resolution of multispectral remote sensing images taking land types into consideration. Background Art

[0002] Limited by the development of sensor hardware technology, existing remote sensing images often restrict each other in spectral, spatial, temporal and radiometric resolution, that is, it is difficult for a single remote sensing image to simultaneously obtain high spectral, high spatial, high temporal and high radiometric resolution. To this end, experts and scholars in the field of remote sensing fusion have proposed a method of fusion of multiple remote sensing images with different advantages, in the hope of obtaining a synthetic image with high spectral, high spatial or high temporal resolution. In recent years, with the continuous maturity of hyperspectral remote sensing image research and application technology, remote sensing image spectral resolution super-resolution has begun to receive widespread attention.

[0003] "Wide-width hyperspectral image reconstruction method based on multi / hyperspectral image fusion" (application number 201310256179.2) proposes a spectral resolution enhancement method for remote sensing multispectral images, that is, using auxiliary multi / hyperspectral data to generate wide-width hyperspectral images, establishing spectral non-injective relationships between images in their overlapping areas, and creating hyperspectral vectors from the matrix in the original multispectral image. The obtained image data has the same spectral resolution and spatial resolution as the original hyperspectral data, and the bands are also the same as the original multispectral data. However, the input pixels must be pure pixels, and the type of pure pixels must be consistent with the actual hyperspectral image to be super-resolved. However, existing hyperspectral images usually have low resolution and are mostly mixed pixels, making it difficult to obtain a sufficient number of pure pixels. Summary of the invention

[0004] At present, remote sensing digital image fusion mainly focuses on the fusion of temporal resolution and spatial resolution, and less attention is paid to the spectral enhancement of multispectral remote sensing images. In order to solve the existing technical problems, the present invention provides a multispectral remote sensing image deep spectral super-resolution method taking into account land types.

[0005] The specific content of the present invention is as follows: A method for deep spectral super-resolution of multi-spectral remote sensing images taking into account land types comprises the following steps: Step (1), constructing a one-dimensional convolution-upsampling fusion spectral super-resolution network model (1DCN-U Spectral SRNet): using a one-dimensional upsampling convolutional network, including a spectral convolution layer, a linear spectral spatial connection layer and a spectral feature upsampling layer. The network input is a multispectral pixel set, and the output is a corresponding hyperspectral pixel set. Step (2), establishing a multispectral-hyperspectral non-injective network set for different land types: for multispectral-hyperspectral pixel pairs of different land types, input the network model training to obtain multispectral-hyperspectral non-injective weights of different land types; Step (3), weighted adversarial pixel spectral prediction and image reconstruction: Spectral prediction is performed pixel by pixel using a spectral non-injective network set to obtain pixel spectral prediction results under multispectral-hyperspectral non-injective networks of different land types. The optimal pixel spectral prediction result is selected based on the spectral coefficient weighted average method and the spectral angle minimum adversarial comparison, and prediction is performed pixel by pixel to obtain a reconstructed hyperspectral image.

[0006] Furthermore, in step (1), the spectral convolution layer is a one-dimensional convolution layer, and 8 spectral band feature extractors with a convolution kernel size of 3 are used in the spectral dimension; the linear spectral space connection layer is a fully connected layer, and the spectral space is expanded in a linear connection manner based on the features extracted by the spectral convolution layer; the spectral feature upsampling layer is a bilinear interpolation resampling of the linear spectral space connection layer, and based on the feature spectrum set output by the spectral feature upsampling layer, a fully connected layer is used to complete the mapping of the target spectral space, and the spectral super-resolution result is output according to the rule that negative values ​​are reset to zero and positive values ​​remain unchanged.

[0007] Furthermore, step (2) includes: Step (21), preprocessing of multispectral-hyperspectral non-injective network input data: for multispectral-hyperspectral data, calculate the total number of pixels of each type according to the classification mask, divide it into a training sample set (train_data) and a test sample set (test_data) in a ratio of 7:3, traverse the image, and split the original multispectral and hyperspectral images into pixel pairs in proportion, wherein the training sample set will be used as the network input data set in step (1) after sample expansion, and the test sample set will be used as the network prediction result comparison verification data set after sample expansion; Step (22), based on the preprocessed data obtained in step (21), input the one-dimensional convolution-upsampling fusion spectral super-resolution network model of step (1), guide the various parameters of the loss function to continuously approach the global optimum through the Adam optimizer, save the non-injective weight file, and the set of non-injective weight files of all land types is the multispectral-hyperspectral non-injective network set of different land types.

[0008] Furthermore, multispectral and hyperspectral data are required to belong to the same region, be acquired in the same period, and have accurate and complete land classification. Furthermore, step (3) includes: Step (31), based on the non-injective weight files of each land class, for the multispectral images in the test sample set, the spectrum is super-classified pixel by pixel into a mapping matrix set of different land classes , multispectral image pixel bands and corresponding matrix sets The corresponding band is used to calculate the inverse distance to obtain the parameters , the spectral angle calculation results in the following parameters: ; Step (32), weighted pixel spectrum prediction: using the spectral coefficient weighted average method, including: calculating the proportion of non-injective weights of each type of area on each pixel as , is an integer, and then the spectral matrix value of each pixel is , the multi-spectral classification sub-images in the test sample set are processed pixel by pixel, and the output result is the weighted pixel spectrum prediction; Step (33), minimum spectral angle spectrum prediction: According to the spectral angle The angle size is used to select the optimal spectral matrix in the pixel land class mapping matrix set. ,After processing the multi-spectral classification sub-images in the test sample set pixel by pixel, the output result is the minimum spectral angle spectrum prediction; Step (34), image reconstruction: based on the weighted pixel spectral prediction and the minimum spectral angle spectral prediction results, the spectral angle (SAM), peak signal-to-noise ratio (PSNR) and relative global dimensionless (ERGAS) are compared with the hyperspectral images in the test sample set, the optimal image is selected and rolled back, and the resulting image is the multispectral image hyperspectral resolution image of the one-dimensional convolution-upsampling fusion spectral super-resolution network.

[0009] The present invention provides a method for establishing a set of multispectral-hyperspectral non-injective models according to different land types, expanding the diversity of spectral super-resolution spectral band selection of multispectral-hyperspectral images, improving the stability and accuracy of non-injective models, and realizing spectral resolution enhancement of multispectral remote sensing images. It reduces the threshold of hyperspectral resolution of multispectral images, further improves the accuracy of hyperspectral resolution, enriches the theory and method of remote sensing image fusion, and has important significance for achieving original results in spectral resolution enhancement of remote sensing. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The specific implementation of the present invention will be further explained below in conjunction with the accompanying drawings.

[0011] Figure 1 It is a schematic diagram of the process of the present invention.

[0012] Figure 2 This is the structural diagram of the one-dimensional convolution-upsampling fusion spectral super-resolution network model (1DCN-U Spectral SRNet) of the present invention.

[0013] Figure 3 It is a schematic diagram of the weighted average method of spectral coefficients of the present invention.

[0014] Figure 4This is a comparison diagram of the hyper-spectral resolution spectra of the Indian Pines dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] Combination Figure 1-Figure 4 The multispectral image super-spectral resolution method of one-dimensional convolution-upsampling fusion spectral super-resolution network proposed in this invention is committed to enhancing the spectral resolution of multispectral remote sensing images. Figure 1 As shown in the figure, it includes the construction of a one-dimensional convolution-upsampling fusion spectral super-resolution network model, the establishment of a multi-spectral-hyperspectral non-injective network set for different land types, and weighted adversarial pixel spectral prediction and image reconstruction. It provides a new technical route for the hyper-spectral resolution of multi-spectral remote sensing images, which can greatly improve the accuracy of hyper-spectral resolution and enrich the theory and method of remote sensing image fusion.

[0016] The specific content of the present invention is as follows: A method for deep spectral super-resolution of multi-spectral remote sensing images taking into account land types comprises the following steps: Step (1), constructing a one-dimensional convolution-upsampling fusion spectral super-resolution network model (1DCN-U Spectral SRNet): using a one-dimensional upsampling convolutional network, including a spectral convolution layer, a linear spectral spatial connection layer and a spectral feature upsampling layer. The network input is a multispectral pixel set, and the output is a corresponding hyperspectral pixel set. Step (2), establishing a multispectral-hyperspectral non-injective network set for different land types: for multispectral-hyperspectral pixel pairs of different land types, input the network model training to obtain multispectral-hyperspectral non-injective weights of different land types; Step (3), weighted adversarial pixel spectral prediction and image reconstruction: Spectral prediction is performed pixel by pixel using a spectral non-injective network set to obtain pixel spectral prediction results under multispectral-hyperspectral non-injective networks of different land types. The optimal pixel spectral prediction result is selected based on the spectral coefficient weighted average method and the spectral angle minimum adversarial comparison, and prediction is performed pixel by pixel to obtain a reconstructed hyperspectral image.

[0017] Preferably, in step (1) of the present embodiment, the spectral convolution layer is a one-dimensional convolution layer, and 8 spectral band feature extractors with a convolution kernel size of 3 are used in the spectral dimension; the linear spectral space connection layer is a fully connected layer, and the spectral space is expanded in a linear connection manner based on the features extracted by the spectral convolution layer; the spectral feature upsampling layer is a bilinear interpolation resampling of the linear spectral space connection layer, and based on the feature spectrum set output by the spectral feature upsampling layer, a fully connected layer is used to complete the mapping of the target spectral space, and the spectral super-resolution result is output according to the rule that negative values ​​are reset to zero and positive values ​​remain unchanged.

[0018] In this embodiment, step (2) preferably includes: Step (21), preprocessing of multispectral-hyperspectral non-injective network input data: for multispectral-hyperspectral data, calculate the total number of pixels of each type according to the classification mask, divide it into a training sample set (train_data) and a test sample set (test_data) in a ratio of 7:3, traverse the image, and split the original multispectral and hyperspectral images into pixel pairs in proportion, wherein the training sample set will be used as the network input data set in step (1) after sample expansion, and the test sample set will be used as the network prediction result comparison verification data set after sample expansion; Step (22), based on the preprocessed data obtained in step (21), input the one-dimensional convolution-upsampling fusion spectral super-resolution network model of step (1), guide the various parameters of the loss function to continuously approach the global optimum through the Adam optimizer, save the non-injective weight file, and the set of non-injective weight files of all land types is the multispectral-hyperspectral non-injective network set of different land types.

[0019] In this embodiment, the multispectral-hyperspectral data are preferably from the same region, acquired in the same period, and have accurate and complete land classification. In this embodiment, step (3) preferably includes: Step (31), based on the non-injective weight files of each land class, for the multispectral images in the test sample set, the spectrum is super-classified pixel by pixel into a mapping matrix set of different land classes , multispectral image pixel bands and corresponding matrix sets The corresponding band is used to calculate the inverse distance to obtain the parameters , the spectral angle calculation results in the following parameters: ; Step (32), weighted pixel spectrum prediction: using the spectral coefficient weighted average method, including: calculating the proportion of non-injective weights of each type of area on each pixel as , is an integer, and then the spectral matrix value of each pixel is , the multi-spectral classification sub-images in the test sample set are processed pixel by pixel, and the output result is the weighted pixel spectrum prediction; Step (33), minimum spectral angle spectrum prediction: According to the spectral angle The angle size is used to select the optimal spectral matrix in the pixel land class mapping matrix set. ,After processing the multi-spectral classification sub-images in the test sample set pixel by pixel, the output result is the minimum spectral angle spectrum prediction; Step (34), image reconstruction: based on the weighted pixel spectral prediction and the minimum spectral angle spectral prediction results, the spectral angle (SAM), peak signal-to-noise ratio (PSNR) and relative global dimensionless (ERGAS) are compared with the hyperspectral images in the test sample set, the optimal image is selected and rolled back, and the resulting image is the multispectral image hyperspectral resolution image of the one-dimensional convolution-upsampling fusion spectral super-resolution network.

[0020] To illustrate the specific implementation of this example, this example uses Windows 10 as the operating system environment, PyCharm Community Edition 2022.1 x64 as the development environment, Python as the development language, PyTorch as the basic framework, and the hyperspectral file is stored as a tif file. The Indian Pines dataset (spectral images collected by the AVIRIS sensor at the Indian Pine Test Site in northwestern Indiana in 1992, containing 145×145 pixels, 220 non-zero bands, and a spatial resolution of 20 meters) is used as the experimental example data. The specific implementation process is as follows: 1. One-dimensional convolution-upsampling fusion spectral super-resolution network model construction: Adopt a one-dimensional upsampling convolution network architecture, including a spectral convolution layer, a linear spectral space fully connected layer, and a spectral feature upsampling layer. The specific steps are: Network model construction: Introduce the nn module in Torch, call Conv1d as the spectral convolution layer, and use 8 spectral feature extractors with a convolution kernel size of 3 in the spectral dimension. Set bias to True. Call Liner as the linear spectral space connection layer to further expand the spectral space based on the feature extraction of the spectral convolution layer. Call Upsample in the spectral feature upsampling layer, and the default is bilinear interpolation. Based on the feature data of the spectral feature upsampling layer, use the fully connected layer to complete the mapping of the target spectral space. At this time, call Relu as the activation function before the fully connected layer.

[0021] 2. Establish a multispectral-hyperspectral non-injective network set for different land types: For multispectral and hyperspectral pixel pairs of different land types, input the network model training to obtain the multispectral-hyperspectral non-injective weights of different land types; It includes the following steps: Step 1: Dataset division and preprocessing According to the spectral response curve of TM data, the Indian Pines image is spectrally resampled to obtain a 6-band image as the experimental multispectral data. The total number of pixels of each type in the Indian Pines image is calculated according to the classification mask, and divided into train_data and test_data at a ratio of 7:3. The image is traversed, and the original multispectral and hyperspectral image is split into pixel pairs in proportion, and then the train_data is sampled and expanded.

[0022] Step 2: Organize the corresponding form of multispectral-hyperspectral pixels in the model The multispectral pixels after sample expansion are used as the input of the network model, and the hyperspectral pixels are used as the output of the network model. The imread function is called to read the same type of pixels into the memory in array form, and then converted into spectral reflectance values ​​and the FloatTensor function is called to convert its data format into a one-dimensional tensor form. The network batch_size is set to 6 multispectral bands. The train_data collection is constructed, which includes the input batch collection, i.e., 6 spectral bands of spectral pixels, and the corresponding label collection, i.e., 220 spectral bands of hyperspectral pixels.

[0023] Step 3 Construction of multispectral-hyperspectral non-injective network set After calling the manual_seed function to set the random seed, the network model is instantiated, Adam is used as the network optimizer, and the Learn read is set to 0.001. MSE is used as the loss function to calculate the error between the true value and the predicted value. Set the appropriate Epoch to loop the train_data into the network model, perform error back propagation after clearing the gradient each time, and then update the parameters. Save the network parameter file as "land type name.pth" under the appropriate error condition. Repeat the above process for each land type to obtain a multispectral-hyperspectral non-injective network set.

[0024] 3. Weighted adversarial pixel spectrum prediction and image reconstruction: Use the spectrum non-injective network set to predict the spectrum pixel by pixel, and obtain the pixel spectrum prediction results under the multi-spectral-hyperspectral non-injective network of different land types. According to the spectral coefficient weighted average method and the spectral angle minimum value adversarial comparison, select the optimal pixel spectrum prediction result, and predict pixel by pixel to obtain the reconstructed hyperspectral image. It includes the following steps: Step 1: Hyperspectral resolution reconstruction of multispectral images Based on the ".pth" weight file of each region, call the load_state_dict function to convert the pixel hyperspectral resolution of the multispectral classification sub-image in the test_data dataset into a mapping matrix set for each region. That is, each pixel in the classification sub-map corresponds to all land class ".pth" weight files for hyperspectral resolution calculation, and the collection of all matrices formed is .

[0025] Step 2 Image fusion using spectral coefficient weighted average method Based on the land class mapping matrix set , each band of the multispectral image is associated with the corresponding matrix set The corresponding band is used to calculate the inverse distance to obtain the parameters . Calculate the proportion of non-injective weights of each category in each pixel as follows: Then let the spectral matrix value of each pixel be The above processing is performed pixel by pixel on the multispectral classification sub-image in the test_data dataset, and the output image is the weighted pixel spectral image fusion image.

[0026] Step 3 Image reconstruction using the spectral angle minimization method Based on the land class mapping matrix set , each band of the multispectral image is associated with the corresponding matrix set The corresponding band is calculated by spectral angle (SAM) to obtain the parameter: According to the spectral angle Angular size, select pixel land class mapping matrix set After processing the multi-spectral classification sub-images in the test_data dataset pixel by pixel, the output result is the minimum spectral angle spectrum prediction.

[0027] Step 4: Comparison between the weighted average method of spectral coefficients and the minimum spectral angle method Based on the weighted pixel spectral prediction and the minimum spectral angle spectral prediction results, the spectral angle (SAM), peak signal-to-noise ratio (PSNR) and relative global dimensionless (ERGAS) are compared with the hyperspectral images in the test_data dataset. The optimal image is selected and rolled back. The resulting image is the multispectral image hyper-spectral resolution image of the one-dimensional convolution-upsampling fusion spectral super-resolution network.

[0028] The multispectral image hyperspectral resolution method based on one-dimensional convolution-upsampling fusion spectrum super-resolution network of the present invention can produce good image data. The spectrum enhanced image and the real hyperspectral image are compared by visual interpretation, SAM, PSNR, and ERGAS, and then the performance of the method is evaluated. After practical verification, the hyperspectral image provided by the present invention and the original image have only a slight color difference in visual interpretation, SAM is only 0.77, PSNR is 45.9, and ERGAS is 0.12.

[0029] Many specific details are described in the above description to facilitate a full understanding of the present invention. However, the above description is only a preferred embodiment of the present invention. The present invention can be implemented in many other ways different from those described herein, so the present invention is not limited to the specific implementation disclosed above. At the same time, any person familiar with the art can make many possible changes and modifications to the technical solution of the present invention using the methods and technical contents disclosed above without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A method for deep spectral super-resolution of multispectral remote sensing images taking into account land types, characterized by: The steps include: Step (1), constructing a one-dimensional convolution-upsampling fusion spectral super-resolution network model: using a one-dimensional upsampling convolutional network, including a spectral convolution layer, a linear spectral spatial connection layer and a spectral feature upsampling layer, the network input is a multispectral pixel set, and the output is a corresponding hyperspectral pixel set; Step (2), establishing a multispectral-hyperspectral non-injective network set for different land types: for multispectral-hyperspectral pixel pairs of different land types, input the network model training to obtain multispectral-hyperspectral non-injective weights of different land types; Step (3), weighted adversarial pixel spectral prediction and image reconstruction: Spectral prediction is performed pixel by pixel using a spectral non-injective network set to obtain pixel spectral prediction results under multispectral-hyperspectral non-injective networks of different land types. The optimal pixel spectral prediction result is selected based on the spectral coefficient weighted average method and the spectral angle minimum adversarial comparison, and prediction is performed pixel by pixel to obtain a reconstructed hyperspectral image.

2. The method for deep spectrum super-resolution of multispectral remote sensing images taking land types into consideration according to claim 1, characterized in that: In step (1), the spectral convolution layer is a one-dimensional convolution layer, and 8 spectral band feature extractors with a convolution kernel size of 3 are used in the spectral dimension; The linear spectral space connection layer is a fully connected layer, which expands the spectral space in a linear connection manner based on the features extracted by the spectral convolution layer; the spectral feature upsampling layer is a bilinear interpolation resampling of the linear spectral space connection layer, and based on the feature spectrum set output by the spectral feature upsampling layer, a fully connected layer is used to complete the mapping of the target spectral space, and the spectral super-resolution result is output according to the rule of negative values ​​being reset to zero and positive values ​​being unchanged.

3. The method for deep spectrum super-resolution of multispectral remote sensing images taking land types into consideration according to claim 1, characterized in that: Step (2) includes: Step (21), preprocessing of multispectral-hyperspectral non-injective network input data: for multispectral-hyperspectral data, calculate the total number of pixels of each type according to the classification mask, divide it into a training sample set and a test sample set according to a ratio of 7:3, traverse the image, and split the original multispectral and hyperspectral images into pixel pairs in proportion, wherein the training sample set will be used as the network input data set in step (1) after sample expansion, and the test sample set will be used as the network prediction result comparison verification data set after sample expansion; Step (22), based on the preprocessed data obtained in step (21), input the one-dimensional convolution-upsampling fusion spectral super-resolution network model of step (1), guide the various parameters of the loss function to continuously approach the global optimum through the Adam optimizer, save the non-injective weight file, and the set of non-injective weight files of all land types is the multispectral-hyperspectral non-injective network set of different land types.

4. The method for deep spectrum super-resolution of multispectral remote sensing images taking land types into consideration according to claim 3, characterized in that: Multispectral and hyperspectral data are required to belong to the same area, be acquired in the same time period, and have accurate and complete land classification.

5. The method for deep spectrum super-resolution of multispectral remote sensing images taking land types into consideration according to claim 1, characterized in that: Step (3) includes: Step (31), based on the non-injective weight files of each land class, for the multispectral images in the test sample set, the spectrum is super-classified pixel by pixel into a mapping matrix set of different land classes , multispectral image pixel bands and corresponding matrix sets The corresponding band is used to calculate the inverse distance to obtain the parameters , the spectral angle calculation results in the following parameters: ; Step (32), weighted pixel spectrum prediction: using the spectral coefficient weighted average method, including: calculating the proportion of non-injective weights of each type of area on each pixel as , is an integer, and then the spectral matrix value of each pixel is , the multi-spectral classification sub-images in the test sample set are processed pixel by pixel, and the output result is the weighted pixel spectrum prediction; Step (33), minimum spectral angle spectrum prediction: According to the spectral angle The angle size is used to select the optimal spectral matrix in the pixel land class mapping matrix set. ,After processing the multi-spectral classification sub-images in the test sample set pixel by pixel, the output result is the minimum spectral angle spectrum prediction; Step (34), image reconstruction: based on the weighted pixel spectral prediction and the minimum spectral angle spectral prediction results, the spectral angle, peak signal-to-noise ratio and relative global dimensionless comparison are made with the hyperspectral images in the test sample set, the optimal image is selected and rolled back, and the resulting image is the multispectral image hyperspectral resolution image of the one-dimensional convolution-upsampling fusion spectral super-resolution network.

Images