A hyperspectral remote sensing image generation method based on semi-supervised transfer spectral reconstruction network
By using a semi-supervised migration spectral reconstruction network, and leveraging the overlapping regions of multispectral satellites and local hyperspectral images, a wide-swath, high-spatial-resolution hyperspectral remote sensing image was generated. This solved the problem of high acquisition costs and enabled high-precision observation and interpretation over a wider area.
Patent Information
- Application Number
- CN202311374280.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-23
- Publication Date
- 2025-11-04
- Estimated Expiration
- 2043-10-23
AI Technical Summary
Acquiring wide-swath, high-spatial-resolution hyperspectral remote sensing images is difficult, and existing technologies are costly and inefficient.
A semi-supervised transfer spectral reconstruction network is adopted, which utilizes the overlapping regions of multispectral satellites and local hyperspectral images. The network is trained by combining semi-supervised and unsupervised learning to generate hyperspectral remote sensing images.
It generated wide-swath, high-spatial-resolution hyperspectral remote sensing images, expanding the observation range and improving the fine interpretation capability while reducing acquisition costs.
Smart Images
Figure CN117315401B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of satellite remote sensing, and relates to a deep neural network and spectral super-resolution technology, in particular to a hyperspectral image generation method. BACKGROUND
[0002] Hyperspectral images generally have multiple spectral bands from infrared spectrum to ultraviolet spectrum, and the rich spectral information of the hyperspectral images makes it easier to separate objects similar in local bands, and the rich spectral features have been widely applied to various tasks.
[0003] However, due to the limitations of imaging technology, it is relatively costly to obtain hyperspectral images containing rich spectral bands. Compared with hyperspectral images, multispectral images generally have fewer spectral bands (usually less than 20), accompanied by low acquisition cost, rich spatial information and continuous temporal information, making it more convenient to distinguish in image details. At the same time, the large number of multispectral satellites far more than hyperspectral satellites also brings a large number of available multispectral images. SUMMARY
[0004] The application aims to solve the problem of difficulty in obtaining large-width high spatial resolution hyperspectral remote sensing images, and proposes a hyperspectral remote sensing image generation method based on a semi-supervised transfer spectral reconstruction network.
[0005] The specific process of the hyperspectral remote sensing image generation method based on the semi-supervised transfer spectral reconstruction network is as follows:
[0006] Step 1, constructing and training a pre-trained spectral reconstruction network to obtain a trained pre-trained spectral reconstruction network;
[0007] Step 2, selecting real multispectral images simultaneously imaged with the hyperspectral image and having an overlapping area, cropping the multispectral images into image blocks as a training set, the overlapping area in the training set as a labeled sample, and the non-overlapping area as an unlabeled sample, training a region discrimination network in a semi-supervised learning manner combined with the labeled sample and the unlabeled sample to obtain a trained region discrimination network;
[0008] Step 3, selecting the multispectral images in the overlapping area in Step 2 and the hyperspectral image, cropping the images into image blocks to construct a hyperspectral-multispectral image supervised training set; selecting the multispectral images in the non-overlapping area, cropping the images into image blocks to construct a multispectral unsupervised training set;
[0009] Step 4, constructing a semi-supervised transfer spectral reconstruction network;
[0010] The semi-supervised transfer spectral reconstruction network comprises a pre-trained spectral reconstruction network, a region discrimination network and a transfer network.
[0011] The semi-supervised transfer spectral reconstruction network is trained in a semi-supervised learning and unsupervised learning manner by alternately using the high-multiple spectral image supervised training set and the multiple spectral unsupervised training set described in step 3, and a trained semi-supervised transfer spectral reconstruction network is obtained.
[0012] Step 5: The trained semi-supervised transfer spectral reconstruction network is used for spectral reconstruction of a to-be-tested multiple spectral image, and a hyperspectral remote sensing image is generated.
[0013] The present application has the following advantages:
[0014] The present application proposes a semi-supervised transfer spectral reconstruction network (SSTLSRN) by generating a hyperspectral remote sensing image with a large width and high spatial resolution from an overlapping large-width multiple spectral image and a local hyperspectral image.
[0015] The present application generates a hyperspectral remote sensing image with consistent data distribution as a local hyperspectral image by using a trained semi-supervised transfer spectral reconstruction network based on a large-width multiple spectral image resource, which provides a usable high spatial resolution and large-scene hyperspectral remote sensing image resource for hyperspectral data application.
[0016] The present application benefits from the short return period and rich quantity of multiple spectral satellites, and when a hyperspectral satellite acquires ground information, a multiple spectral satellite acquired at the same time can be found, and the acquired multiple spectral image has a width much larger than the hyperspectral image. In this case, it is quite meaningful to obtain the mapping relationship between the hyperspectral image and the multiple spectral image through the overlapping area, and to convert the multiple spectral image into a hyperspectral image as a kind of calculation alternative. The generated hyperspectral image will have higher spatial resolution and larger width, which makes it possible to conduct overall and high-precision fine interpretation of the region.
[0017] The present application can generate a large-width multiple spectral image into its corresponding hyperspectral image by combining historical information and overlapping area information of the hyperspectral image generation technology, that is, a high spatial resolution hyperspectral image for observing a wider area is generated, which enhances the observation range of the hyperspectral satellite; and the region is overall and widely analyzed in detail by using a hyperspectral classification model. BRIEF DESCRIPTION OF DRAWINGS
[0018] Figure 1 The flow chart of the hyperspectral remote sensing image generation system based on the semi-supervised transfer spectral reconstruction network is shown in the figure.
[0019] Figure 2 The overall structure diagram of the semi-supervised transfer spectral reconstruction network is shown in the figure.
[0020] Figure 3 Structure diagram of region discrimination network;
[0021] Figure 4 Structure diagram of migration network;
[0022] Figure 5 Structure diagram of 3D spectral self-attention block;
[0023] Figure 6 Structure diagram of spectral self-attention block;
[0024] Figure 7 Schematic diagram of scaling loss function.
[0025] Figure 8 Schematic diagram of migration loss function DETAILED DESCRIPTION
[0026] Embodiment one: combined with Figure 1 In this embodiment, the specific process of the hyperspectral remote sensing image generation method based on the semi-supervised migration spectral reconstruction network is as follows:
[0027] Step 1, constructing and training a pre-trained spectral reconstruction network to obtain a trained pre-trained spectral reconstruction network;
[0028] Step 2, selecting a real large-format wide multi-spectral image that is simultaneously imaged with the hyperspectral image and has an overlapping region, and cropping it into image blocks as a training set, the overlapping region in the training set as a labeled sample, and the non-overlapping region as an unlabeled sample, and training the region discrimination network in a semi-supervised learning manner combined with the labeled sample and the unlabeled sample to obtain a trained region discrimination network;
[0029] Step 3, selecting the multi-spectral image and the hyperspectral image in the overlapping region in step 2, and cropping them into image blocks to construct a hyperspectral-multi-spectral image supervised training set; selecting the multi-spectral image in the non-overlapping region, and cropping it into image blocks to construct a multi-spectral unsupervised training set;
[0030] Step 4, constructing a semi-supervised migration spectral reconstruction network;
[0031] The semi-supervised migration spectral reconstruction network includes a pre-trained spectral reconstruction network, a region discrimination network and a migration network, and the specific network structure is as shown in Figure 3
[0032] The semi-supervised migration spectral reconstruction network is trained in a supervised learning and unsupervised learning manner by alternately using the hyperspectral-multi-spectral image supervised training set and the multi-spectral unsupervised training set in step 3, and a trained semi-supervised migration spectral reconstruction network is obtained;
[0033] Step 5, the trained semi-supervised transfer spectral reconstruction network is used for spectral reconstruction of the to-be-tested multi-spectral image (including the overlapping region part and the non-overlapping region part) to generate a hyperspectral remote sensing image.
[0034] Specific implementation method two: the difference between this implementation method and the specific implementation method one is that the detailed steps of the training process of the pre-trained spectral reconstruction network in step 1 include:
[0035] The spectral response function of the multi-spectral satellite sensor is used to perform spectral resampling on the existing hyperspectral data set to obtain multi-spectral data corresponding to the hyperspectral data; the real hyperspectral data and the resampled multi-spectral data are used as training data sets, and the network parameters of the pre-trained spectral reconstruction network are optimized by the training data sets to obtain the trained pre-trained spectral reconstruction network.
[0036] The other steps and parameters are the same as those in the specific implementation method one.
[0037] Specific implementation method three: the difference between this implementation method and the specific implementation method one or two is that the spectral response function of the multi-spectral satellite sensor is used to perform spectral resampling on the existing hyperspectral data set to obtain multi-spectral data corresponding to the hyperspectral data; the specific process is as follows:
[0038] 1), the spectral response function of the multi-spectral satellite sensor used and the waveband of the hyperspectral image are queried; and and are used as continuous spectral data and multi-spectral data respectively, and R is used as the spectral response function, then The relationship between the i-th waveband of and R can be expressed as:
[0039] where λ is any wavelength within the range of the spectral response function R, and R i represents the spectral response function for the i-th waveband;
[0040] 2), the spectral response function of the multi-spectral satellite sensor is adjusted to the normalized spectral response function R H from hyperspectral to multi-spectral based on the principle of linear interpolation, and the normalized spectral response function R H represents the mapping relationship from the hyperspectral image to the multi-spectral image, then The relationship between the i-th waveband of and the hyperspectral curve H and R can be expressed as:
[0041]
[0042] H represents hyperspectral data, h represents the number of hyperspectral bands, R Hi represents the normalized spectral response function for the i-th band;
[0043] 3) multiply the hyperspectral image and the hyperspectral-to-multiplespectral normalized spectral response function R H to obtain the corresponding multiple spectral image L' M = L' H R' H , L' M , L' H and R' H are respectively represented in matrix form and R H .
[0044] The other steps and parameters are the same as those in embodiment one or two.
[0045] Embodiment four: the difference between this embodiment and one of embodiments one to three is that the pre-training spectral reconstruction network in step 1 can use any deep learning-based spectral reconstruction network, such as AWAN, PSJN, MST++, etc.
[0046] The other steps and parameters are the same as those in one of embodiments one to three.
[0047] Embodiment five: the difference between this embodiment and one of embodiments one to four is that the region discrimination network in step 2 sequentially includes a convolution layer, an activation layer, a convolution layer, an activation layer, a convolution layer, an activation layer, a convolution layer, an activation layer, a flattening layer, a fully connected layer, a fully connected layer, a fully connected layer, and a Softmax layer, as shown in Figure 2 .
[0048] The basic principle of the deep learning-based region discrimination network is to construct a region discrimination network based on binary classification.
[0049] The region discrimination network can determine whether other multiple spectral image blocks and overlapping region multiple spectral image blocks are similar in data distribution to a degree of 0.8 through training.
[0050] The other steps and parameters are the same as those in one of embodiments one to four.
[0051] Embodiment six: the difference between this embodiment and one of embodiments one to five is that the detailed steps of the training process for training the region discrimination network in step 2 include:
[0052] (1) Select a real large wide multi-spectral image that is imaged at the same time as the hyperspectral image and has an overlapping area, and crop it into image blocks as a training set. The overlapping area in the training set is assigned a label of 1 as a labeled sample, and the non-overlapping area is assigned an initial label of 0 as an unlabeled sample;
[0053] (2) Optimize the network parameters of the region discrimination network through the training set;
[0054] (3) After each iteration of the training set, all unlabeled samples are discriminated by the region discrimination network. The specific process is as follows:
[0055] A number in the range of 0 to 1 is obtained through the Softmax layer of the region discrimination network;
[0056] Values greater than or equal to 0.5 obtained through the Softmax layer of the region discrimination network are determined as label 1, and values less than 0.5 are determined as label 0. Only the original label is determined, but not changed;
[0057] Samples in the unlabeled sample that are determined to be label 1 and have a value greater than or equal to 0.8 are changed from label 0 to label 1. The original label is changed;
[0058] Samples in the unlabeled sample that are determined to be label 0 and have a value less than 0.2 are changed from label 1 to label 0. The original label is changed;
[0059] (4) Stop after 100 iterations or when the number of samples changed to label 1 exceeds 50%, and obtain the trained region discrimination network.
[0060] The other steps and parameters are the same as one of the first to fifth embodiments.
[0061] Specific embodiment seven: this embodiment is different from one of the first to sixth embodiments in that the semi-supervised transfer spectral reconstruction network is constructed in step 4;
[0062] The semi-supervised transfer spectral reconstruction network includes a pre-trained spectral reconstruction network, a region discrimination network, and a transfer network. The specific network structure is as shown in Figure 3 ;
[0063] The specific process is as follows:
[0064] The transfer network is composed of five groups of 3D spectral self-attention blocks connected in series. The network structure of the transfer network in the semi-supervised transfer spectral reconstruction network is as shown in Figure 4 . The transfer network realizes the transfer of spectral information through spectral feature extraction;
[0065] The input of the region discrimination network is consistent with that of the pre-trained spectral reconstruction network, and the output is 0 or 1;
[0066] The output of the pre-trained spectral reconstruction network is taken as the input of the transfer network;
[0067] The output of the transfer network is multiplied by the output of the region discrimination network and added to the output of the pre-trained spectral reconstruction network, which is collectively taken as the output of the semi-supervised transfer spectral reconstruction network;
[0068] The total system response of the semi-supervised transfer spectral reconstruction network is represented as
[0069] HSI SR = H Basic (MSI; θ1) + H Dis (MSI; θ2) x H Transfer (H Basic (MSI; θ3)
[0070] where H Basic represents the system response of the pre-trained spectral reconstruction network, H Dis represents the system response of the region discrimination network, H Transfer represents the system response of the transfer network, θ1 represents the network parameters of the pre-trained spectral reconstruction network, θ2 represents the network parameters of the region discrimination network, and θ3 represents the network parameters of the transfer network. MSI is the input multispectral data, and HSI SR is the reconstructed hyperspectral data output by the semi-supervised transfer spectral reconstruction network;
[0071] When the value obtained by the Softmax layer of the region discrimination network is greater than or equal to 0.5, it indicates that the similarity between the multispectral image and the multispectral image of the overlapping region in the data distribution is greater than or equal to 0.5, at this time, the output H Dis (MSI; θ2) of the region discrimination network is 0, that is, the final output is the output of the base model, and at this time, the system response of the semi-supervised transfer spectral reconstruction network is
[0072] HSI SR = H Basic (MSI; θ1)
[0073] When the value obtained by the Softmax layer of the region discrimination network is less than 0.5, it indicates that the similarity between the multispectral image and the multispectral image of the overlapping region in the data distribution is less than 0.5, at this time, the output H Dis (MSI; θ2) of the region discrimination network is 1, that is, the result of adding the output of the base model and the output of the transfer model is taken as the final output, and at this time, the system response of the semi-supervised transfer spectral reconstruction network is
[0074] HSI SR = H Basic (MSI; θ1) + H Transfer (H Basic(MSI; θ3))
[0075] The pre-trained spectral reconstruction network has the same structure as the pre-trained spectral reconstruction network in step 1.
[0076] Region discrimination networks can determine whether input multispectral image patches and overlapping region multispectral image patches are similar in spectral distribution;
[0077] The migration network system response is represented as
[0078] I Transfer =H Transfer (I Pre )=H 3-5 (H 3-4 (H 3-3 (H 3-2 (H 3-1 (I Pre )))))
[0079] Where H 3-1 H 3-2 H 3-3 H 3-4 and H 3-5 The system responses of the five 3D spectral self-attention blocks are respectively, I Pre To input the multispectral image into the pre-reconstructed hyperspectral image obtained from the pre-trained spectral reconstruction network trained in step 1, I Transfer The migration residuals output by the migration network.
[0080] The other steps and parameters are the same as those in one of the specific implementation methods one to six.
[0081] Specific Implementation Method Eight: This implementation method differs from one of the specific implementation methods one to seven in that the 3D spectral self-attention block includes a spectral self-attention block, a first 3D residual block, a second 3D residual block, a third 3D residual block, a fourth 3D residual block, a 3D stitching layer, and a 3D post-processing layer.
[0082] like Figure 5 As shown, the 3D residual block is a residual structure that uses 3D convolution instead of 2D convolution;
[0083] The output features of the spectral self-attention block are input into the first 3D residual block, the output features of the first 3D residual block are input into the second 3D residual block, the output features of the second 3D residual block are input into the third 3D residual block, and the output features of the third 3D residual block are input into the fourth 3D residual block.
[0084] The output features of the spectral self-attention block, the first 3D residual block, the second 3D residual block, the third 3D residual block, and the fourth 3D residual block are respectively input into the 3D stitching layer for feature stacking;
[0085] The output feature of the 3D splicing layer is input into a 3D post-processing layer, and the 3D post-processing layer outputs the feature as a 3D spectral self-attention block output feature;
[0086] The 3D post-processing layer sequentially comprises a 3D convolution layer, an activation layer and a channel attention block;
[0087] A 3D residual block is used to process the extracted spectral feature information, and a total of four 3D residual blocks are used to process features of different depths;
[0088] A 3D splicing layer is used to stack the different depth features of the extracted multi-spectral image extracted by the spectral self-attention block and the 3D residual block;
[0089] A 3D post-processing layer comprising a 3D convolution layer, a 3D activation layer and a channel attention block is used to post-process the stacked features to obtain hyperspectral features.
[0090] The specific structure of the spectral self-attention block is as follows:
[0091] The input image features pass through a full connection layer to obtain feature A and feature B respectively;
[0092] Feature A and feature B are multiplied to obtain feature C;
[0093] Feature C is subjected to a softmax operation to obtain feature D;
[0094] Feature D is multiplied with the input image feature to obtain feature E;
[0095] Feature E is subjected to a full connection layer to obtain feature F;
[0096] Feature F is the output feature of the spectral self-attention block.
[0097] The structure of the spectral self-attention block is as shown in Figure 6 The weight is calculated by the autocorrelation of the input data, and finally the calculated weight is multiplied with the input, wherein the three variable weights weight 1, weight 2 and weight 3 are realized by three full connection layers.
[0098] A spectral self-attention block is used to extract the spectral feature information of the input multi-spectral image.
[0099] The other steps and parameters are the same as one of the first to eighth embodiments.
[0100] Specific implementation nine: different from one of the specific implementations one to eight, the step 4 is that the semi-supervised transfer spectral reconstruction network is trained in a supervised learning manner by the high-multiple spectral image supervised training set in step 3, and the network is constrained by a root mean square error loss function; specifically:
[0101] The semi-supervised transfer spectral reconstruction network parameters are optimized by minimizing the root mean square error loss function shown in the following formula:
[0102] Loss = L (H Basic (MSI; θ1) + H Dis (MSI; θ2) x H Transfer (H Basic (MSI; θ3), HSI)
[0103] Where Loss is the loss function, L is the root mean square error loss function, H W is the system response of the complete network, MSI is the input multi-spectral data, HSI is the original hyperspectral data overlapping with the input multi-spectral data MSI, and θ is the semi-supervised transfer spectral reconstruction network parameter;
[0104] The step 4 is that the semi-supervised transfer spectral reconstruction network is trained in an unsupervised learning manner by the multiple spectral unsupervised training set in step 3, and the network is constrained by a scaling loss function and a transfer loss function; specifically:
[0105] The semi-supervised transfer spectral reconstruction network parameters are optimized by minimizing the sum of the scaling loss function Loss1 and the transfer loss function Loss2; the expression is:
[0106] Loss = Loss1 + αLoss2
[0107] Where Loss is the loss function, and α is the weight;
[0108] The use of the scaling loss function is as shown in the following formula: Figure 7 The spectral amplitude of the input multi-spectral image is multiplied by 0.5, 1 and 2 respectively, the spectral amplitude of the hyperspectral image output by the semi-supervised transfer spectral reconstruction network is multiplied by 2, 1 and 0.5 respectively, and whether the images change after the input and output are scaled by the same ratio is compared (the root mean square error before the output without scaling is calculated, and the two are added), the first half of the following formula is that the input MSI x 2 output x 0.5 is compared with the input and output both x 1; the second half is that the input MSI x 0.5 output x 2 is compared with the input and output both x 1;
[0109] The scaling loss function value Loss1 calculation process is:
[0110] The specific expression of the scaling loss function is as follows:
[0111] Loss1=L(0.5×H W (2×MSI;θ),H W (MSI;θ))+L(2×H W (0.5×MSI;θ),H W (MSI;θ))wherein H W (2×MSI;θ)=
[0112] (H Basic (2×MSI;θ1)+H Dis (2×MSI;θ2)×H Transfer (H Basic (2×MSI;θ3))
[0113] wherein H W (MSI;θ)=
[0114] (H Basic (1×MSI;θ1)+H Dis (1×MSI;θ2)×H Transfer (H Basic (1×MSI;θ3))
[0115] wherein H W (0.5×MSI;θ)=
[0116] (H Basic (0.5×MSI;θ1)+H Dis (0.5×MSI;θ2)×H Transfer (H Basic (0.5×MSI;θ3))
[0117] wherein L is a root mean square error loss function, H W is a total system response of the semi-supervised transfer spectral reconstruction network, MSI is input multi-spectral data, and θ is a semi-supervised transfer spectral reconstruction network parameter;
[0118] The transfer loss function value Loss2 calculation process is as follows:
[0119] The use of the transfer loss function is as shown in Figure 8 ;
[0120] The multi-spectral image is input into the pre-trained spectral reconstruction network trained in step 1 to obtain a pre-reconstructed hyperspectral image;
[0121] The multi-spectral image is input into the semi-supervised transfer spectral reconstruction network in step 4 to obtain a reconstructed hyperspectral image;
[0122] The migration loss function value between the pre-reconstruction hyperspectral image and the reconstructed hyperspectral image is calculated to optimize the semi-supervised migration spectral reconstruction network parameters.
[0123] The other steps and parameters are the same as one of embodiments one to eight.
[0124] Embodiment ten: different from one of embodiments one to nine, the migration loss function value between the pre-reconstruction hyperspectral image and the reconstructed hyperspectral image is calculated; the specific process is as follows:
[0125] The migration loss function adopts the gradient direction consistent loss function, and the specific calculation process of the gradient direction consistent loss function is as follows:
[0126] I. Generate the gradient direction sequence corresponding to the pre-reconstruction hyperspectral image; the specific process is as follows:
[0127] 1. Calculate the spectral difference value of every four wavebands in the spectral curve of each pixel in the pre-reconstruction hyperspectral image as the gradient of the current waveband;
[0128] 2. Calculate the average value of the gradient of the current waveband and the adjacent four wavebands in 1 as the average gradient;
[0129] 3. Sample the average gradient in 2 every three wavebands to obtain the sampled average gradient;
[0130] 4. Normalize the sampled average gradient in 3 to -1 to +1 to obtain the normalized gradient;
[0131] 5. Assign +1 to the normalized gradient greater than 0.3, assign -1 to the normalized gradient less than -0.3, and assign 0 to the intermediate value, thereby generating the gradient direction sequence corresponding to the pre-reconstruction hyperspectral image;
[0132] II. Generate the gradient direction sequence corresponding to the reconstructed hyperspectral image; the specific process is as follows:
[0133] 1). Calculate the spectral difference value of every four wavebands in the spectral curve of each pixel in the reconstructed hyperspectral image as the gradient of the current waveband;
[0134] 2). Calculate the average value of the gradient of the current waveband and the adjacent four wavebands in 1) as the average gradient;
[0135] 3). Sample the average gradient in 2) every three wavebands to obtain the sampled average gradient;
[0136] 4). Normalize the sampled average gradient in 3) to -1 to +1 to obtain the normalized gradient;
[0137] 5) Assigning +1 to the normalized gradient in 4) greater than 0.3, -1 to the normalized gradient less than -0.3, and 0 to the intermediate, thereby generating a gradient direction sequence corresponding to the reconstructed hyperspectral image;
[0138] III. Calculate the RMSE value between the generated gradient direction sequence corresponding to the pre-reconstructed hyperspectral image and the generated gradient direction sequence corresponding to the reconstructed hyperspectral image, and the RMSE value as the migration loss function value between the pre-reconstructed hyperspectral image and the reconstructed hyperspectral image;
[0139] RMSE is the root mean square error between the gradient direction sequence corresponding to the pre-reconstructed hyperspectral image and the gradient direction sequence corresponding to the reconstructed hyperspectral image.
[0140] The other steps and parameters are the same as one of the first to ninth embodiments.
[0141] In addition, the application also provides a hyperspectral remote sensing image generation system based on a semi-supervised transfer spectral reconstruction network, which comprises simultaneously acquiring a multi-spectral image and a local hyperspectral image in an overlapping area from a multi-spectral satellite and a hyperspectral satellite, and generating a complete hyperspectral remote sensing image through the hyperspectral remote sensing image generation method based on the semi-supervised transfer spectral reconstruction network.
[0142] The beneficial effects of the application are verified by the following embodiments:
[0143] 16 hyperspectral images obtained by the Orbiter hyperspectral satellite OHS2A, OHS2C and OHS2D are used to construct a hyperspectral spectral reconstruction pre-training set, three regions in the overlapping area of one hyperspectral image and one Gaofen-1 multi-spectral image are selected to construct a high-multiple spectral image supervised training set, and the remaining area of the Gaofen-1 multi-spectral image is used to construct a multi-spectral unsupervised training set.
[0144] The Orbiter hyperspectral satellite can download 32 bands at 400-1000nm; the Gaofen-1 satellite has 4 bands at 400-1000nm. In the training process, the hyperspectral image, the multi-spectral image obtained by downsampling, and the real multi-spectral image are input into the network as image blocks with a spatial size of 16x16, the ADAM optimizer is used for training, the initial learning rate is 0.0005, the learning rate is reduced by 10% for each epoch iteration, and the minimum is reduced to 0.00002. The root mean square error RMSE, the average relative error MRAE and the spectral angle function SAM are used to measure the similarity between the generated hyperspectral image and the real hyperspectral image, and the smaller the RMSE, MRAE and SAM, the higher the reconstruction accuracy.
[0145] Table 1 Similarity measurement on four groups of data sets
[0146]
[0147] As can be seen from Table 1, the hyperspectral remote sensing image generation method based on the semi-supervised transfer spectral reconstruction network of the present example has smaller RMSE, MRAE and SAM than direct training or pre-training spectral reconstruction, indicating that the hyperspectral image generated by the present method has higher similarity with the real hyperspectral image after combining the pre-training network information and the overlapping region information, and can effectively generate the spectral features of the hyperspectral image.
[0148] In summary, in order to conveniently and low-costly obtain a hyperspectral remote sensing image, the present embodiment provides a hyperspectral remote sensing image generation method based on a semi-supervised transfer spectral reconstruction network, which can effectively generate a corresponding wide-width high-resolution hyperspectral remote sensing image based on the multispectral image and the hyperspectral image with the overlapping region.
[0149] The present application can also have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, but these corresponding changes and modifications should all belong to the protection scope of the claims attached to the present application.
Claims
1. A method for generating a hyperspectral remote sensing image based on a semi-supervised transfer spectral reconstruction network, characterized in that: The method specifically comprises the following steps: Step 1, constructing and training a pre-training spectral reconstruction network to obtain a trained pre-training spectral reconstruction network; Step 2, selecting a real multispectral image simultaneously imaged with the hyperspectral image and existing an overlapping region, and cutting the real multispectral image into image blocks as a training set, the overlapping region in the training set as a labeled sample, and the non-overlapping region as an unlabeled sample, and training a region discrimination network in a semi-supervised learning manner by combining the labeled sample and the unlabeled sample to obtain a trained region discrimination network; Step 3, selecting the multispectral image and the hyperspectral image in the overlapping region in step 2, and cutting the multispectral image and the hyperspectral image into image blocks to construct a hyperspectral-multispectral image supervised training set; selecting the multispectral image in the non-overlapping region, and cutting the multispectral image into image blocks to construct a multispectral unsupervised training set; Step 4, constructing a semi-supervised transfer spectral reconstruction network; The semi-supervised transfer spectral reconstruction network comprises the pre-training spectral reconstruction network, the region discrimination network and a transfer network; The semi-supervised transfer spectral reconstruction network is trained in a supervised learning and unsupervised learning manner by alternately using the hyperspectral-multispectral image supervised training set and the multispectral unsupervised training set to obtain a trained semi-supervised transfer spectral reconstruction network; The specific process is as follows: The transfer network is composed of five groups of 3D spectral self-attention blocks connected in series; The input of the region discrimination network is consistent with that of the pre-training spectral reconstruction network, and the output is 0 or 1; The output of the pre-training spectral reconstruction network is used as the input of the transfer network; The output of the transfer network is multiplied by the output of the region discrimination network, and then added to the output of the pre-training spectral reconstruction network, and the sum is used as the output of the semi-supervised transfer spectral reconstruction network; The 3D spectral self-attention block comprises a spectral self-attention block, a first 3D residual block, a second 3D residual block, a third 3D residual block, a fourth 3D residual block, a 3D concatenation layer and a 3D post-processing layer; The output feature of the spectral self-attention block is input into the first 3D residual block, the output feature of the first 3D residual block is input into the second 3D residual block, the output feature of the second 3D residual block is input into the third 3D residual block, and the output feature of the third 3D residual block is input into the fourth 3D residual block; The output features of the spectral self-attention block, the first 3D residual block, the second 3D residual block, the third 3D residual block and the fourth 3D residual block are respectively input into the 3D concatenation layer for feature stacking; The output feature of the 3D concatenation layer is input into the 3D post-processing layer, and the output feature of the 3D post-processing layer is used as the output feature of the 3D spectral self-attention block; The 3D post-processing layer comprises a 3D convolution layer, an activation layer and a channel attention block in sequence; The specific structure of the spectral self-attention block is as follows: The input image features are respectively input into a full connection layer to obtain features A and B; Features A and B are multiplied to obtain feature C; Feature C is input into a softmax to obtain feature D; Feature D is multiplied with the input image features to obtain feature E; Feature E is input into a full connection layer to obtain feature F; Feature F is the output feature of the spectral self-attention block; Step 5, the trained semi-supervised transfer spectral reconstruction network is used for spectral reconstruction of a to-be-tested multi-spectral image, and a hyperspectral remote sensing image is generated.
2. The hyperspectral remote sensing image generation method based on the semi-supervised transfer spectral reconstruction network according to claim 1, characterized in that: The detailed steps of the training process of the pre-trained spectral reconstruction network in step 1 include: Spectral resampling is performed on existing hyperspectral data sets by using a spectral response function of a multi-spectral satellite sensor to obtain multi-spectral data corresponding to the hyperspectral data; the real hyperspectral data and the resampled multi-spectral data are used as training data sets, and the network parameters of the pre-trained spectral reconstruction network are optimized by using the training data sets to obtain the trained pre-trained spectral reconstruction network.
3. The hyperspectral remote sensing image generation method based on semi-supervised transfer spectral reconstruction network according to claim 2, characterized in that: The spectral resampling is performed on existing hyperspectral data sets by using a spectral response function of a multi-spectral satellite sensor to obtain multi-spectral data corresponding to the hyperspectral data; the specific process is as follows: 1) the spectral response function of the multispectral satellite sensor used for the query and the wavebands used for the hyperspectral image; with and respectively as continuous spectral data and multispectral data, with R as the spectral response function, then the i-th waveband of the hyperspectral image and and R can be expressed as: where λ is any wavelength within the spectral response function R i denotes the spectral response function for the i-th band; 2) Adjusting the spectral response function of a multispectral satellite sensor to a normalized spectral response function R from hyperspectral to multispectral H , the normalized spectral response function R H represents the mapping relationship from hyperspectral image to multispectral image, then The relationship between the i-th band of the hyperspectral curve and R H can be expressed as: wherein, represents hyperspectral data, h represents the number of hyperspectral bands, R Hi represents the normalized spectral response function for the i-th band; 3) Multiplying the hyperspectral image with the hyperspectral-to-multiplespectral normalization spectral response function R H M H H M H H H . 4. The hyperspectral remote sensing image generation method based on semi-supervised transfer spectral reconstruction network according to claim 3, characterized in that: The pre-trained spectral reconstruction network in step 1 uses any deep learning-based spectral reconstruction network.
5. The hyperspectral remote sensing image generation method based on semi-supervised transfer spectral reconstruction network according to claim 4, characterized in that: The regional discrimination network in step 2 sequentially includes a convolutional layer, an activation layer, a convolutional layer, an activation layer, a convolutional layer, an activation layer, a convolutional layer, an activation layer, a flattening layer, a fully connected layer, a fully connected layer, a fully connected layer, and a Softmax layer.
6. The hyperspectral remote sensing image generation method based on semi-supervised transfer spectral reconstruction network according to claim 5, characterized in that: The detailed steps of the training process of the regional discrimination network in step 2 include: (1) selecting real multi-spectral images that are simultaneously imaged with the hyperspectral image and have overlapping regions, and cropping the images into image blocks as a training set, and assigning a label 1 to the overlapping regions in the training set as labeled samples and assigning a label 0 to the non-overlapping regions as unlabeled samples; (2) optimizing the network parameters of the regional discrimination network by using the training set; (3) after each iteration of the training set, all unlabeled samples are discriminated by the regional discrimination network; the specific process is as follows: values greater than or equal to 0.5 obtained by the Softmax layer of the regional discrimination network are determined as label 1, and values less than 0.5 are determined as label 0; samples in the unlabeled samples that are determined as label 1 and have values greater than or equal to 0.8 are converted from label 0 to label 1; samples in the unlabeled samples that are determined as label 0 and have values less than 0.2 are converted from label 1 to label 0; (4) stopping after 100 iterations or when the number of samples converted to label 1 exceeds 50%, and obtaining the trained regional discrimination network.
7. The hyperspectral remote sensing image generation method based on semi-supervised transfer spectral reconstruction network according to claim 6, characterized in that: The total system response of the semi-supervised transfer spectral reconstruction network in step 4 is represented as HSI SR = H Basic (MSI; θ1) + H Dis (MSI; θ2) x H Transfer (H Basic (MSI; θ3) where H Basic denotes the system response of the pre-trained spectral reconstruction network, H Dis denotes the system response of the region discrimination network, H Transfer denotes the system response of the transfer network, θ1 denotes the network parameters of the pre-trained spectral reconstruction network, θ2 denotes the network parameters of the region discrimination network, θ3 denotes the network parameters of the transfer network, MSI is the input multispectral data, HSI SR is the reconstructed hyperspectral data output by the semi-supervised transfer spectral reconstruction network. The multispectral image is input into the trained region discrimination network, and when the value obtained by the Softmax layer of the region discrimination network is greater than or equal to 0.5, the region discrimination network outputs H Dis (MSI; θ2) is 0, and at this time, the system response of the semi-supervised transfer spectral reconstruction network is HSI SR = H Basic (MSI; θ1) When the value obtained by the region discrimination network Softmax layer is less than 0.5, at this time the region discrimination network output H Dis (MSI; θ2) is 1, at this time the semi-supervised transfer spectral reconstruction network system response is HSI SR = H Basic (MSI; θ1) + H Transfer (H Basic (MSI; θ3)) The system response of the transfer network is represented as I Transfer = H Transfer (I Pre ) = H 3-5 (H 3-4 (H 3-3 (H 3-2 (H 3-1 (I Pre ))))) where H 3-1 , H 3-2 , H 3-3 , H 3-4 , and H 3-5 are the system responses of 5 3D spectral self-attention blocks, respectively, I Pre is the pre-reconstruction hyperspectral image obtained by inputting the multispectral image into the pre-trained spectral reconstruction network trained in step 1, I Transfer is the migration residual result output by the migration network.
8. The hyperspectral remote sensing image generation method based on semi-supervised transfer spectral reconstruction network according to claim 7, characterized in that: In the training process of the semi-supervised transfer spectral reconstruction network in step 4, the network is constrained by using the high-multi-spectral image supervised training set in step 3 in a supervised learning manner; specifically: The semi-supervised transfer spectral reconstruction network parameters are optimized by minimizing the root mean square error loss function shown in the following formula; Loss = L(H Basic (MSI; θ1) + H Dis (MSI; θ2) x H Transfer (H Basic (MSI; θ3), HSI) wherein Loss is a loss function, L is a root mean square error loss function, H W is a system response of the complete network, MSI is input multispectral data, HSI is original hyperspectral data overlapping with the input multispectral data MSI, and θ is a semi-supervised transfer spectral reconstruction network parameter; In the training process of the semi-supervised transfer spectral reconstruction network in step 4, the network is constrained by using the multi-spectral unsupervised training set in step 3 in an unsupervised learning manner; specifically: The semi-supervised transfer spectral reconstruction network parameters are optimized by minimizing the sum of the scaling loss function Loss1 and the transfer loss function Loss2; the expression is as follows: Loss = Loss1 + aLoss2 wherein Loss is a loss function, and a is a weight; The scaling loss function value Loss1 calculation process is: The specific expression of the scaling loss function is as follows: Lossl = L(0.5 x H W (2 x MSI; θ), H W (2 x H W (0.5 x MSI; θ), H W (2 x H wherein, H W (2 x MSI; theta) = (H Basic (2 x MSI; θ1) + H Dis (2 x MSI; θ2) x H Transfer (H Basic (2 x MSI; θ3) H W (MSI; θ) = (H Basic (1 x MSI; θ1) + H Dis (1 x MSI; θ2) x H Transfer (H Basic (1 x MSI; θ3) H W (0.5 x MSI; θ) = (H Basic (0.5 x MSI; θ1) + H Dis (0.5 x MSI; θ2) x H Transfer (H Basic (0.5 x MSI; θ3)) where L is the root mean square error loss function, H W is the total system response of the semi-supervised transfer spectral reconstruction network, MSI is the input multispectral data, and θ is the semi-supervised transfer spectral reconstruction network parameter; The migration loss function value Loss2 calculation process is: Step 1, inputting the multispectral image into the pre-trained spectral reconstruction network trained to obtain a pre-reconstructed hyperspectral image; Step 4, inputting the multispectral image into the semi-supervised migration spectral reconstruction network to obtain a reconstructed hyperspectral image; Step 5, calculating the migration loss function value between the pre-reconstructed hyperspectral image and the reconstructed hyperspectral image.
9. The hyperspectral remote sensing image generation method based on semi-supervised transfer spectral reconstruction network according to claim 8, characterized in that: The specific process of calculating the migration loss function value between the pre-reconstructed hyperspectral image and the reconstructed hyperspectral image is: The migration loss function adopts a gradient direction consistent loss function, and the specific calculation process of the gradient direction consistent loss function is: I. Generate the gradient direction sequence corresponding to the pre-reconstructed hyperspectral image; the specific process is:
1. Calculate the spectral difference value of every four bands in the spectral curve of each pixel in the pre-reconstructed hyperspectral image as the gradient of the current band; 2. Calculate the average value of the gradients of the adjacent four bands in 1 as the average gradient; 3. Sample the average gradient in 2 every three bands to obtain the sampled average gradient; 4. Normalize the sampled average gradient in 3 to -1 to +1 to obtain the normalized gradient; 5. Assign +1 to the normalized gradient in 4 greater than 0.3, assign -1 to the normalized gradient less than -0.3, and assign 0 to the intermediate, thereby generating the gradient direction sequence corresponding to the pre-reconstructed hyperspectral image; II. Generate the gradient direction sequence corresponding to the reconstructed hyperspectral image; the specific process is: 1). Calculate the spectral difference value of every four bands in the spectral curve of each pixel in the reconstructed hyperspectral image as the gradient of the current band; 2). Calculate the average value of the gradients of the adjacent four bands in 1) as the average gradient; 3). Sample the average gradient in 2) every three bands to obtain the sampled average gradient; 4). Normalize the sampled average gradient in 3) to -1 to +1 to obtain the normalized gradient; 5). Assign +1 to the normalized gradient in 4) greater than 0.3, assign -1 to the normalized gradient less than -0.3, and assign 0 to the intermediate, thereby generating the gradient direction sequence corresponding to the reconstructed hyperspectral image; III. Calculate the RMSE value of the gradient direction sequence corresponding to the pre-reconstructed hyperspectral image generated in I and the gradient direction sequence corresponding to the reconstructed hyperspectral image generated in II, and the RMSE value is taken as the migration loss function value between the pre-reconstructed hyperspectral image and the reconstructed hyperspectral image; The RMSE is the root mean square error between the gradient direction sequence corresponding to the pre-reconstructed hyperspectral image and the gradient direction sequence corresponding to the reconstructed hyperspectral image.