A hyperspectral image simulation method based on deep learning model

Abstract

The present invention discloses a hyperspectral image simulation method based on a deep learning model, which relates to the field of earth observation and remote sensing technology. The method extracts end-member hyperspectral reflectance from existing hyperspectral images and constructs a hyperspectral spectrum library. The method utilizes the spectral response function of a multispectral image to obtain the end-member multispectral reflectance of different ground objects with the same number of multispectral bands by calculation, constructs a multispectral spectrum library, adds an attention mechanism to pre-process the multispectral image data, uses the multispectral image data, the multispectral spectrum library and the hyperspectral spectrum library as the query, key and value of the attention mechanism, optimizes the input multispectral image data, and uses a neural network to optimize the detail points of the simulated spectral image after the linear combination of the end-members, thereby achieving a very good simulation effect.

Description

Technical Field

[0001] The present invention relates to the field of earth observation and remote sensing technology, and specifically to a hyperspectral image simulation method based on a deep learning model. Background Art

[0002] Hyperspectral remote sensing data has a high spectral resolution and contains richer information. However, due to the limitations of observation conditions, we cannot obtain hyperspectral images of the required area. Therefore, simulating hyperspectral remote sensing images based on multispectral images through simulation technology has always been one of the most important research directions in the field of remote sensing. The data collected by ground-based spectrometers used in existing models and algorithms are quite different from those onboard, and it is difficult to accurately restore the spectral curve of real space-borne hyperspectral images. Summary of the invention

[0003] The purpose of the present invention is to provide a hyperspectral image simulation method based on a deep learning model to solve the problems raised in the above background technology.

[0004] A hyperspectral image simulation method based on a deep learning model comprises the following steps:

[0005] S1-1, acquiring hyperspectral image data and multispectral image data;

[0006] S1-2, constructing hyperspectral and multispectral libraries;

[0007] S1-3, input the preprocessed data into the neural network layer NET to obtain simulated hyperspectral reflectance;

[0008] S1-4, obtain a simulated hyperspectral image according to the simulated hyperspectral reflectance, and end.

[0009] Specifically, in step S1-2, the construction of the hyperspectral spectral library and the multispectral spectral library includes the following steps:

[0010] S2-1, clustering the hyperspectral images into N types of objects, extracting the class centers of the N types of objects as the end-member hyperspectral reflectance of the objects, and constructing a hyperspectral spectral library;

[0011] S2-2, using the spectrum in the hyperspectral library as input, using the spectral response function of the multispectral, by calculating the end-member multispectral reflectance of different ground objects with the same number of multispectral bands, constructing a multispectral library;

[0012] S2-3, the reflectance of the multispectral image pixel is used as the query q t , t=1,...,n, n is the number of reflectance data of multispectral image pixels, multispectral end member spectral library and hyperspectral spectral library are used as key-value pairs, and m key-value pairs (k1 ,v 1 ),...,(k m ,v m ), m is the number of endmember spectra in the multispectral endmember library and the hyperspectral library.

[0013] Specifically, in step S2-1, the clustering method uses K-means clustering to cluster the hyperspectral image into N clusters, and extracts the class center of each cluster as the endmember spectrum of the class, where N is a positive integer;

[0014] In step S2-2, the spectral response function of the multi-spectrum is:

[0015]

[0016] Among them, LB and L(λ) are the signal energy values ​​(radiance brightness or reflectivity) of the remote sensing image band and the finer band in the spectral library respectively, and β(λ) represents the weight value of the spectral response function corresponding to different bands. β(λ) is obtained through experimental measurement or simulation. Through this method, the spectral energy can be redistributed within different bands.

[0017] Specifically, in step S1-3, the following steps are also included:

[0018] S5-1, set t=1;

[0019] S5-2, query q t The key-value pairs are input into the additive attention module to obtain the weighted value α of each end-member spectrum in the hyperspectral spectral library. t ;

[0020] S5-3, query q t And weighted values ​​are concatenated to obtain the input α of the neural network layer NET t ⊙q t , ⊙ means multiplication of corresponding elements;

[0021] S5-4, input the spliced ​​features into the neural network layer NET, which includes H linear layers and ReLU layers to obtain the predicted value of the hyperspectral pixel reflectance H is a positive integer, and a predicted value of hyperspectral pixel reflectance corresponds to a multispectral image pixel reflectance;

[0022] S5-5, calculating the loss value between the predicted value of the hyperspectral pixel reflectance and the true value of the hyperspectral pixel reflectance, and updating the linear layer parameters by a small batch gradient descent algorithm; the linear layer parameters include weights and biases;

[0023] S5-6, assign the value of t+1 to t, and repeat steps S5-2 to S5-6 until t=n+1, at which time all the concatenated features have completed a small batch gradient descent method to update the parameters;

[0024] S5-7, recalculate all hyperspectral pixel reflectance prediction values, and calculate the average of the loss values ​​between all hyperspectral pixel reflectance prediction values ​​and the true values, set the threshold and the maximum number of cycles, if the average of the loss values ​​is less than the threshold or reaches the maximum number of cycles, end the model training, save the parameters and end the model training, otherwise re-sort the multispectral image pixel reflectance data and enter S5-1.

[0025] In step S5-2, the additive attention module expression is:

[0026]

[0027] in, α is the attention weight function; is the vector space composed of all multispectral image reflectances, is the vector space composed of all endmember spectra in the multi-spectral endmember spectral library, It is the vector space composed of all end-member spectra in the hyperspectral spectral library.

[0028] The attention weight function is obtained by the following steps:

[0029] S7-1, first query q t and key k i Input linear layer W respectively q and W k , and get two output vectors W of the same dimension q q t and W k k i , W q and W k is an updateable parameter;

[0030] S7-2, after adding the two output vectors, pass through a tanh activation function layer;

[0031] S7-3, the result in step S7-2 is passed through an output feature size and value v i Linear layers with the same feature size Get the α(q before normalization t ,k i ), where the value is v i The feature size is equal to the number of bands in the hyperspectral image. is an updateable parameter;

[0032] S7-4, use the softmax function to normalize and get the final α(q t ,k i ), the final

[0033] The softmax function is used to normalize all values ​​to between 0 and 1, and the sum of all normalized values ​​is equal to 1.

[0034] In step S5-4, the linear layer is a fully connected layer, and the calculation formula of the fully connected layer is:

[0035]

[0036] Among them, z i is the output of the ith neuron in the fully connected layer, x j is the jth input feature of the neuron, x j The input X from the fully connected layer includes the output of the previous fully connected layer and the query q t Concatenate the weighted feature value α t ⊙q t , h is the total number of inputs to the neuron, U T For this fully connected layer, the weight, p, represents the bias of the fully connected layer, which is randomly initialized;

[0037] After the fully connected layer, the ReLU function is used as the activation function, and the calculation formula of the function is:

[0038] ReLU(z i )=max(0,z i ).

[0039] The weights and biases of the linear layer are updated using mini-batch gradient descent, which includes the following steps:

[0040] S8-1, specify the batch size A and learning rate η, and each A data is regarded as a batch;

[0041] S8-2, traverse and calculate the weight U of all data in a batch T And the gradient of bias p with respect to the loss value is calculated as follows:

[0042]

[0043]

[0044] Where L is the loss function and y is the output of the linear layer;

[0045] S8-3, calculate the average of all data weights and bias gradients in the batch;

[0046] S8-4, update weights and biases according to the average of gradients and learning rate:

[0047]

[0048] Specifically, the loss function is the mean square error function MSEloss, and the formula is:

[0049] is the predicted reflectance value of the hyperspectral pixel, y t is the true reflectance value of the hyperspectral pixel.

[0050] The mini-batch gradient descent method uses a small number of samples in each training iteration to estimate the gradient and update the parameters, which allows more frequent parameter adjustments and speeds up the convergence of the algorithm. The mini-batch gradient descent method updates the parameters by using the average gradient of a small sample set to make the gradient estimation more stable. Compared with the random gradient descent of a single sample, the mini-batch gradient descent can reduce the variance of the parameter update, resulting in a more stable training process. The mini-batch gradient descent method provides a way to introduce randomness and diversity in training, which helps prevent the model from falling into a local optimal solution, improves the generalization ability of the model, and reduces the risk of overfitting.

[0051] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the endmember spectra used to generate the model image are extracted from the remote sensing image to be simulated, which can make the spectral curve of the simulated image closer to the real hyperspectral image; a neural network is used to optimize the detail points of the simulated spectral image after the linear combination of the endmembers, thereby achieving a very good simulation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0053] Figure 1 This is a network structure diagram of a hyperspectral image simulation method based on a deep learning model in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] A hyperspectral image simulation method based on a deep learning model in an embodiment of the present invention comprises the following steps:

[0056] S1-1, acquiring hyperspectral image data and multispectral image data;

[0057] S1-2, constructing hyperspectral and multispectral libraries;

[0058] S1-3, input the preprocessed data into the neural network layer NET to obtain simulated hyperspectral reflectance;

[0059] S1-4, obtain a simulated hyperspectral image according to the simulated hyperspectral reflectance, and end.

[0060] In step S1-2, constructing a hyperspectral spectral library and a multispectral spectral library includes the following steps:

[0061] S2-1, clustering the hyperspectral images into N types of objects, extracting the class centers of the N types of objects as the end-member hyperspectral reflectance of the objects, and constructing a hyperspectral spectral library;

[0062] S2-2, using the spectrum in the hyperspectral library as input, using the spectral response function of the multispectral, by calculating the end-member multispectral reflectance of different ground objects with the same number of multispectral bands, constructing a multispectral library;

[0063] S2-3, the reflectance of the multispectral image pixel is used as the query q t , t=1,...,n, n is the number of reflectance data of multispectral image pixels, multispectral end member spectral library and hyperspectral spectral library are used as key-value pairs, and m key-value pairs (k 1 ,v 1 ),...,(k m ,v m ), m is the number of endmember spectra in the multispectral endmember library and the hyperspectral library.

[0064] In step S2-1, the K-means clustering method is used to cluster the hyperspectral image into N clusters, and the class center of each cluster is extracted as the endmember spectrum of the category, where N is a positive integer; the sum of squares of clustering errors SSE or other metrics under different cluster numbers are calculated, and a relationship between the number of clusters and the clustering error is plotted. When increasing the number of clusters cannot significantly reduce the clustering error, an obvious inflection point appears, which is the appropriate number of clusters N.

[0065] In step S2-2, the spectral response function of the multi-spectrum is:

[0066]

[0067] Among them, LB and L(λ) are the signal energy values ​​of the remote sensing image band and the finer band in the spectral library respectively, β(λ) represents the weight value of the spectral response function corresponding to different bands, and β(λ) is obtained through experimental measurement or simulation. Through this method, the spectral energy can be redistributed within different bands.

[0068] In step S1-3, the following steps are also included:

[0069] S5-1, set t=1;

[0070] S5-2, query q t The key-value pairs are input into the additive attention module to obtain the weighted value α of each end-member spectrum in the hyperspectral spectral library. t ;

[0071] S5-3, query q t And weighted values ​​are concatenated to obtain the input α of the neural network layer NET t ⊙q t , ⊙ means multiplication of corresponding elements;

[0072] S5-4, input the spliced ​​features into the neural network layer NET, which contains 3 linear layers and a ReLU layer to obtain the predicted value of the hyperspectral pixel reflectance H is a positive integer, and a predicted value of hyperspectral pixel reflectance corresponds to a multispectral image pixel reflectance;

[0073] S5-5, calculating the loss value between the predicted value of the hyperspectral pixel reflectance and the true value of the hyperspectral pixel reflectance, and updating the linear layer parameters by a small batch gradient descent algorithm; the linear layer parameters include weights and biases;

[0074] S5-6, assign the value of t+1 to t, and repeat steps S5-2 to S5-6 until t=n+1, at which time all the concatenated features have completed a small batch gradient descent method to update the parameters;

[0075] S5-7, recalculate all hyperspectral pixel reflectance prediction values, and calculate the average of the loss values ​​between all hyperspectral pixel reflectance prediction values ​​and the true values, set the threshold and the maximum number of cycles, if the average of the loss values ​​is less than the threshold or reaches the maximum number of cycles, end the model training, save the parameters and end the model training, otherwise re-sort the multispectral image pixel reflectance data and enter S5-1.

[0076] In step S5-2, the additive attention module expression is:

[0077]

[0078] in, α is the attention weight function; is the vector space composed of all multispectral image reflectances, is the vector space composed of all endmember spectra in the multi-spectral endmember spectral library, It is the vector space composed of all end-member spectra in the hyperspectral spectral library;

[0079] The attention weight function is obtained by the following steps:

[0080] S7-1, first query q t and key k i Input linear layer W respectively q and W k , and get two output vectors W of the same dimension q q t and W k k i , W q and W k is an updateable parameter;

[0081] S7-2, after adding the two output vectors, pass through a tanh activation function layer;

[0082] S7-3, the result in step S7-2 is passed through an output feature size and value v i Linear layers with the same feature size Get the α(q before normalization t ,k i ), where the value is v i The feature size is equal to the number of bands in the hyperspectral image. is an updateable parameter;

[0083] S7-4, use the softmax function to normalize and get the final α(q t ,k i ), the final

[0084] The softmax function is used to normalize all values ​​to between 0 and 1, and the sum of all normalized values ​​is equal to 1.

[0085] In step S5-4, the linear layer is a fully connected layer, and the calculation formula of the fully connected layer is:

[0086]

[0087] Among them, z i is the output of the ith neuron in the fully connected layer, v j is the jth input feature of the neuron, x jThe input X from the fully connected layer includes the output of the previous fully connected layer and the query q t Concatenate the weighted feature value α t ⊙q t , h is the total number of inputs to the neuron, U T is the weight of this fully connected layer, p represents the bias of this fully connected layer, and is randomly initialized;

[0088] After the fully connected layer, the ReLU function is used as the activation function, and the calculation formula of the function is:

[0089] ReLU(z i )=max(0,z i ).

[0090] The weights and biases of the linear layer are updated using mini-batch gradient descent, which includes the following steps:

[0091] S8-1, specify the batch size A as 200, the learning rate η as 0.05, and every 200 data as a batch. The batch size A and the learning rate η can also be set to other values ​​according to time conditions;

[0092] S8-2, traverse and calculate the weight U of all data in a batch T And the gradient of bias p with respect to the loss value is calculated as follows:

[0093]

[0094]

[0095] Where L is the loss function and y is the output of the linear layer;

[0096] S8-3, calculate the average of all data weights and bias gradients in the batch;

[0097] S8-4, update weights and biases according to the average of gradients and learning rate:

[0098]

[0099] Specifically, the loss function is the mean square error function MSEloss, and the formula is:

[0100] is the predicted reflectance value of the hyperspectral pixel, y t is the true reflectance value of the hyperspectral pixel.

[0101] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0102] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A hyperspectral image simulation method based on a deep learning model, characterized in that The following steps are involved: S1-1, acquiring hyperspectral image data and multispectral image data; S1-2, constructing a hyperspectral spectral library and a multispectral spectral library, including steps S2-1 to S2-3; S2-1, clustering the hyperspectral images into N types of objects, extracting the class centers of the N types of objects as the end-member hyperspectral reflectance of the objects, and constructing a hyperspectral spectral library; S2-2, using the spectrum in the hyperspectral library as input, using the spectral response function of the multispectral, by calculating the end-member multispectral reflectance of different ground objects with the same number of multispectral bands, constructing a multispectral library; S2-3, the reflectance of the multispectral image pixel is used as the query q t , t=1,...,n, n is the number of reflectance data of multispectral image pixels, multispectral end member spectral library and hyperspectral spectral library are used as key-value pairs, and m key-value pairs (k1,v1),...,(k m ,v m ), m is the number of endmember spectra in the multispectral endmember library and the hyperspectral library; S1-3, query q t The key-value pairs are input into the additive attention module to obtain the weighted value α of each end-member spectrum in the hyperspectral spectral library. t ; query q t And weighted values ​​are concatenated to obtain the input α of the neural network layer NET t ⊙q t , ⊙ means corresponding element multiplication; the concatenated features are input into the neural network layer NET to obtain the simulated hyperspectral reflectance; S1-4, obtain a simulated hyperspectral image according to the simulated hyperspectral reflectance, and end.

2. The hyperspectral image simulation method based on a deep learning model according to claim 1, characterized in that: In step S2-1, the K-means clustering method is used to cluster the hyperspectral image into N clusters, and the class center of each cluster is extracted as the endmember spectrum of the class, where N is a positive integer.

3. The hyperspectral image simulation method based on a deep learning model according to claim 1, characterized in that: In step S2-2, the spectral response function of the multi-spectrum is: Among them, LB and L(λ) are the signal energy values ​​of the remote sensing image band and the finer band in the spectral library respectively, β(λ) represents the weight value of the spectral response function corresponding to different bands, and β(λ) is obtained through experimental measurement or simulation. Through this method, the spectral energy can be redistributed within different bands.

4. The hyperspectral image simulation method based on a deep learning model according to claim 1, characterized in that: In step S1-3, the following steps are also included: S5-1, set t=1; S5-2, query q t The key-value pairs are input into the additive attention module to obtain the weighted value α of each end-member spectrum in the hyperspectral spectral library. t ; S5-3, query q t And weighted values ​​are concatenated to obtain the input α of the neural network layer NET t ⊙q t , ⊙ means multiplication of corresponding elements; S5-4, input the spliced ​​features into the neural network layer NET, which includes H linear layers and ReLU layers to obtain the predicted value of the hyperspectral pixel reflectance H is a positive integer, and a predicted value of hyperspectral pixel reflectance corresponds to a multispectral image pixel reflectance; S5-5, calculating the loss value between the predicted value of the hyperspectral pixel reflectance and the true value of the hyperspectral pixel reflectance, and updating the linear layer parameters by a small batch gradient descent algorithm; the linear layer parameters include weights and biases; S5-6, assign the value of t+1 to t, and repeat steps S5-2 to S5-6 until t=n+1, at which time all the concatenated features have completed a small batch gradient descent method to update the parameters; S5-7, recalculate all hyperspectral pixel reflectance prediction values, and calculate the average of the loss values ​​between all hyperspectral pixel reflectance prediction values ​​and the true values, set the threshold and the maximum number of cycles, if the average of the loss values ​​is less than the threshold or reaches the maximum number of cycles, end the model training, save the parameters and end the model training, otherwise re-sort the multispectral image pixel reflectance data and enter S5-1.

5. The method for simulating hyperspectral images based on a deep learning model according to claim 4, characterized in that: In step S5-2, the additive attention module expression is: in, t=1,...,m, α is the attention weight function; is the vector space composed of all multispectral image reflectances, is the vector space composed of all endmember spectra in the multi-spectral endmember spectral library, It is the vector space composed of all end-member spectra in the hyperspectral spectral library.

6. The hyperspectral image simulation method based on a deep learning model according to claim 5, characterized in that: The attention weight function is obtained by the following steps: S7-1, first query q t and key k i Input linear layer W respectively q and W k , and get two output vectors W of the same dimension q q t and W k k i , W q and W k is an updateable parameter; S7-2, after adding the two output vectors, pass through a tanh activation function layer; S7-3, the result in step S7-2 is passed through an output feature size and value v i Linear layers with the same feature size Get the α(q before normalization t ,k i ), where the value is v i The feature size is equal to the number of bands in the hyperspectral image. is an updateable parameter; S7-4, use the softmax function to normalize and get the final α(q t ,k i ), the final 7. The method for simulating hyperspectral images based on a deep learning model according to claim 4, characterized in that: The weights and biases are updated using the mini-batch gradient descent method, which includes the following steps: S8-1, specify the batch size A and learning rate η, and each A data is regarded as a batch; S8-2, traverse and calculate the gradient of weights and biases of all data in a batch relative to the loss value; S8-3, calculate the average of all data weights and bias gradients in the batch; S8-4, update the weights and biases according to the average of the gradient and the learning rate.

8. The method for simulating hyperspectral images based on a deep learning model according to claim 4, characterized in that: In step S5-7, the loss value is obtained by calculating the mean square error between the predicted value of the hyperspectral pixel reflectance and the true value of the hyperspectral pixel reflectance.

Images