Hyperspectral image change detection method based on unsupervised spectral unmixing neural network

Abstract

This invention addresses the technical problems of existing unsupervised change detection algorithms, which mainly rely on algebraic transformations and spectral unmixing of spectra, resulting in poor performance and limited applicability in hyperspectral data with mixed pixels. It provides a hyperspectral image change detection method based on an unsupervised spectral unmixing neural network. The method includes the following steps: 1. Segmenting and mixing two hyperspectral images from different time phases to obtain hyperspectral image patch data, which is used as the training dataset; 2. Constructing a spectral unmixing network to generate endmember matrices and abundance matrices with image features; 3. Reconstructing the hyperspectral image patch data based on the endmember matrices and abundance matrices, and selecting reconstruction errors to train the spectral unmixing network; 4. Generating a change grayscale image based on the trained spectral unmixing network and the abundance matrix output by the abundance matrix generation module; 5. Binarizing the change grayscale image using a threshold segmentation algorithm to obtain the final hyperspectral image change detection result.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method for detecting hyperspectral image changes based on an unsupervised spectral unmixing neural network. Background Technology

[0002] Change detection is a method designed to identify changes occurring in the same geographic area at different observation times. It plays a vital role in fields such as urban planning, environmental monitoring, and precision agriculture. In recent years, with the development of hyperspectral remote sensing technology, thanks to its higher spectral resolution, change detection using hyperspectral remote sensing images has yielded even better results.

[0003] Influenced by the development of deep learning, many change detection algorithms based on convolutional neural networks have been proposed, such as the early end-to-end fully connected early fusion (FC-EF) change detection algorithm. Furthermore, to improve the ability to extract change features, residual networks, U-Nets, and 3D convolutions have also been gradually applied to the field of change detection. However, most convolutional network-based algorithms are supervised algorithms, requiring prior knowledge of the ground truth of the change, and their application scenarios are limited, making it difficult to generalize these algorithms to practical applications.

[0004] Traditional unsupervised change detection algorithms primarily rely on algebraic transformations of the spectrum to convert spectral data into a new variable space for change detection. Examples include Iteratively Reweighted Multivariate Alteration Detection (IR-MAD), change vector analysis based on principal component analysis, subspace-based change detection (SCD), and slow feature analysis. However, these methods are all pixel-level change detection methods and perform poorly on hyperspectral data with mixed pixels.

[0005] To address the problem of mixed pixels in change detection, some researchers have proposed change detection algorithms based on spectral unmixing in recent years to obtain sub-pixel level change information. However, since these algorithms require a spectral library similar to the types of land features in the detected image to achieve good results, their applicability is greatly limited compared to traditional unsupervised algorithms.

[0006] Therefore, there is a need for a hyperspectral change detection algorithm that can solve mixed pixels and has wide applicability, but existing methods are difficult to meet this requirement. Summary of the Invention

[0007] The purpose of this invention is to address the technical problems of existing unsupervised change detection algorithms, which are mainly based on algebraic transformation and spectral unmixing, and have poor performance and limited applicability in hyperspectral data with mixed pixels. Therefore, this invention provides a hyperspectral image change detection method based on an unsupervised spectral unmixing neural network.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for detecting hyperspectral image changes based on an unsupervised spectral unmixing neural network, characterized by the following steps:

[0010] Step 1: Obtain two images from different time periods, both of which are [images with different shapes]. R×C×P hyperspectral images X 1 and X 2. Divide and mix them separately to obtain several shapes. H×W×P The hyperspectral image patches were used as the training dataset, and their spectral information was used as the training dataset.

[0011] in, R The number of rows in the hyperspectral image. C The number of columns in the hyperspectral image. P is the number of spectrum segments; H This represents the row number of the segmented hyperspectral image patch. W This represents the number of columns in the segmented hyperspectral image block; H < R, W < C ;

[0012] Step 2: Construct a spectral unmixing network and use the training data from the training dataset to generate an abundance matrix and an endmember matrix with image features;

[0013] Step 3: Based on the abundance matrix and endmember matrix with image features, the reconstructed hyperspectral image patch data is obtained through the reconstruction module, and the reconstruction error is selected as the loss function to train the spectral unmixing network.

[0014] Step 4: Based on the trained spectral unmixing network, generate a changing grayscale image according to the abundance matrix;

[0015] Step 5: Use a threshold segmentation algorithm to segment the changing grayscale values. Figure 2 Values ​​are then converted to obtain the final hyperspectral image change detection results.

[0016] Furthermore, step 2 specifically involves:

[0017] 2.1 Constructing a spectral unmixing network:

[0018] The spectral unmixing network includes an encoding module, an abundance matrix generation module, an endmember matrix generation module, and a reconstruction module. The input of the encoding module is used to receive the segmented hyperspectral image block data. The output of the encoding module is connected to the inputs of both the abundance matrix generation module and the endmember matrix generation module. The outputs of both the abundance matrix generation module and the endmember matrix generation module are connected to the input of the reconstruction module.

[0019] The encoding module is used for preliminary extraction of image features and data preprocessing of the hyperspectral image; the abundance matrix generation module is used for extracting image features of the input hyperspectral image patch and generating the abundance matrix of the hyperspectral image; the endmember matrix generation module is used for generating the endmember matrix of the hyperspectral image; the reconstruction module is used for reconstructing the hyperspectral image patch based on the endmember matrix and the abundance matrix with image features.

[0020] 2.2 Generating endmember matrices and abundance matrices using training data:

[0021] The training dataset obtained in step 1 is input into the spectral unmixing network, and the abundance matrix and endmember matrix with image features are obtained by training through the abundance matrix generation module and the endmember matrix generation module, respectively.

[0022] Furthermore, step 4 specifically involves:

[0023] 4.1. Hyperspectral images from different time phases X 1 and X 2. Combine the data in the row direction to synthesize a hyperspectral image. X hyperspectral images X The shape is 2 R×C×P And its spectral data is input into the pre-trained spectral unmixing network;

[0024] 4.2. After learning, the abundance matrix generated by the abundance matrix generation module is obtained. A Its shape is 2RC×N Then the hyperspectral image X The abundance matrix corresponding to 1 A 1 is the abundance matrix A The former RC Okay, hyperspectral image X The abundance matrix corresponding to 2 A 2 is the abundance matrix A After RC Okay, that is ;

[0025] 4.3 Calculation of hyperspectral images X 1 and X 2 Abundance Matrix Difference A 1- A 2, and sum along the column directions to obtain the shape as RC A grayscale vector of size ×1;

[0026] 4.4. Perform normalization processing, scaling the grayscale vector result to 0~1, and then perform shape transformation on the grayscale vector to obtain the shape as follows. R×C hyperspectral images X 1 and X Grayscale image showing the changes between 2 and 2.

[0027] Furthermore, step 5 specifically includes:

[0028] A threshold segmentation algorithm is used to segment the changing grayscale image to obtain the global threshold of the changing grayscale image. T For grayscale images with varying values ​​greater than or equal to T The pixels with a value of 1 are assigned a value of 0, and the rest of the pixels are assigned a value of 0, thus obtaining a binary grayscale change map. In the binary change map, a pixel with a value of 1 represents that the pixel has changed, and a value of 0 represents that the pixel has not changed, thus completing the detection of hyperspectral image changes.

[0029] Furthermore, in step 2:

[0030] The encoding module includes a convolutional layer, a LeakyReLU activation layer, and a first shape transformation unit connected in sequence; the convolutional kernel size of the convolutional layer is 3×3, and the number of convolutional kernels is [missing information]. P ;

[0031] The abundance matrix generation module includes at least one fully connected layer, and the number of output nodes of the last fully connected layer is . N , N ≤ P ;

[0032] The endmember matrix generation module includes two parallel feature extraction channels and a first matrix multiplication unit and a Softmax activation layer connected in sequence; one feature extraction channel includes a fully connected layer, and the other includes at least one fully connected layer and a transpose unit connected in sequence, with the last fully connected layer having a certain number of output nodes. N The outputs of both feature extraction channels are connected to the first matrix multiplication unit.

[0033] The reconstruction module includes a second matrix multiplication unit and a second shape transformation unit connected in sequence.

[0034] Further, in step 1, the hyperspectral images are respectively... X 1 and X 2. Divide the space into segments with a step size of 8 and a size of 32×32.

[0035] Further, in step 3, the reconstruction error is the root mean square error between the input hyperspectral image patch data and the reconstructed hyperspectral image patch data;

[0036] The loss function is:

[0037] in x (i,j) For the input hyperspectral image patch data, the location is in ( i,j The spectral vector of the pixel at position (). y (i,j) The location in the reconstructed hyperspectral image patch data is ( i,j The spectral vector of the pixel at position ).

[0038] Furthermore, in step 2, the abundance matrix generation module includes two fully connected layers; N=20.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] 1. The hyperspectral image change detection method based on unsupervised spectral unmixing neural network provided by this invention obtains the endmember matrix and abundance matrix of two temporal hyperspectral data based on the unsupervised spectral unmixing neural network. By using the idea of ​​spectral unmixing, each pixel is decomposed into the abundance matrix of several endmembers, thereby obtaining sub-pixel level information of the hyperspectral image to solve the problem of mixed pixels in hyperspectral change detection. Change detection based on sub-pixel level information can reduce the negative impact of mixed pixels and improve the accuracy of change detection.

[0041] 2. The hyperspectral image change detection method based on unsupervised spectral unmixing neural network provided by the present invention uses unsupervised spectral unmixing neural network for training and generating endmember matrix. It does not require prior knowledge of the actual type of ground objects and the true value of changes. It can automatically generate the corresponding endmember matrix and abundance matrix based on the two hyperspectral images to be tested, which greatly improves the applicability of the algorithm.

[0042] 3. The hyperspectral image change detection method based on unsupervised spectral unmixing neural network provided by the present invention involves a neural network structure with few parameters that need to be trained and a fast training speed. The trainable parameters are only in one convolutional layer and five fully connected layers. The shape change operation, transpose operation, and matrix multiplication operation in the network do not contain parameters that need to be trained. Attached Figure Description

[0043] Figure 1 This is a flowchart of the hyperspectral image change detection method based on an unsupervised spectral unmixing neural network of the present invention;

[0044] Figure 2 This is a schematic diagram of the unsupervised spectral unmixing neural network structure in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, advantages, and features of this invention clearer, the following detailed description of a hyperspectral image change detection method based on an unsupervised spectral unmixing neural network, proposed in this invention, is provided in conjunction with the accompanying drawings and specific embodiments. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this invention and are not intended to limit the scope of protection of this invention.

[0046] like Figure 1 As shown, the hyperspectral image change detection method based on an unsupervised spectral unmixing neural network proposed in this embodiment specifically includes the following steps:

[0047] Step 1: Acquire two hyperspectral images from different time phases. X 1 and X 2. Divide and mix them separately to obtain several shapes. H×W×P The hyperspectral image patches were used as the training dataset;

[0048] Hyperspectral images X 1 and X The shapes of 2 are all R×C×P In this embodiment, hyperspectral images are respectively... X 1 and X 2. Divide the image into segments with a step size of 8 and a spatial size of 32×32, and then combine the hyperspectral images from the two time phases. X 1 and X The data blocks that were split off are mixed together in a random order to obtain the training dataset.

[0049] in, R The number of rows in the hyperspectral image. C The number of columns in the hyperspectral image. P is the number of spectrum segments; H This represents the row number of the segmented hyperspectral image patch. W This represents the number of columns in the segmented hyperspectral image block.

[0050] Step 2: Construct a spectral unmixing network and use the training data in the training dataset to generate an abundance matrix and an endmember matrix with image features;

[0051] 2.1 Constructing a spectral unmixing network

[0052] like Figure 2 As shown, the spectral unmixing network includes an encoding module, an abundance matrix generation module, an endmember matrix generation module, and a reconstruction module. The input of the encoding module receives the segmented hyperspectral image block data, and the output of the encoding module is connected to the inputs of both the abundance matrix generation module and the endmember matrix generation module. The outputs of both the abundance matrix generation module and the endmember matrix generation module are connected to the input of the reconstruction module.

[0053] The encoding module includes a convolutional layer, a LeakyRelu activation layer, and a first shape transformation unit connected in sequence, which are used for preliminary extraction of image features and data preprocessing of hyperspectral images.

[0054] The abundance matrix generation module includes at least one fully connected layer, and the number of output nodes of the last fully connected layer is [value missing]. N , N ≤ P It is used to extract features from the input hyperspectral image patch and generate the abundance matrix of the hyperspectral image.

[0055] The endmember matrix generation module includes two parallel feature extraction channels, a first matrix multiplication unit, and a Softmax activation layer. One feature extraction channel includes a fully connected layer, and the other includes two fully connected layers and a transpose unit connected in sequence. The outputs of both feature extraction channels are connected to the first matrix multiplication unit, multiplied, and then enter the Softmax activation layer to generate the endmember matrix of the hyperspectral image.

[0056] The reconstruction module includes a second matrix multiplication unit and a second shape transformation unit connected in sequence, used to reconstruct hyperspectral image patches based on the endmember matrix and the abundance matrix with image features.

[0057] In this embodiment, the kernel size of the convolutional layer of the encoding module is 3×3, and the number of convolutional kernels is [number missing]. P The first shape transformation unit transforms the shape of the input hyperspectral image patch from... H×W×P Convert to HW×P The feature map.

[0058] The abundance matrix generation module consists of two fully connected layers, and the number of output nodes of the two fully connected layers are respectively P and N The number of output nodes in one of the fully connected layers for extracting feature channels in the endmember matrix generation module is... P The number of output nodes of the two fully connected layers in the other feature extraction channel are respectively P and N After repeated verification, when N When the value is 20, the feature data extraction effect of hyperspectral image blocks is the best.

[0059] 2.2 Generating endmember matrices and abundance matrices using training data

[0060] The training data is input into the spectral unmixing network, and the abundance matrix and endmember matrix with image features are obtained by training through the abundance matrix generation module and the endmember matrix generation module, respectively.

[0061] Step 3: Based on the abundance matrix and endmember matrix with image features, the reconstructed hyperspectral image patch data is obtained through the reconstruction module. A suitable reconstruction error is selected as the loss function to train the spectral unmixing network.

[0062] The abundance matrix and endmember matrix containing image features are sequentially passed through the second matrix multiplication unit and the second shape transformation unit, and the shape of the hyperspectral image patch is then transformed by... HW×P Convert to H×W×P .

[0063] In this embodiment, the root mean square error between the input hyperspectral image patch data and the reconstructed hyperspectral image patch data is used as the loss function. L As shown below:

[0064]

[0065] in x (i,j) For the input hyperspectral image patch data, the location is in ( i,j The spectral vector of the pixel at position (). y (i,j) The location in the reconstructed hyperspectral image patch data is ( i,j The spectral vector of the pixel at position ).

[0066] Step 4: Based on the trained spectral unmixing network, generate a changed grayscale image according to the abundance matrix output by the abundance matrix generation module;

[0067] 4.1. Hyperspectral images from different time phases X 1 and X 2. Combine the data in the row direction to synthesize a hyperspectral image. X hyperspectral images X The shape is 2 R×C×P And its spectral data is input into the pre-trained spectral unmixing network;

[0068] 4.2. After learning, the abundance matrix generated by the abundance matrix generation module is obtained. A Its shape is 2RC×N Then the hyperspectral image X The abundance matrix corresponding to 1 A 1 is the abundance matrix A The former RC Okay, hyperspectral image X The abundance matrix corresponding to 2 A 2 is the abundance matrix A After RC Okay, that is ;

[0069] 4.3 Calculation of hyperspectral images X 1 andX 2 Abundance Matrix Difference A 1- A 2, and sum along the column directions to obtain the shape as RC A grayscale vector of size ×1;

[0070] 4.4. Perform normalization processing to scale the grayscale vector result to the 0~1 range, and perform shape transformation on the grayscale vector to obtain the shape as follows. R×C hyperspectral images X 1 and X Grayscale image showing the changes between 2 and 2.

[0071] Step 5: Use a threshold segmentation algorithm to segment the changing grayscale values. Figure 2 Values ​​are then converted to obtain the final hyperspectral image change detection results.

[0072] Specifically, the OTSU algorithm (threshold segmentation algorithm) is used to segment the changing grayscale image to obtain the global threshold of the changing grayscale image. T For grayscale images with varying values ​​greater than or equal to T The pixels with a value of 1 are assigned a value of 0, and the rest of the pixels are assigned a value of 0, thus obtaining a binary grayscale change map. In the binary change map, a pixel with a value of 1 represents that the pixel has changed, and a value of 0 represents that the pixel has not changed. This completes the detection of hyperspectral image changes.

[0073] This invention obtains the endmember matrix and abundance matrix of two temporal hyperspectral data based on an unsupervised spectral unmixing neural network. By using the idea of ​​spectral unmixing, each pixel is decomposed into the abundance matrix of several endmembers, thereby obtaining sub-pixel level information of the hyperspectral image to solve the problem of mixed pixels in hyperspectral change detection. Change detection based on sub-pixel level information can reduce the negative impact of mixed pixels and improve the accuracy of change detection.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A method for detecting hyperspectral image changes based on an unsupervised spectral unmixing neural network, characterized in that, Includes the following steps: Step 1: Obtain two hyperspectral images X1 and X2 from different time phases, both with the shape R×C×P. Segment and mix them to obtain several hyperspectral image patches with the shape H×W×P. Use their spectral information as the training dataset. Where R is the number of rows in the hyperspectral image, C is the number of columns in the hyperspectral image, and P is the number of spectral bands; H is the number of rows in the segmented hyperspectral image block, and W is the number of columns in the segmented hyperspectral image block; H < R, W < C. Step 2: Construct a spectral unmixing network and generate an abundance matrix and endmember matrix with image features using the training data from the training dataset; specifically: 2.1 Constructing a spectral unmixing network: The spectral unmixing network includes an encoding module, an abundance matrix generation module, an endmember matrix generation module, and a reconstruction module. The input of the encoding module is used to receive the segmented hyperspectral image block data. The output of the encoding module is connected to the inputs of both the abundance matrix generation module and the endmember matrix generation module. The outputs of both the abundance matrix generation module and the endmember matrix generation module are connected to the input of the reconstruction module. The encoding module is used for preliminary extraction of image features and data preprocessing of the hyperspectral image; the abundance matrix generation module is used for extracting image features of the input hyperspectral image patch and generating the abundance matrix of the hyperspectral image; the endmember matrix generation module is used for generating the endmember matrix of the hyperspectral image; the reconstruction module is used for reconstructing the hyperspectral image patch based on the endmember matrix and the abundance matrix with image features. 2.2 Generating endmember matrices and abundance matrices using training data: The training dataset obtained in step 1 is input into the spectral unmixing network, and the abundance matrix and endmember matrix with image features are obtained by training through the abundance matrix generation module and the endmember matrix generation module, respectively. Step 3: Based on the abundance matrix and endmember matrix with image features, the reconstructed hyperspectral image patch data is obtained through the reconstruction module, and the reconstruction error is selected as the loss function to train the spectral unmixing network. Step 4: Based on the trained spectral unmixing network, generate a changing grayscale image according to the abundance matrix; specifically: 4.1 Combine hyperspectral images X1 and X2 from different time phases in the row direction to synthesize a hyperspectral image X. The shape of hyperspectral image X is 2R×C×P, and its spectral data is input into the trained spectral unmixing network. 4.2 After learning, the abundance matrix A generated by the abundance matrix generation module is obtained. Its shape is 2RC×N. Then, the abundance matrix A1 corresponding to the hyperspectral image X1 is the first RC rows of the abundance matrix A, and the abundance matrix A2 corresponding to the hyperspectral image X2 is the last RC rows of the abundance matrix A, that is... 4.3 Calculate the difference A1-A2 between the abundance matrices of hyperspectral images X1 and X2, and sum them in the column directions to obtain a grayscale vector of shape RC×1; 4.4 Perform normalization processing to scale the grayscale vector result to 0~1, and perform shape transformation on the grayscale vector to obtain the grayscale variation between the hyperspectral image X1 and X2 with shape R×C. Step 5: Use a threshold segmentation algorithm to binarize the changing grayscale image to obtain the final hyperspectral image change detection result.

2. The unsupervised spectral unmixing neural network based hyperspectral image change detection method of claim 1, wherein, Step 5 specifically involves: A threshold segmentation algorithm is used to segment the changing grayscale image to obtain a global threshold T. Pixels in the changing grayscale image with a value greater than or equal to T are assigned a value of 1, and the remaining pixels are assigned a value of 0, resulting in a binary grayscale change image. In the binary change image, a pixel with a value of 1 represents that the pixel has changed, and a value of 0 represents that the pixel has not changed, thus completing the detection of hyperspectral image changes.

3. The unsupervised spectral unmixing neural network based hyperspectral image change detection method according to claim 1 or 2, characterized in that, In step 2.1: The encoding module includes a convolutional layer, a LeakyReLU activation layer, and a first shape transformation unit connected in sequence; the convolutional kernel size of the convolutional layer is 3×3, and the number of convolutional kernels is P. The abundance matrix generation module includes at least one fully connected layer, and the number of output nodes of the last fully connected layer is N, where N≤P; The endmember matrix generation module includes two parallel feature extraction channels and a first matrix multiplication unit and a Softmax activation layer connected in sequence; one feature extraction channel includes a fully connected layer, and the other includes at least one fully connected layer and a transpose unit connected in sequence, and the number of output nodes of the last fully connected layer is N; the output terminals of both feature extraction channels are connected to the first matrix multiplication unit. The reconstruction module includes a second matrix multiplication unit and a second shape transformation unit connected in sequence.

4. The hyperspectral image change detection method based on an unsupervised spectral unmixing neural network according to claim 3, characterized in that: In step 1, the hyperspectral images X1 and X2 are divided into segments with a step size of 8 and a spatial size of 32×32.

5. The hyperspectral image change detection method based on an unsupervised spectral unmixing neural network according to claim 4, characterized in that: In step 3, the reconstruction error is the root mean square error between the input hyperspectral image patch data and the reconstructed hyperspectral image patch data; The loss function is: Where x (i,j) Given the spectral vector of the pixel at position (i,j) in the input hyperspectral image patch data, y (i,j) This is the spectral vector of the pixel at position (i,j) in the reconstructed hyperspectral image patch data.

6. The hyperspectral image change detection method based on an unsupervised spectral unmixing neural network according to claim 5, characterized in that: In step 2.1, the abundance matrix generation module includes two fully connected layers; N = 20.

Images