Hyperspectral remote sensing image blind unmixing method and system based on deep convolution auto-encoder

Abstract

The invention provides a hyperspectral remote sensing image blind unmixing method and system based on a deep convolution auto-encoder, and belongs to the technical field of hyperspectral remote sensing image processing. The method employs a Web application architecture and an integrated convolution auto-encoder neural network algorithm, achieves the end member and abundance estimation of a hyperspectral image, and improves the accuracy of hyperspectral remote sensing image blind unmixing. Comprising the steps of file uploading and storage, hyperspectral image information extraction, online parameter adjustment, online unmixing, visual display and export of an unmixing result and the like. And the system is in a Web application mode, the interface is simple, the process is clear, the experience of the user on the hyperspectral image unmixing process is improved, and meanwhile, the economic cost is also greatly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hyperspectral remote sensing image processing, and in particular to a hyperspectral remote sensing image blind unmixing method and system based on a deep convolutional autoencoder. BACKGROUND

[0002] The research on hyperspectral remote sensing image unmixing technology can improve the performance of hyperspectral remote sensing technology in various application fields such as mineral exploration, environmental monitoring, and change monitoring, and can achieve sub-pixel level remote sensing application from pixel level. In recent years, with the development of artificial intelligence, new unmixing methods have emerged, and neural networks are one of them. Neural networks have unique advantages in data feature extraction and good results, and neural networks have been widely used in the application field of hyperspectral remote sensing technology, such as material classification and identification, change monitoring, etc. Although the application in unmixing problem is not much at present, it has good results in data feature extraction and great development and application space in this problem.

[0003] The communication tower is the core infrastructure of the modern communication network, and its structure health state directly affects the signal transmission quality and network stability. The traditional inspection mainly relies on manual or visible light camera, which has problems such as low efficiency and insufficient detail recognition. SUMMARY

[0004] In order to solve the above technical problems, the present application provides a hyperspectral remote sensing image blind unmixing method based on a deep convolutional autoencoder. The hyperspectral remote sensing technology can capture the fine spectral features of the tower surface, and the blind unmixing method combined with the deep convolutional autoencoder (DCAE) can realize the accurate detection and classification of the tower surface material, rust, coating peeling and other subtle abnormalities, and improve the intelligent level of inspection.

[0005] The technical scheme of the present application is:

[0006] A hyperspectral remote sensing image blind unmixing method based on a deep convolutional autoencoder, comprising the following steps:

[0007] a) The user uploads the hyperspectral image file (simulation data.mat or real data.img / .hdr) through the Web interface;

[0008] b) Analyze the file information, including image size, band number and endmember number (simulation data);

[0009] c) The user sets the unmixing parameters online: endmember number, iteration number, batch size, patch number, patch size, L2 penalty coefficient, learning rate;

[0010] d) Perform unmixing based on the convolutional autoencoder:

[0011] Encoder: extract spatial features by convolution layer (CONV1: 5x5x48, LeakyReLU), and the hidden layer outputs the abundance map by Softmax;

[0012] Decoder: linear reconstruction of the spectrum using a weight matrix, whose weights are directly used as the endmember matrix;

[0013] e) For simulation data: automatically match the result endmember with the reference endmember (minimum SAD principle), and calculate the SAD / RMSE index

[0014] f) The result visualization shows the endmember spectral curve, abundance map and loss curve, and supports PDF report export.

[0015] Further,

[0016] The expression of the spatial-spectral model is:

[0017]

[0018] Further,

[0019] Settings:

[0020] 1. Spatial-spectral joint unmixing model: convolution layer extracts spatial features, autoencoder hidden layer outputs abundance map, and decoder weight directly generates endmember matrix;

[0021] 2. Dual-mode processing framework: for simulation data (automatic matching of reference endmember / abundance) and real data (support for RGB band rendering);

[0022] 3. Parameter adaptive matching algorithm: automatic alignment of result endmember and reference endmember by minimizing SAD / RMSE.

[0023] In addition, the application also provides a system for implementing the above method, comprising

[0024] User information management and image file upload module, the system supports registering different users and login function, and provides file upload function after login, different users have different file libraries to store their uploaded files, the supported uploaded files are only simulation data "*.mat" and real hyperspectral image files, one is hyperspectral image header file "*.hdr" and the other is hyperspectral image information file "*.img", the files between different users will not be shared.

[0025] Hyperspectral data reading and image display module, the system will read the hyperspectral information file according to the type of the read file, obtain the number of bands, the size of the image, the number of endmembers in the simulation data, etc. And according to the information file, the image is displayed, and the user can also set the band value of R, G and B three colors to render the image.

[0026] The hyperspectral remote sensing image unmixing module integrates the convolutional autoencoder unmixing algorithm. In the module interface, the user can adjust some hyperparameters in the algorithm, such as the number of endmembers, the number of iterations, batch sample processing data, learning rate, etc. For simulation test data, the real endmember and real abundance matrix in the simulation data are read after unmixing, the result endmember matrix and result abundance matrix are calculated, and each endmember matrix and abundance map are displayed on the front end, and the spectral angle distance (SAD) of the endmember and the root mean square error (RMSE) of the abundance are calculated and displayed. For real data, the algorithm loss curve, endmember spectrum curve, abundance map, etc. are visualized after the algorithm is run.

[0027] The data result export and analysis module can export the results displayed on the page in the form of pdf, which is convenient for further analysis through the results after the unmixing algorithm is finished.

[0028] The beneficial effects of the present application are

[0029] The problems of complex operation and high cost of traditional unmixing software are solved, and the unmixing efficiency and user experience are significantly improved. The user experience of hyperspectral image unmixing process is improved, and the economic cost is greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a workflow diagram of the present application;

[0031] Figure 2 is a convolutional autoencoder diagram;

[0032] Figure 3 is a system basic model diagram;

[0033] Figure 4 is a system overall structure diagram;

[0034] Figure 5 is a system flowchart. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] I. System development and deployment environment

[0037] 1. Hardware configuration requirements:

[0038] Server side: Intel i7 and above processor, 32 GB RAM, NVIDIA RTX 3080 GPU (CUDA acceleration supported)

[0039] Client side: WebGL enabled browser (Chrome / Firefox), no need for dedicated hardware 2, software dependent libraries:

[0040]

[0041] Unlike the widely used LMM model, this model allows all pixels in a f x f neighborhood centered on a pixel to contribute to the reconstruction of that pixel. The present invention studies the estimation problem of the endmember matrix M and the abundance s p of each pixel in hyperspectral data, which is interpreted as a blind unmixing problem and solved using a CNN autoencoder.

[0042] The algorithm is trained on N patches of the image. The first layer in the network is the input layer. The second layer, CONV1, is a two-dimensional convolutional layer containing 48 feature maps with a filter size of 3 x 3 and using a LeakyReLU activation function in between. We apply batch normalization after the convolutional layer to speed up the learning. Next, spatial dropout is applied, a version of dropout more suitable for CNNs to reduce overfitting and improve generalization with a dropout rate of 0.2. This means that the entire feature map is set to 0 randomly with a probability of 20%.

[0043] Next is another convolutional layer, CONV 2, with R feature maps, filter size of 1 x 1, and using a LeakyReLU activation function. Again, batch normalization and spatial dropout are applied with a dropout rate of 0.2 after the convolutional layer, which is also the key to generating the abundance map; the next layer in the encoder part of the autoencoder performs ASC on each pixel. It applies a softmax function pixel-wise to the set of feature maps output by the previous layer, but first scales their values by a coefficient a, where a = 3.5. So the output of this layer is a feature map, which is also the abundance of each endmember, and their sum is 1.

[0044] The last layer in the network is a linear decoder layer. This is a convolutional layer with B feature maps, filter size f x f (where f is an odd number), and linear activation. This layer reconstructs the spectral features of a pixel by combining the output of the hidden layer through the decoder weights. If the filter size is 1 x 1, it is easy to see that the filter weights W are a B x R matrix with the number of endmembers as columns. If the filter size is f x f, we will get f 2a B x R matrix, because the filter is Moreover, the reconstruction formula for a pixel p in a patch can be rewritten directly as

[0045]

[0046] Table 2-1 M m and W m Matrix Index Table

[0047]

[0048] Continuation of Table 2-1

[0049]

[0050] where the single index m indexes the position in the f x f patch as shown in Table 2-1. Thus, for each position in the i-th row and j-th column of the f x f neighborhood, we have an element of the endmember matrix, i.e., a matrix of the same dimension as the endmember matrix. This expression can be rearranged to obtain

[0051] By comparing with the HSI spectral-spatial model (2-4), we can determine

[0052] s p = h c (2-8)

[0053]

[0054] Therefore, the abundance map is obtained by encoding the HSI with the autoencoder, and the endmembers are directly obtained from the weights of the decoder layers.

[0055] Algorithm advantages: Compared with the traditional autoencoder, the parameters of the filters in the convolutional layer can be shared in the convolutional autoencoder, greatly reducing the number of required parameters; the data can be down-sampled, and the model has good robustness to noise; it directly uses patches of hyperspectral data without using any pooling layer or up-sampling layer, which can well preserve spatial information.

[0056] The hyperparameters of the CNN algorithm include epoch (number of iterations), batch_size (batch size), num_patches (patch number), patch_size (patch size), learning_rate (learning rate), l2 (L2 penalty coefficient), etc.

[0057] Where epoch: represents the number of times the training set is put into the neural network model, a complete training is called an epoch; batch_size: refers to the number of samples in the batch, mainly used in the batch gradient descent algorithm (Batch Gradient Descent), which traverses all samples in the batch in each iteration and determines the optimal direction according to these samples. num_patches: In the convolutional neural network, instead of processing a whole picture at a time, the picture is divided into multiple small blocks, and the kernel or filter processes only one block of the picture at a time. This small block is called a patch, and then the filter moves to the next patch of the image; patch_size: refers to the size of each patch.

[0058] learning_rate: In the gradient descent method, the speed of obtaining the optimal parameter is determined by the learning rate, but if the learning rate is too large (the speed is too fast), it is easy to miss the optimal solution, the function cannot converge, and even diverge; on the contrary, if the learning rate is too small, the training efficiency may be too low and the calculation time may be too long. Generally set at (0.001-0.01) at the beginning; l2: regularization can prevent model overfitting, the larger the l2 regularization coefficient, the more obvious the regularization effect (which may also cause the model to be difficult to converge); if the coefficient is too small, the regularization effect is not obvious, and the model is prone to overfitting.

[0059] The simulation data set used in the experiment is mainly composed of image, reference end member, reference abundance, noise and other matrices, which can quantify its precision in end member extraction and abundance estimation to evaluate the current proposed CNN algorithm. The data used in the experiment are all linear mixing models, and the specific data sets are: LMM_32x32_3_50dB_1 (indicating that the image size is 32x32, there are 3 end members, and the noise is 50dB), LMM_48x48_3_20dB_, LMM_48x48_3_35dB_1, LMM_48x48_3_50dB_1, LMM_48x48_4_50dB_1, LMM_48x48_5_50dB_1, LMM_64x64_3_50dB_1, each of which has 224 wavebands. Two common real data sets, Samson and Urban, were also used in the experiment.

[0060] The Samson data set is a relatively simple real hyperspectral data set, and the original image size is 952 ×952 pixels, each pixel records 156 bands information, each wavelength covers the range from 401 nm to 889 nm, and the spectral resolution is 3.31 nm. However, due to the large size of the original data and high computational cost, only a part of the original image is taken for testing in the experiment, and the size of the region is 95 × 95 pixels, the data will not be seriously disturbed by noise. The endmembers contained in the data are mainly "soil", "tree", and "water".

[0061] The Urban data set is a real hyperspectral data set commonly used in hyperspectral remote sensing image unmixing. The size of the image is 307 × 307 pixels, each pixel represents a 2 × square meter area. The image has 210 bands ranging from 400 nm to 2500 nm, with a spectral resolution of 10 nm. The data is affected by dense water vapor and atmosphere. After removing channels 1-4, 76, 87, 101-111, 136-153, and 198-210, 162 bands are retained. The main endmembers present in the data are "asphalt", "grass", "tree", and "roof".

[0062] II. Key module implementation details

[0063] 1. Hyperspectral data parsing module

[0064] In the simulation data test experiment, spectral angle distance (SAD) and root mean square error (RMSE) are used as precision indicators to evaluate the accuracy of the endmembers and abundances extracted by the algorithm. The formula for calculating the endmember SAD is as follows

[0065]

[0066] The formula for calculating the abundance RMSE is as follows

[0067]

[0068] where R is the number of endmembers, and are the endmembers and estimated abundance matrices extracted by the algorithm, respectively, m i and s iis the real endmember and abundance matrix used as reference. RMSE is the root mean square error of abundance, the larger the value represents the greater the difference between the estimated abundance and the reference abundance, and vice versa; similarly, the smaller the SAD value represents the smaller the gap between the two endmembers. The experiment includes experiment 1 and experiment 2. Experiment 1 tests the effects of different parameter configurations of convolutional autoencoders on the unmixing of the simulation data set. Since different autoencoders are used, for the convenience of recording, the names of the various autoencoders and their configurations are shown in Table 2-2. Experiment 2 is the test analysis of the convolutional autoencoder on the real data set.

[0069] Table 2-2 Different autoencoder parameter configuration table

[0070]

[0071] (1) Test the effects of different parameter configurations of convolutional autoencoders on the unmixing of the simulation data set

[0072] This experiment first shows the AE-2 autoencoder algorithm unmixing performance on the LMM_48x48_5_50dB_1 data set, and then uses four different configurations of autoencoders to compare the unmixing effect. Among them, AE-2 is used on the LMM_48x48_5_50dB_1 data set

[0073] Comparison of the results of the endmembers on the LMM_48x48_5_50dB_1 data set with the real endmember curves.

[0074] Result analysis: The more endmembers in the unmixing data, the more difficult the unmixing is. The algorithm performs well on the overall unmixing of the 5-endmember data. For some relatively simple endmembers, such as endmember 1, the overall basic coincidence is good, and the smoothness is somewhat poor. For endmembers 2 and 4, although there are some differences with the real endmember in detail, the smoothness is the best, and in some smooth areas it is very close to the real value. For endmember 3, which is relatively complex, the fitting effect is the worst, and the curve is still very jittery. In terms of abundance estimation, the smaller the endmember abundance, the more difficult it is to estimate the abundance, and there are still many differences between the abundance map and the real abundance map. Then the effects of different parameter autoencoders on the unmixing performance need to be tested.

[0075] In order to avoid the limitations of the experiment, test the effects of different parameter autoencoders on the unmixing performance, then use autoencoders with different parameters to test all the simulation data. First, control the variables, divide the 7 simulation data into 3 groups according to the number of endmembers, size, and signal-to-noise ratio for testing. The first group of data has the same number of endmembers and signal-to-noise ratio, including LMM_32x32_3_50dB_1, LMM_48x48_3_50dB_1, and LMM_64x64_3_50dB_1. The second group of data has the same size and signal-to-noise ratio, including

[0076] LMM_48x48_3_50dB_1, LMM_48x48_4_50dB_1, LMM_48x48_5_50dB_1; the third group of data size, the number of end members is consistent, including LMM_48x48_3_20dB_1, LMM_48x48_3_35dB_1, LMM_48x48_3_50dB_1. Table 2-3, table 2-4, table 2-5 respectively give different parameters of autoencoder on data unmixing precision index end member SAD and abundance RMSE, also have the contrast situation of unmixing time.

[0077] Table 2-3 precision index comparison of autoencoder with different parameter configurations on the first group of data

[0078]

[0079]

[0080] It can be seen from the data of AE-1 to AE-3 in LMM_64x64_3_50dB_1 data in table 2-3 that the end member SAD index performance will be better by appropriately increasing the patch value; by comparing AE-4 and AE-2 data, the abundance RMSE index performance will be better by appropriately increasing the patch number, but increasing the values of both will also increase the calculation amount, and the unmixing time will be longer. Overall, the LMM_32x32_3_50dB_1 data uses AE-2 configuration to evaluate the index optimally, and the LMM_48x48_3_50dB_1 and LMM_64x64_3_50dB_1 data use AE-5 configuration to evaluate the index optimally, which shows that the autoencoder with large patch number and patch size configuration is suitable for unmixing large data set, otherwise for smaller data set, the values of the two should be appropriately reduced, otherwise it is easy to overtrain and waste time.

[0081] Table 2-4 precision index comparison of autoencoder with different parameter configurations on the second group of data

[0082]

[0083] Here from the overall view of Table 2-4 data, only in the case of different end members, each data set in AE-3 configuration, each index of unmixed performance best, it is proved that the patch size and the number of patches in the autoencoder are less relevant to the number of end members in the data set; By comparing the unmixed index of each data set, it is found that the performance is good, and the difference is small, which shows that the increase of the number of end members has low influence on the algorithm unmixed performance. Then from the overall view of Table 2-5 data, only in the case of different signal-to-noise ratio, each data set in AE-5 configuration, the unmixed index performance best, also shows that the patch size and the number of patches in the autoencoder are less relevant to the noise in the data; Then compare the unmixed index between each data set, with the increase of noise, the difference of each autoencoder's unmixed performance is low, and sometimes the unmixed performance is better at low signal-to-noise ratio, which shows that the autoencoder has significantly improved the data denoising ability after adding the convolutional neural network, and has good robustness.

[0084] Table 2-5 precision index comparison of different parameter configurations of autoencoder on the third group of data

[0085]

[0086] (2) Test analysis of convolutional autoencoder on real data

[0087] Here the patch size of the autoencoder is set to 15, and the number of patches is 150. Because there is no real end member and abundance in the real data set to compare with the results, this experiment can only qualitatively analyze the unmixed effect from the visual experience, here only show their abundance map.

[0088] (a) First, test the Samon data set, set the number of end members to 3, and the abundance map of each subgraph is the distribution of each end member. Because the data set comes from an island image data, it contains a part of sea water and island. Water is distributed in patches, so the second subgraph is the distribution of water, and the outline is clear; observing the first two subgraphs, there is an obvious boundary between the island and the sea; so the last subgraph corresponds to the end member "soil". The three subgraphs are consistent with the display of the original image, which shows that the unmixed effect is good, and the algorithm has good robustness.

[0089] (b) Then test the Urban data set, set the number of end members to 4, and show the distribution of 4 end members. This data set comes from a suburban image data, with houses, vegetation, roads, etc. First, compared with the real image, the subgraphs from left to right correspond to "road", "grass", "roof", "tree" respectively. The third subgraph is not obvious with the first division, and some places are repeated with the second subgraph. The analysis may be due to the small proportion of roads, which is difficult to distinguish. The roof and tree are dense and mixed to a high degree. Overcoming these problems will also be a direction for optimization in the future.

[0090] Example 1: Samson data set unmixing

[0091] User operation process:

[0092] Log in to the system → upload samson.img and samson.hdr

[0093] Click "unmixing" in the file list to enter the parameter page (automatic parsing of the number of bands = 156) Set parameters:

[0094] Number of end members = 3 (known ground objects: water / soil / vegetation)

[0095] Patch size = 15, patch number = 150

[0096] RGB rendering band: R = 600 nm, G = 550 nm, B = 450 nm

[0097] Click "online unmixing" to start the algorithm

[0098] Backend execution process:

[0099] Load 156 band data (95x95 pixels)

[0100] Randomly crop 150 15x15 image blocks

[0101] Train convolutional autoencoder (100 iterations, batch_size = 32)

[0102] Extract decoder weights as end member spectra (3x156 matrix)

[0103] Generate abundance map (3 95x95 heat maps)

[0104] Result display:

[0105] End member 1 spectral curve: strong absorption peak at 900 nm → match water characteristic abundance Figure 2 : Continuous large area region → corresponding water area distribution accuracy index: SAD_avg = 0.215 (no reference value, only show curve similarity) Example 2: Urban data set unmixing optimization

[0106] 1. Parameter tuning comparison:

[0107]

[0108] 2. Spatial feature extraction effect:

[0109] When the patch size = 15, the CONV1 layer feature map shows:

[0110] Filter 3: High response to roof edge

[0111] Filter 28: High response to vegetation texture

[0112] Prove that the convolution kernel effectively captures the spatial structure

[0113] The embodiment completely covers the whole chain of system construction → data processing → algorithm implementation → performance optimization → effect verification, and enhances the implementability through code segments, parameter tables, performance curves, etc. The above only describes the preferred embodiment of the present application, which is only used to illustrate the technical scheme of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application is included in the protection scope of the present application.

Claims

1. A hyperspectral remote sensing image blind unmixing method based on deep convolutional autoencoder, characterized in that, Upload the hyperspectral image file, adjust the endmember number, iteration number, and patch size online; automatically parse the data and perform unmixing to generate endmember spectral curve, abundance distribution graph, and precision index, support result visualization and PDF report export.

2. The method of claim 1, characterized in that, Specifically comprising the following steps: a) Upload the hyperspectral image file, i.e. simulation data.mat or real data.img / .hdr, through the Web interface; b) Parse the file information, including image size, band number, and endmember number; c) User online setting unmixing parameters: endmember number, iteration number, batch size, patch number, patch size, L2 penalty coefficient, learning rate; d) Perform unmixing based on convolutional autoencoder: Encoder: extract spatial features through convolutional layer, and the hidden layer outputs abundance map through Softmax; Decoder: linearly reconstruct the spectrum using weight matrix, and the weight is directly used as endmember matrix; e) For simulation data: automatically match the result endmember with the reference endmember, and calculate SAD / RMSE index f) Result visualization shows endmember spectral curve, abundance map, and loss curve, and supports PDF report export.

3. The method of claim 2, characterized in that: Set Space-spectrum joint unmixing model, convolutional layer extracts spatial features, autoencoder hidden layer outputs abundance map, and decoder weight directly generates endmember matrix; Dual-mode processing framework, respectively for simulation data and real data; Parameter adaptive matching algorithm: automatically align the result endmember with the reference endmember by minimizing SAD / RMSE.

4. The method of claim 3, characterized in that: The expression of the space-spectrum joint unmixing model is:

5. A system for implementing the method of claims 1-4, characterized in that, Including: User information management and image file upload module, supporting different user registration and login functions, and providing file upload function after login, different users have different file libraries to store their uploaded files, and the supported uploaded files are only simulation data "*.mat" and real hyperspectral image files, one is hyperspectral image header "*.hdr" and the other is hyperspectral image information file "*.img", the files between different users will not be shared; Hyperspectral data reading and image display module, according to the read file type, read the hyperspectral information file to obtain the band number, image size, and endmember number in simulation data; and display the image according to the information file, and the user can also set the R, G, B three color band values to render the image; The hyperspectral remote sensing image unmixing module integrates the convolutional autoencoder unmixing algorithm. In the module interface, the user adjusts some hyperparameters in the algorithm. For simulation test data, the real endmember and real abundance matrix in the simulation data are read after unmixing, the result endmember matrix and result abundance matrix are calculated, and each endmember matrix and abundance map are displayed on the front end. The spectral angle distance SAD and root mean square error RMSE of the endmember are calculated and displayed. For real data, the results are visualized after the algorithm is run. The data result export and analysis module exports the results in pdf format after the unmixing algorithm is completed, which is convenient for in-depth analysis of the results.

6. The system of claim 5, wherein The simulation data "*.mat" and the real hyperspectral image file are a hyperspectral image header "*.hdr" and another hyperspectral image information file "*.img". The files between different users are not shared.

7. The system of claim 5, wherein The patch size is dynamically recommended according to the data set size. For small data sets, the patch size is 5x5, and for large data sets, the patch size is 15x15. The optimal arrangement of the result abundance and the reference abundance is achieved by the Hungarian algorithm.

8. The system of claim 5, wherein The Web system is used: a) Front-end framework: Bootstrap + JavaScript; b) Back-end framework: Django + PyTorch; c) Database: Store user files and unmixing parameters (MySQL).

Images