Linear hyperspectral image unmixing method and system based on spatial-spectral joint features

Through the linear hyperspectral image demixing method based on the combination of null spectral features, the demix network modeling spectral variability and spatial characteristics are used to solve the problem of low demixing accuracy in the existing technology, and high-precision substance recognition and abundance are achieved degree estimate.

CN119006329APending Publication Date: 2024-11-22HUNAN SHENGDING TECH DEV CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411443671.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-16
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing linear hyperspectral image demixing methods ignore end element spectral variability caused by environmental factors and substances themselves and image spatial information, resulting in low demixing accuracy.

Method used

Using linear hyperspectral image demixing method based on the joint characteristics of the null spectrum, pixels are collectively clustered through spatial Euclidean distance and spectral similarity, superpixels are generated, and the encoder and decoder of the demix network are modeled. Spectral variability, extract end elements and abundance distributions, and reconstruct superpixels.

Benefits of technology

The accuracy of hyperspectral image demixing is improved, spatial and spectral features are fully utilized, image processing complexity is reduced, and sub-cell-level fine recognition of matter is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006329A_ABST
    Figure CN119006329A_ABST
Patent Text Reader

Abstract

The invention discloses a linear hyperspectral image unmixing method and system based on spatial-spectral joint features. The method comprises the following steps: performing joint clustering on pixels by using a spatial Euclidean distance and spectral similarity to obtain segmented superpixels; taking the superpixel as a training sample and a test sample of a de-mixing network, and training an encoder and a decoder of the de-mixing network through the training sample; training and optimizing the unmixing network by reconstructing a training sample, and testing the unmixing performance of the unmixing network; and unmixing the to-be-unmixed hyperspectral image to obtain an unmixing result of the to-be-unmixed hyperspectral image. In the technical scheme of the invention, space and spectral features are fully utilized, so that the superpixel segmentation can provide convenient calculation elements to calculate image features, and the complexity of subsequent image processing tasks is greatly reduced; the basic framework of the unmixing network is a variational automatic encoder, spectrum variability is modeled, and the unmixing precision is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image processing, and in particular to a linear hyperspectral image unmixing method and system based on spatial-spectral joint features. Background Art

[0002] Hyperspectral images usually contain hundreds of continuous spectral bands, which have the advantage of high spectral resolution, but often at the expense of spatial resolution, resulting in low spatial resolution of hyperspectral images. The limited spatial resolution means that a single image pixel may contain multiple substances in the scene. The mixed pixel in the hyperspectral image is a pixel formed by the superposition of the spectral response curves of multiple different substances, which only uses one spectral curve to record the combined effect of the spectra of multiple substances. Due to the low spatial resolution of hyperspectral images and the complex distribution and variety of substances in nature, most pixels exist in the image in the form of mixed pixels. The unmixing of hyperspectral images, that is, the mixed pixel decomposition technology, aims to obtain the spectral characteristics of various pure substances contained in each pixel at the sub-pixel scale, that is, the end members; and their distribution ratio in the corresponding pixel, that is, the abundance. Therefore, the unmixing of hyperspectral images can realize the fine identification of substances at the sub-pixel level, which is of great significance for quantitative research using hyperspectral images. However, the current linear spectral unmixing method has the following shortcomings: (1) The endmember spectral variability caused by environmental factors (light, atmosphere, topography, etc.) and the material itself is often ignored during the unmixing process; (2) In the process of pixel decomposition, the utilization of image spatial information is often neglected.

[0003] Therefore, in view of the limitations of the existing algorithms, the present invention proposes a hyperspectral image unmixing algorithm that jointly uses the spatial spectral characteristics of hyperspectral images and considers the spectral variability of endmembers to improve the overall pixel unmixing accuracy. Summary of the invention

[0004] The main purpose of the present invention is to provide a linear hyperspectral image unmixing method based on spatial-spectral joint features, aiming to solve the problems of insufficient utilization of end-member spectral variability and image spatial information caused by environmental factors (light, atmosphere, terrain, etc.) and the material itself, and low pixel unmixing accuracy in existing linear spectral unmixing methods.

[0005] To achieve the above object, the present invention proposes a linear hyperspectral image unmixing method based on spatial-spectral joint features, which comprises the following steps: The original hyperspectral image is clustered together using spatial Euclidean distance and spectral similarity to obtain segmented superpixels. Using the superpixels as training samples and test samples of an unmixing network, and training an encoder and a decoder of the unmixing network by using the training samples to model spectral variability; The encoder encodes the input superpixel into an abundance distribution and a hidden layer expression of an end member, the hidden layer expression of the end member generates an estimated end member through the decoder, and the superpixel is reconstructed through the estimated end member and the abundance distribution; Using the test sample to test the unmixing performance of the unmixing network; The unmixing network whose unmixing performance reaches a preset value is used to process the hyperspectral image to be unmixed, so as to obtain an unmixing result of the hyperspectral image to be unmixed.

[0006] Preferably, the step of jointly clustering pixels of the original hyperspectral image using spatial Euclidean distance and spectral similarity to obtain segmented superpixels comprises: Initialize the image segmentation block: first generate k uniformly distributed seed points on the image; for each pixel, search for the nearest seed point around the pixel, and classify the pixel and the seed point into one category until all the pixels are classified, and finally obtain k superpixels; Update seed points: calculate the average vector value of all pixels in the k superpixels, obtain the new K cluster centers, and then search for the most similar pixels around the K cluster centers until all pixels are reclassified into K superpixels and the cluster centers are updated; Iterations are repeated until convergence or the preset maximum number of iterations is reached.

[0007] Preferably, the clustering features used in the step of jointly clustering pixels of the original hyperspectral image using spatial Euclidean distance and spectral similarity to obtain segmented superpixels include: ; in, The table indicates and The square of the Euclidean distance between two pixels, and Respectively represent and The spectrum of a pixel, Indicates the search range of the SLIC algorithm, Represents a hyperparameter used to balance the pixel space distance and spectral similarity image.

[0008] Preferably, the step of encoding the input superpixel into a hidden layer expression of abundance distribution and endmembers by the encoder comprises: Use self-attention mechanism to calculate spectral attention map and spatial attention map of superpixel sequence; Generate a joint feature of space and spectrum through the spectral attention map, the spatial attention map and the two-stream attention mechanism; Hidden layer representations of abundance distribution and endmembers are generated through a feed-forward neural network and joint features of spatial spectra.

[0009] Preferably, the encoder encodes the input superpixel into an abundance distribution and a hidden layer expression of an endmember, the hidden layer expression of the endmember generates an estimated endmember through the decoder, and the step of reconstructing the superpixel through the estimated endmember and the abundance distribution includes: The encoder encodes the input superpixel into a hidden layer expression of abundance distribution and end members, which is expressed as follows:

[0010] in, represents all trainable parameters of the encoder, an abundance map of superpixel patches representing the estimated current position; represents the hidden layer expression of the end member, The dimension is ; Hidden layer variables The estimated end-member is generated by the decoder and is expressed as follows:

[0011] in, represents all trainable parameters of the decoder, is the estimated endmember matrix of the current position; Finally, the superpixel can be estimated by the endmember matrix and abundance distribution Reconstruction is performed as follows: .

[0012] A linear hyperspectral image unmixing system based on spatial-spectral joint features includes: a superpixel segmentation unit, an encoder, a decoder and a reconstruction unit, wherein the superpixel segmentation unit is used to jointly cluster pixels according to spatial Euclidean distance and spectral similarity to obtain segmented superpixels, the encoder includes an endmember hidden subunit and an abundance estimation subunit, the endmember hidden subunit is used to encode the input superpixel into a hidden layer expression of the endmember, the abundance estimation subunit is used to encode the input superpixel into an abundance distribution, the decoder is used to generate an estimated endmember through the hidden layer expression of the endmember, and the reconstruction unit is used to reconstruct superpixels through the estimated endmember and the abundance distribution.

[0013] Preferably, the end member hidden subunit includes a spatial-spectral dual-stream Transformer, an average pooling layer, a first fully connected layer and a ReLU layer connected in sequence, and a standard deviation connection layer and a mean connection layer both connected to the ReLU layer, and the spatial-spectral dual-stream Transformer is used to simultaneously perform global spatial-spectral feature joint extraction.

[0014] Preferably, the abundance estimation subunit includes three spatial-spectral dual-stream Transformers connected in sequence, a second fully connected layer and a The spatial-spectral dual-stream Transformer is used to simultaneously extract global spatial-spectral features.

[0015] Preferably, the spatial-spectral dual-stream Transformer comprises a first-layer normalization module, a dual-stream attention module, a second-layer normalization module and a feedforward neural network module connected in sequence; The dual-stream attention module includes a multi-head spatial self-attention sub-module that models long-range feature dependencies of spatial dimensions, and a multi-head spectral self-attention sub-module that models long-range feature dependencies of spectral dimensions.

[0016] Preferably, the feedforward neural network module comprises a first linear layer, a ReLU activation function and a second linear layer connected in sequence.

[0017] In the technical solution of the present invention, cluster segmentation is performed on hyperspectral images to make full use of spatial and spectral features, so that superpixel segmentation can provide convenient computing primitives to calculate image features and greatly reduce the complexity of subsequent image processing tasks; the basic framework of the unmixing network is a variational autoencoder, which can simultaneously perform endmember extraction and abundance estimation in an unsupervised manner, and model spectral variability to improve the unmixing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0019] Figure 1 The module flow diagram of the linear hyperspectral image unmixing method based on spatial-spectral joint features of the present invention is shown in FIG.

[0020] Figure 2 It is a flow chart of the spatial-spectral dual-stream Transformer in the linear hyperspectral image unmixing method based on spatial-spectral joint features of the present invention.

[0021] Figure 3 This is a pseudo-color image of the Samson data set in the linear hyperspectral image unmixing method based on spatial-spectral joint features of the present invention.

[0022] Figure 4 This is the result of clustering and segmentation of the Samson dataset in the linear hyperspectral image unmixing method based on spatial-spectral joint features of the present invention (247 superpixels).

[0023] Figure 5 Quantitative results of the unmixing accuracy of different unmixing methods on the Samson dataset.

[0024] Figure 6 Abundance estimation diagram for different unmixing methods on the Samson dataset.

[0025] Figure 7 Endmember extraction diagram of different unmixing methods on the Samson dataset.

[0026] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0029] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0030] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0031] Please refer to Figure 1-Figure 7 The present invention proposes a linear hyperspectral image unmixing method based on spatial-spectral joint features, comprising the following steps: S100, the original hyperspectral image is jointly clustered using spatial Euclidean distance and spectral similarity to obtain segmented superpixels; S200, using the superpixels as training samples and test samples of an unmixing network, training an encoder and a decoder of the unmixing network through the training samples, and modeling spectral variability; S300, the encoder encodes the input superpixel into an abundance distribution and a hidden layer expression of an endmember, the hidden layer expression of the endmember generates an estimated endmember through the decoder, and the superpixel is reconstructed through the estimated endmember and the abundance distribution; S400, using the test sample to test the demixing performance of the demixing network; S500, using the unmixing network whose unmixing performance reaches a preset value to process the hyperspectral image to be unmixed, to obtain an unmixing result of the hyperspectral image to be unmixed.

[0032] In the technical solution of the present invention, cluster segmentation is performed on hyperspectral images to make full use of spatial and spectral features, so that superpixel segmentation can provide convenient computing primitives to calculate image features and greatly reduce the complexity of subsequent image processing tasks; the basic framework of the unmixing network is a variational autoencoder (VAE), which can simultaneously perform endmember extraction and abundance estimation in an unsupervised manner, and model spectral variability to improve the unmixing accuracy.

[0033] Specifically, the patent of the present invention is applied to the field of hyperspectral image unmixing, and proposes a linear hyperspectral image unmixing method that jointly uses spatial spectral features and considers the spectral variability of end members in the unmixing process. The linear mixture model (Linear Mixture Model, LMM) assumes that various pure substances are evenly distributed in the field of view at the macro level, and that the incident light is received by the sensor after interacting with only one pure substance each time, without considering the scattering effect of light between different substances. The linear mixture model believes that the observed pixel spectrum is the linear weighting of the end members according to a certain proportion (abundance). The spectral variability is due to the complex environmental material composition in the real world, the changing atmospheric observation conditions and lighting conditions, and the observation time. The end member spectra of the same substance in different pixels will also change, which is often called the "same object, different spectrum" phenomenon.

[0034] First, the proposed unmixing method is implemented on hyperspectral images segmented into superpixels, considering that the spectral similarity of materials in neighboring regions is higher. Under the premise of considering spectral variability, the spectral decomposition is performed on each superpixel separately based on the assumption that endmembers are shared within superpixels, and the estimated abundance map and endmember set on this superpixel are obtained. In the superpixel unmixing process, the variational autoencoder (VAE) is used as the main framework of the unmixing network to model spectral variability, and a spatial-spectral two-stream Transformer module (TST) is proposed in the network to jointly use global features in spatial and spectral dimensions.

[0035] The goal of the superpixel segmentation method (Simple Linear Iterative Clustering, SLIC) is to group all pixels into multiple similar and compact regions to form continuous image blocks with semantic information. Superpixel segmentation can provide convenient primitives to calculate image features and greatly reduce the complexity of subsequent image processing tasks. The SLIC algorithm is based on the K-means clustering algorithm and has made two important improvements on it: First, the complexity of the algorithm is reduced by limiting the search space, and the number of pixels is obtained. Secondly, the spectral and spatial similarities are simultaneously incorporated into the weighted distance measurement to control the size and compactness of superpixels.

[0036] In another embodiment of the present invention, the step S100 includes: S110, initializing the image segmentation block: first, generating k uniformly distributed seed points on the image; for each pixel, searching for the nearest seed point around the pixel, and classifying the pixel and the seed point into one category, until all the pixels are classified, and finally obtaining k superpixels; S120, updating the seed point: calculating the average vector value of all the pixels in the k superpixels, obtaining the new K cluster centers, and then searching for the most similar pixels around the K cluster centers until all the pixels are reclassified into the K superpixels, and the cluster centers are updated; S130, repeating iterations until convergence or reaching a preset maximum number of iterations.

[0037] Specifically, (1) Initialize the image segmentation block: First, generate seed points (cluster centers), the The seed points are evenly distributed on the image. Assume that the original image has pixels, then each superpixel obtained after segmentation has approximately pixels; (2) Clustering: For each pixel, Search for the nearest seed point in the surrounding area and classify the pixel and the seed point into one category until all pixels are classified and finally obtain superpixels; (3) Update the seed point: The superpixels calculate the average vector value of all pixels and obtain the new cluster centers, and then The most similar pixels are searched around the cluster centers until all pixels are reclassified as superpixels and update the cluster centers; (4) Repeated iteration: Repeat steps 2 and 3 until convergence or the maximum number of iterations is reached.

[0038] In another embodiment of the present invention, the size of the original hyperspectral image is ,in and Respectively represent the length and width of the hyperspectral space size, Represents the spectral dimension of the hyperspectral image.

[0039] First, the original hyperspectral image is clustered and segmented using the spatial Euclidean distance and spectral similarity through the SLIC algorithm to obtain superpixels. , For the superpixels, is the number of pixels contained in the superpixel, that is, the sequence length. The total number of pixels in the hyperspectral image is , the number of superpixels generated by cluster segmentation is .

[0040] The superpixels generated after clustering and segmentation are used as training and testing samples of the unmixing network. The designed unmixing network is trained and optimized by reconstructing the training samples, and the unmixing performance of the network is tested using the test samples.

[0041] In another embodiment of the present invention, the clustering features used in step S100 include: ; in, Indicates and The square of the Euclidean distance between two pixels, and Respectively represent and The spectrum of a pixel, Indicates the search range of the SLIC algorithm, Represents a hyperparameter used to balance the pixel space distance and spectral similarity image.

[0042] Specifically, despite the endmember variability in the unmixing process, it is reasonable to assume that the endmembers are shared in local regions because the spectral estimates in spatially compact regions will show similar characteristics. Therefore, the hyperspectral image can be segmented into local sub-images for separate processing.

[0043] In another embodiment of the present invention, the step of encoding the input superpixel into a hidden layer expression of abundance distribution and endmembers in S300 comprises: S310, using a self-attention mechanism to calculate a spectral attention map and a spatial attention map of the superpixel sequence; S320, generating a joint feature of the spatial spectrum through the spectral attention map, the spatial attention map and the dual-stream attention mechanism; S330, generates hidden layer expressions of abundance distribution and end members through the joint features of feedforward neural network and spatial spectrum.

[0044] Specifically, we use the input to generate Query (Q), Key (K) and Value (V) to calculate the output of a self-attention layer. First, assume that the network input is ,in is the channel dimension, is the sequence length. is mapped to , and to encode contextual information across spectral channels.

[0045] ; ; ; in , It is used to calculate , The model parameters.

[0046] Then, a self-attention mechanism is used to compute the spectral attention map of the superpixel sequence.

[0047] By using ) and Key ( ) to generate the spectral attention map .

[0048] ; in is the dimension of the attention matrix, The role of is to normalize the attention score. Finally, the output of the spectral self-attention layer is the attention matrix and Value ( ) is the product of .

[0049]

[0050] Multi-Head Spatial Self-Attention (MspaA): Aims to apply self-attention to global spatial locations and generate a spatial attention map to model long-range dependencies and spatial interactions. On the branch of MSpaA, is mapped to , and .

[0051] Then, a self-attention mechanism is used to compute the spatial attention map of the superpixel sequence.

[0052] By using ) and Key ( ) to generate a spatial attention map .

[0053] ; Finally, the output of the spatial self-attention layer is the attention matrix and Value ( ) is the product of .

[0054] ; Finally, the joint feature of the spatial spectrum generated using the two-stream attention mechanism is .

[0055] .

[0056] A spatial-spectral two-stream Transformer (TST) module is proposed within the unmixing network, which can simultaneously utilize the global information of the joint spatial and spectral dimensions to improve the accuracy of unmixing.

[0057] In another embodiment of the present invention, the step of encoding the input superpixel into a hidden layer expression of abundance distribution and endmembers in S300 comprises: The encoder encodes the input superpixel into an abundance distribution and a hidden layer expression of an end member, the hidden layer expression of the end member generates an estimated end member through the decoder, and the step of reconstructing the superpixel through the estimated end member and the abundance distribution includes: S360, the encoder encodes the input superpixel into a hidden layer expression of abundance distribution and endmembers, which is expressed as the following formula:

[0058] in, represents all trainable parameters of the encoder, an abundance map of superpixel patches representing the estimated current position; represents the hidden layer expression of the end member, The dimension is ; S370, hidden layer variables The estimated end-member is generated by the decoder and is expressed as follows:

[0059] in, represents all trainable parameters of the decoder, is the estimated endmember matrix of the current position; S380, the superpixel can be estimated by the end member matrix and abundance distribution Reconstruction is performed as follows: .

[0060] A linear hyperspectral image unmixing system based on spatial-spectral joint features includes: a superpixel segmentation unit, an encoder, a decoder and a reconstruction unit, wherein the superpixel segmentation unit is used to jointly cluster pixels according to spatial Euclidean distance and spectral similarity to obtain segmented superpixels, the encoder includes an endmember hidden subunit and an abundance estimation subunit, the endmember hidden subunit is used to encode the input superpixel into a hidden layer expression of the endmember, the abundance estimation subunit is used to encode the input superpixel into an abundance distribution, the decoder is used to generate an estimated endmember through the hidden layer expression of the endmember, and the reconstruction unit is used to reconstruct superpixels through the estimated endmember and the abundance distribution.

[0061] Specifically, it consists of two parts: clustering segmentation of hyperspectral images and unmixing network. Clustering segmentation of hyperspectral images is implemented using the SLIC algorithm, incorporating spatial distance and spectral similarity into weighted distance measurement to control the size and compactness of the obtained superpixels. The unmixing network is based on the basic framework of variational autoencoder (VAE), which simultaneously performs endmember extraction and abundance estimation, and models the spectral variability of endmembers. At the same time, a spatial-spectral dual-stream Transformer (TST) module is designed in the network, so that the global features of spatial and spectral dimensions can be jointly used in the unmixing process.

[0062] The workflow of the entire algorithm is briefly described as follows: (1) First, the hyperspectral image is clustered and segmented using the SLIC algorithm to segment the hyperspectral image into superpixels. In the specific implementation, the pixels are jointly clustered using spatial Euclidean distance and spectral similarity; (2) The obtained superpixels are the training and test samples of the unmixing network. The unmixing network is trained by reconstructing superpixels, and these superpixels are used to test the unmixing performance of the network.

[0063] The main framework of the unmixing network is the variational autoencoder (VAE), which consists of two parts: encoder and decoder. The proposed unmixing network has two branches designed in the encoder part, one to predict the hidden layer expression of the end member, and the other to estimate the abundance distribution of the input superpixel. The hidden layer expression of the end member is a data distribution composed of the mean and standard deviation. The spectral variability of the end member is modeled by predicting the distribution of the end member. The overall block diagram of the proposed unmixing network is shown in the figure below. Figure 1 shown.

[0064] First, the encoder converts the input superpixels Encoded as abundance distribution and potential representation of end members , expressed as the following formula:

[0065] in, represents all trainable parameters of the encoder, and They are the estimated abundance map of the superpixel block at the current position and the hidden layer expression of the end member, The dimension is . The abundance estimation branch was used The layer is used as the output layer to ensure the non-negativity of the abundance and the “sum is 1” property.

[0066] Then, the hidden layer variables of the above reasoning The estimated end-units are generated by the decoder and at the end of the decoder Function to ensure the non-negativity of the end members and limit the generated end members to The process is expressed as follows:

[0067] in, represents all trainable parameters of the decoder, is the estimated endmember matrix of the current position.

[0068] Finally, the superpixel block can be estimated by the endmember matrix and abundance distribution Reconstruction is performed as follows:

[0069] In another embodiment of the present invention, the end member hidden subunit includes a spatial-spectral dual-stream Transformer, an average pooling layer, a first fully connected layer and a ReLU layer connected in sequence, and a standard deviation connection layer and a mean connection layer both connected to the ReLU layer, and the spatial-spectral dual-stream Transformer is used to simultaneously perform global spatial-spectral feature joint extraction.

[0070] Specifically, the spatial-spectral two-stream Transformer (TST) module can simultaneously extract global spatial-spectral features. The TST module adopts the typical architecture of the Transformer, which consists of a two-stream attention part (TSA) and a feed-forward module MLP. Its flow chart is as follows: Figure 2 shown.

[0071] In another embodiment of the present invention, the abundance estimation subunit includes three spatial-spectral dual-stream Transformers connected in sequence, a second fully connected layer and an output layer. The spatial-spectral dual-stream Transformer is used to simultaneously extract global spatial-spectral features.

[0072] Specifically, the Two-Stream Attention (TSA) module consists of Multi-Head Spatial Self-Attention (MspA) and Multi-Head Spectral Self-Attention (MspA), which model the long-range feature dependencies in the spatial and spectral dimensions respectively.

[0073] In another embodiment of the present invention, comprising: The spatial-spectral two-stream Transformer includes a first-layer normalization module, a two-stream attention module, a second-layer normalization module, and a feedforward neural network module connected in sequence; The dual-stream attention module includes a multi-head spatial self-attention sub-module that models long-range feature dependencies of spatial dimensions, and a multi-head spectral self-attention sub-module that models long-range feature dependencies of spectral dimensions.

[0074] In another embodiment of the present invention, the feedforward neural network module includes a first linear layer, a ReLU activation function and a second linear layer connected in sequence.

[0075] Feedforward neural network MLP: Feedforword neural network is crucial to the good performance of Transformer. Its role in Transformer is to perform nonlinear transformation on the vector of each position, thereby introducing more nonlinear capabilities and improving the expressiveness of the model. Specifically, the feedforward neural network consists of two linear layers and an activation function.

[0076] The first linear layer will expand the dimension of the tensor, and the second linear layer will reduce the dimension of the expanded vector, so that the input vector can be mapped to a higher dimensional space and different feature information can be fused through nonlinear transformation.

[0077] In the proposed TST module, the most common MLP network is used in the feedforward neural network part. The linear layer of the MLP network is a fully connected layer, and the activation function used is ReLU. The purpose of layer normalization (LN) in the feedforward network is to normalize the output of the network so that the output has the same mean and variance, normalize its distribution, which is beneficial to the stability of the network and enhances the learning ability of the network. In the training process of deep neural networks, as the network deepens, the input distribution may change, making the network training process difficult.

[0078] The normalization operation is performed through the LN layer, making the output of the middle layer more stable in terms of value, which helps to accelerate network convergence and improve the network training effect and the generalization ability of the model.

[0079] In one embodiment, see Figure 3 , taking the Samson dataset as an example: The Samson dataset is a dataset containing The hyperspectral image has 156 pixels. The spectral channels of the pixels are 156, covering a spectral wavelength range of 401 to 889 nm, and the spectral resolution reaches 3.13 nm. In the experiment, the dataset is set to contain three types of substances, namely "1. Soil", "2. Trees" and "3. Water".

[0080] Please refer to Figure 4 , in the superpixel segmentation step, ,in Represents the range covered by each superpixel at initialization. N is the total number of pixels in the hyperspectral image, and K is the total number of initial superpixels set. For the Samson dataset, S=6, K=250, and m=10, and finally 247 superpixels are generated.

[0081] Comparison of different unmixing methods: In order to evaluate the performance of the proposed algorithm in hyperspectral unmixing, the unmixing method STTHU proposed in the present invention is compared with several other unmixing methods. These unmixing methods are Cycunet, PGMSU, DGMSUCNN, DGMSUGCN, DGMSUSAN and DeepTrans. Among them, Cycunet and DeepTrans algorithms are unmixing methods based on linear spectral mixing and do not consider the influence of spectral variability; PGMSU, DGMSUCNN, DGMSUGCN and DGMSUSAN methods are unmixing algorithms that consider spectral variability in a linear framework.

[0082] The input of the PGMSU method is pixel by pixel, and the other three algorithms are the same as the proposed method STTHU, which is a hyperspectral image unmixing method based on superpixel segmentation.

[0083] Three accuracy indicators are used to quantitatively evaluate the accuracy of the unmixing algorithm, namely, Abundance Root Mean Square Error (ARMSE), Reconstruction Root Mean Square Error (RRMSE), and Reconstruction Spectral Angle Distance (RSAD). The smaller the values ​​of these three indicators, the higher the unmixing accuracy.

[0084] Figure 6 and Figure 7 They are respectively the visual result diagrams of abundance estimation and endmember estimation. GT in the abundance estimation diagram stands for GroundTruth, which is the reference diagram for the true abundance estimation. In all abundance diagrams, the redder the color, the higher the material content, and vice versa, the bluer the color, the lower the material content. The red line in each small box in the endmember extraction diagram is the reference endmember spectral reflectance curve provided, and the blue curve is the estimated endmember reflectance curve. The horizontal axis of the box diagram is the number of spectral channels (Bands), and the vertical axis is the spectral reflectance value (Reflectance). Figure 7 As shown in the results, the STTHU method proposed in the present application models the endmember spectral variability well and achieves relatively accurate endmember and abundance estimation.

[0085] comprehensive Figure 5 The quantitative data results in the table and Figure 6 and Figure 7 From the qualitative visual effect, the proposed unmixing method STTHU performs very competitively, which also verifies its good generalization. The experimental results also prove the necessity of considering spectral variability in the unmixing process of hyperspectral images, and the superiority of jointly using global features in spatial and spectral domains in unmixing.

[0086] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings, or directly / indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A linear hyperspectral image unmixing method based on spatial-spectral joint features, It is characterized in that The steps include: The original hyperspectral image is clustered together using spatial Euclidean distance and spectral similarity to obtain segmented superpixels. Using the superpixels as training samples and test samples of an unmixing network, and training an encoder and a decoder of the unmixing network by using the training samples to model spectral variability; The encoder encodes the input superpixel into an abundance distribution and a hidden layer expression of an end member, the hidden layer expression of the end member generates an estimated end member through the decoder, and the superpixel is reconstructed through the estimated end member and the abundance distribution; Using the test sample to test the unmixing performance of the unmixing network; The unmixing network whose unmixing performance reaches a preset value is used to process the hyperspectral image to be unmixed, so as to obtain an unmixing result of the hyperspectral image to be unmixed.

2. The linear hyperspectral image unmixing method based on spatial-spectral joint features as claimed in claim 1, It is characterized in that The step of jointly clustering pixels of the original hyperspectral image using spatial Euclidean distance and spectral similarity to obtain segmented superpixels includes: Initialize the image segmentation block: first generate k uniformly distributed seed points on the image; for each pixel, search for the nearest seed point around the pixel, and classify the pixel and the seed point into one category until all the pixels are classified, and finally obtain k superpixels; Update seed points: calculate the average vector value of all pixels in the k superpixels, obtain the new K cluster centers, and then search for the most similar pixels around the K cluster centers until all pixels are reclassified into K superpixels and the cluster centers are updated; Iterations are repeated until convergence or the preset maximum number of iterations is reached.

3. The linear hyperspectral image unmixing method based on spatial-spectral joint features as claimed in claim 1, It is characterized in that The clustering features used in the step of jointly clustering pixels of the original hyperspectral image using spatial Euclidean distance and spectral similarity to obtain segmented superpixels include: ; in, Indicates and The square of the Euclidean distance between pixels, and Respectively represent and The spectrum of a pixel, Indicates the search range of the SLIC algorithm, Represents the hyperparameter used to balance the pixel space distance and spectral similarity imagery.

4. The linear hyperspectral image unmixing method based on spatial-spectral joint features as claimed in claim 1, It is characterized in that The step of encoding the input superpixel into a hidden layer expression of abundance distribution and end members by the encoder comprises: Use self-attention mechanism to calculate spectral attention map and spatial attention map of superpixel sequence; Generate a joint feature of space and spectrum through the spectral attention map, the spatial attention map and the two-stream attention mechanism; Hidden layer representations of abundance distribution and endmembers are generated through a feed-forward neural network and joint features of spatial spectra.

5. The linear hyperspectral image unmixing method based on spatial-spectral joint features as described in any one of claims 1 to 4, It is characterized in that The encoder encodes the input superpixel into an abundance distribution and a hidden layer expression of an end member, the hidden layer expression of the end member generates an estimated end member through the decoder, and the step of reconstructing the superpixel through the estimated end member and the abundance distribution includes: The encoder encodes the input superpixel into a hidden layer expression of abundance distribution and end members, which is expressed as follows: ; in, represents all trainable parameters of the encoder, an abundance map of superpixel patches representing the estimated current position; represents the hidden layer expression of the end member, The dimension is ; Hidden layer variables The estimated end-member is generated by the decoder and is expressed as follows: ; in, represents all trainable parameters of the decoder, is the estimated endmember matrix of the current position; Finally, the superpixel can be estimated by the endmember matrix and abundance distribution Reconstruction is performed as follows: 。 6. A linear hyperspectral image unmixing system based on spatial-spectral joint features, It is characterized in that include: A superpixel segmentation unit, an encoder, a decoder and a reconstruction unit, wherein the superpixel segmentation unit is used to jointly cluster pixels according to spatial Euclidean distance and spectral similarity to obtain segmented superpixels, the encoder includes an endmember hidden subunit and an abundance estimation subunit, the endmember hidden subunit is used to encode the input superpixel into a hidden layer expression of the endmember, the abundance estimation subunit is used to encode the input superpixel into an abundance distribution, the decoder is used to generate an estimated endmember through the hidden layer expression of the endmember, and the reconstruction unit is used to reconstruct superpixels through the estimated endmember and the abundance distribution.

7. The linear hyperspectral image unmixing system based on spatial-spectral joint features as claimed in claim 6, It is characterized in that The endmember hidden subunit includes a spatial-spectral dual-stream Transformer, an average pooling layer, a first fully connected layer and a ReLU layer connected in sequence, and a standard deviation connection layer and a mean connection layer both connected to the ReLU layer. The spatial-spectral dual-stream Transformer is used to simultaneously perform global spatial-spectral feature joint extraction.

8. The linear hyperspectral image unmixing system based on spatial-spectral joint features as claimed in claim 6, It is characterized in that The abundance estimation subunit includes three spatial-spectral dual-stream Transformers connected in sequence, a second fully connected layer and an output layer. The spatial-spectral dual-stream Transformer is used to simultaneously extract global spatial-spectral features.

9. The linear hyperspectral image unmixing system based on spatial-spectral joint features as claimed in any one of claims 6 to 8, It is characterized in that include: The spatial-spectral two-stream Transformer includes a first-layer normalization module, a two-stream attention module, a second-layer normalization module, and a feedforward neural network module connected in sequence; The dual-stream attention module includes a multi-head spatial self-attention sub-module that models long-range feature dependencies of spatial dimensions, and a multi-head spectral self-attention sub-module that models long-range feature dependencies of spectral dimensions.

10. The linear hyperspectral image unmixing system based on spatial-spectral joint features as claimed in claim 9, It is characterized in that The feedforward neural network module includes a first linear layer, a ReLU activation function and a second linear layer connected in sequence.

Citation Information

Cited By

  • Hyperspectral blind unmixing method

    CN119810660A

  • Resonance metasurface design method and device based on spectrum sensing improvement

    CN120030900A