Blind hyperspectral unmixing method and system based on endmember query design and transformer attention mechanism

Abstract

The blind hyperspectral unmixing method and system based on end member query design and transformer attention mechanism relate to the field of hyperspectral image processing. The problems of current simplified mixing mechanism mainly being linear mixing model, low accuracy of feature abundance and corresponding end member, and deep learning blind hyperspectral unmixing method depending on good end member initialization and end member initialization seriously reducing unmixing performance are solved. The method comprises the following steps: constructing an abundance mapping module, extracting features in the hyperspectral image, and decoding the features to form an estimated abundance map; mapping the abundance map into abundance embedding; constructing a mixing mechanism fitting transformer module, combining the end member query, the reconstruction query and the abundance embedding through the mixing mechanism fitting transformer module to obtain the reconstruction query of the end member information and the abundance information; and remodeling the original hyperspectral image according to the reconstruction query. The method is applied to the field of blind hyperspectral image unmixing.

Description

Technical Field

[0001] The present invention relates to the field of hyperspectral image processing, and in particular to a blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism. Background Art

[0002] Blind hyperspectral unmixing methods based on deep learning have become a major topic of research and have made significant progress. However, many existing research methods or experiments are mainly based on simplified mixing mechanism assumptions. By simplifying the mixing mechanism in the hyperspectral imaging process, the decoding structure of the autoencoder model is constructed, and the input spectrum is reconstructed to achieve simultaneous estimation of abundance and endmembers. The current simplified mixing mechanism is mainly a linear mixing model. It is generally believed that ground objects are linearly mixed. The deep learning model constructed on this basis lacks the nonlinear mixing characteristics of ground objects, and the estimated ground object abundance and corresponding endmembers are less accurate. In addition, blind hyperspectral unmixing methods based on deep learning generally rely on good endmember initialization, such as vertex component analysis. Without good endmember initialization, the unmixing performance degrades significantly, which greatly limits the practical application of current deep learning methods. Summary of the Invention

[0003] In view of the current simplified mixing mechanism mainly being a linear mixing model, the low accuracy of feature abundance and corresponding endmembers, and the fact that deep learning blind hyperspectral unmixing methods generally rely on good endmember initialization, the unmixing performance degrades severely when the endmembers are initialized. The present invention proposes a blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism, which includes:

[0004] Construct an abundance mapping module, extract features from hyperspectral images through the convolutional neural network in the abundance mapping module, and decode the features into an estimated abundance map;

[0005] Mapping abundance maps to abundance embeddings;

[0006] Construct a hybrid mechanism fitting Transformer module, which combines endmember query, reconstruction query and abundance embedding to obtain reconstruction query of endmember information and abundance information;

[0007] Reshape the original hyperspectral image according to the reconstruction query.

[0008] Further, it is proposed that the abundance mapping module is composed of a U-shaped structure, extracts features through a convolution layer, and uses a Swish activation function for nonlinear transformation; the input hyperspectral image is mapped to an abundance map through twice down-sampling and up-sampling operations after the convolution layer, and the output features of the encoder are optimized to the specified abundance space by using the abundance non-negative constraint ASC and the abundance sum-to-one ANC constraint:

[0009]

[0010] wherein X (i,h,w) represents the i-th feature to be activated by Softmax at the spatial position (h, w), e represents the natural base, and ω represents the scale factor.

[0011] Further, it is proposed that the abundance is mapped to an abundance embedding with the same size as the input hyperspectral image through a convolutional neural network, and the abundance embedding with the same size is obtained by any one of preserving the original spatial size or using a down-sampling operation.

[0012] Further, it is proposed that the mixed mechanism fitting Transformer module includes:

[0013] Q represents the Query vector in the Transformer attention mechanism, K represents the Key vector in the Transformer attention mechanism, V represents the Value vector in the Transformer attention mechanism, and the output Z is represented as:

[0014]

[0015] wherein, for scale control;

[0016] The attention mechanism is optimized by using a multi-head attention mechanism for the reconstruction query.

[0017] The Transformer module decoder is represented as:

[0018]

[0019] wherein Z (l,1) represents the self-attention mechanism of the l-th decoder, and Z (l,2) represents the cross-attention mechanism of the l-th decoder.

[0020] The entire Transformer module uses a cascaded decoder structure to gradually optimize the reconstruction query.

[0021] Furthermore, a preferred method is proposed, in which the structure of the cascade decoder is: taking the optimized reconstruction query, endmember query and abundance embedding of the previous layer as input to share the fully connected layer mapping and reshape it into a hyperspectral image, and gradually optimizing the generated reconstructed image from coarse to fine.

[0022] Furthermore, a preferred method is proposed, wherein the method further comprises constructing a loss function including three perspectives: reconstruction, abundance, and endmembers:

[0023]

[0024] L final =L cas +L aux ,

[0025] Among them, L cas represents the cascade loss, Indicates that the focus of the entire model is the final result, represents the reconstructed image corresponding to the lth decoder, I represents the input hyperspectral image, represents the estimated abundance matrix, represents the estimated end-member matrix, N represents the number of cascaded decoders, represents the auxiliary output hyperspectral image under the linear mixed model, represents the transpose of the estimated endmember matrix, L aux represents additional linear mixing auxiliary constraints.

[0026] Furthermore, a preferred method is proposed, wherein the end member is obtained as follows:

[0027]

[0028] in, represents the i-th end member, It represents the scale factor that controls the number of endmember queries corresponding to each endmember. represents the endmember query; R represents the number of endmembers; L represents the number of bands of the image;

[0029] When K=1, the endmember query degenerates into the endmember.

[0030] Based on the same inventive concept, the present invention also proposes a blind hyperspectral unmixing system based on endmember query design and Transformer attention mechanism, the system comprising:

[0031] An abundance map acquisition unit is used to construct an abundance mapping module, extract features from the hyperspectral image through a convolutional neural network in the abundance mapping module, and perform abundance decoding on the features to form an estimated abundance map;

[0032] A mapping unit for mapping the abundance map to an abundance embedding;

[0033] The information acquisition unit is used to construct a hybrid mechanism fitting Transformer module, which combines the endmember query, reconstruction query and abundance embedding through the hybrid mechanism fitting Transformer module to obtain the reconstructed query of endmember information and abundance information;

[0034] The hyperspectral image acquisition unit is used to reshape the original hyperspectral image according to the reconstruction query.

[0035] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism according to any one of the above-mentioned methods.

[0036] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism as described in any of the above items are executed.

[0037] The present invention is beneficial in that:

[0038] The blind hyperspectral unmixing method proposed in this paper is based on endmember query design and Transformer attention mechanism. On the basis of Transformer attention mechanism, the concepts of endmember query, reconstruction query and mixing mechanism fitting are proposed. The endmembers are decomposed in the spectral dimension in the form of endmember query, so that the accurate endmember estimation is decomposed into the degree of attention of different queries in different bands, capturing the existing mutual relationships, and achieving training convergence and accurate estimation under random initialization of endmembers; the reconstruction query is learned by the cross-attention mechanism, and the inherent mixing mechanism fitting of the hyperspectral image itself is achieved from the perspective of data fitting, without relying on any prior assumptions about the mixing mechanism. The present invention can realize "end-to-end" unsupervised blind hyperspectral unmixing, improve the accuracy of ground feature abundance and endmember estimation, and provide more accurate sub-pixel information for analysis and interpretation of subsequent applications.

[0039] Traditional blind unmixing methods usually rely on the accuracy of endmember initialization. If the initialization is not good, the unmixing effect will be significantly affected. Through the endmember query design module, the present invention dynamically queries and optimizes endmembers during the unmixing process, avoiding the limitations of traditional methods that rely on prior information. This not only reduces the error of endmember initialization, but also improves the flexibility and accuracy of blind unmixing. In the hyperspectral image unmixing task, the use of the Transformer module can better handle the complex relationship between different bands, thereby improving the accuracy of the unmixing results, especially under nonlinear and complex mixing models. The abundance mapping module extracts features from the hyperspectral image through a convolutional neural network, and then generates an abundance map through abundance decoding. The use of convolutional neural networks (CNNs) can fully explore the spatial features and local information of the image, providing high-quality input for subsequent abundance decoding, thereby improving the accuracy of abundance estimation during the unmixing process. By mapping the abundance map to abundance embedding, the model can better learn the abundance information of the ground objects and fit the mixing mechanism in combination with the Transformer module. This method can not only handle linear mixing problems, but also adapt to more complex nonlinear mixing situations, helping to achieve more accurate and robust demixing.

[0040] Existing deep learning methods usually rely on good endmember initialization, while the endmember query design of the present invention can adaptively adjust the endmembers during the unmixing process, avoiding the initialization difficulties in traditional methods and making the unmixing process more robust. Even without accurate prior endmember information, the model can still achieve good unmixing effects.

[0041] The present invention is applied to the field of blind hyperspectral image unmixing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flowchart of the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism described in embodiment 1;

[0043] Figure 2 This is a schematic diagram of the abundance mapping module described in Implementation Method 2;

[0044] Figure 3 Schematic diagram of the Transformer module for hybrid mechanism fitting described in Implementation Method 4;

[0045] Figure 4 This is a schematic diagram of the cascade decoding structure described in Embodiment 5;

[0046] Figure 5 This is the abundance estimation result of different methods described in embodiment 11 on the Jasper Ridge dataset, where: Figure 5 (a) is the true value result, Figure 5(b) is the abundance estimation result obtained by FCLSU method, Figure 5 (c) is the abundance estimation result obtained using the SCLSU method, Figure 5 (d) is the abundance estimation result obtained using the NUSAL method, Figure 5 (e) is the abundance estimation result obtained using the DAEU method, Figure 5 (f) is the abundance estimation result obtained using the SIDAEU method, Figure 5 (g) is the abundance estimation result obtained by MLAEM method, Figure 5 (h) is the abundance estimation result obtained using the DeepTrans method. Figure 5 (i) is the abundance estimation result obtained using the MLM-1DAE method, Figure 5 (j) is the abundance estimation result obtained using the A2SN method, Figure 5 (k) is the abundance estimation result obtained using the A2SAN method, Figure 5 (l) is the abundance estimation result obtained using the UNTR method; Figure 5 (m) is the abundance estimation result obtained using the UNTR-ds method;

[0047] Figure 6 The endmember estimation results of different methods described in Implementation 11 on the Jasper Ridge dataset are shown in the figure. From top to bottom, trees, water bodies, soil, and roads are represented respectively. The red curve is the true value, and the blue curve represents the estimation results of each method. The horizontal axis represents the band, and the vertical axis represents the normalized reflectance. Figure 6 (a) is the estimation result using the VCA method; Figure 6 (b) is the estimation result using the DAEU method; Figure 6 (c) is the estimation result using the SIDAEU method; Figure 6 (d) is the estimation result using the MLAEM method; Figure 6 (e) Estimation results using the DeepTrans method; Figure 6 (f) is the estimation result using the MLM-1DAE method; Figure 6 (g) is the estimation result using the A2SN method; Figure 6 (h) is the estimation result using the A2SAN method; Figure 6 (i) is the estimation result using the UNTR method; Figure 6 (j) is the estimation result using the UNTR-ds method;

[0048] Figure 7 The abundance estimation results of different methods described in Embodiment 11 on the Apex dataset are shown in the figure, which represent roads, trees, roofs, and water bodies from top to bottom. Figure 7 (a) is the true value result, Figure 7 (b) is the abundance estimation result obtained by FCLSU method, Figure 7 (c) is the abundance estimation result obtained using the SCLSU method, Figure 7 (d) is the abundance estimation result obtained using the NUSAL method, Figure 7 (e) is the abundance estimation result obtained using the DAEU method, Figure 7 (f) is the abundance estimation result obtained using the SIDAEU method, Figure 7 (g) is the abundance estimation result obtained by MLAEM method, Figure 7 (h) is the abundance estimation result obtained using the DeepTrans method. Figure 7 (i) is the abundance estimation result obtained using the MLM-1DAE method, Figure 7 (j) is the abundance estimation result obtained using the A2SN method, Figure 7 (k) is the abundance estimation result obtained using the A2SAN method, Figure 7 (l) is the abundance estimation result obtained using the UNTR method; Figure 7 (m) is the abundance estimation result obtained using the UNTR-ds method;

[0049] Figure 8 The endmember estimation results of different methods described in Implementation 11 on the Apex dataset are shown in the figure. From top to bottom, roads, trees, roofs, and water bodies are represented respectively. The red curve is the true value, and the blue curve represents the estimation results of each method. The horizontal axis represents the band, and the vertical axis represents the normalized reflectance. Figure 8 (a) is the estimation result using the VCA method; Figure 8 (b) is the estimation result using the DAEU method; Figure 8 (c) is the estimation result using the SIDAEU method; Figure 8 (d) is the estimation result using the MLAEM method; Figure 8 (e) Estimation results using the DeepTrans method; Figure 8 (f) is the estimation result using the MLM-1DAE method; Figure 8 (g) is the estimation result using the A2SN method; Figure 8 (h) is the estimation result using the A2SAN method; Figure 8 (i) is the estimation result using the UNTR method; Figure 8 (j) is the estimation result using the UNTR-ds method;

[0050] Figure 9The abundance estimation results of different methods described in embodiment 11 on the Urban dataset with four-endmember annotation are shown in the figure. From top to bottom, asphalt, lawn, tree, and roof are represented respectively; Figure 9 (a) is the true value result, Figure 9 (b) is the abundance estimation result obtained by FCLSU method, Figure 9 (c) is the abundance estimation result obtained using the SCLSU method, Figure 9 (d) is the abundance estimation result obtained using the NUSAL method, Figure 9 (e) is the abundance estimation result obtained using the DAEU method, Figure 9 (f) is the abundance estimation result obtained using the SIDAEU method, Figure 9 (g) is the abundance estimation result obtained by MLAEM method, Figure 9 (h) is the abundance estimation result obtained using the DeepTrans method. Figure 9 (i) is the abundance estimation result obtained using the MLM-1DAE method, Figure 9 (j) is the abundance estimation result obtained using the A2SN method, Figure 7 (k) is the abundance estimation result obtained using the A2SAN method, Figure 9 (l) is the abundance estimation result obtained using the UNTR method; Figure 9 (m) is the abundance estimation result obtained using the UNTR-ds method;

[0051] Figure 10 The endmember estimation results of different methods described in Implementation 11 on the Urban dataset with four endmember annotations are shown. From top to bottom, the figure represents asphalt, lawn, tree, and roof. The red curve is the true value, and the blue curve represents the estimation results of each method. The horizontal axis represents the band, and the vertical axis represents the normalized reflectance. Figure 10 (a) is the estimation result using the VCA method; Figure 10 (b) is the estimation result using the DAEU method; Figure 10 (c) is the estimation result using the SIDAEU method; Figure 10 (d) is the estimation result using the MLAEM method; Figure 10 (e) is the estimation result using the DeepTrans method; Figure 10 (f) is the estimation result using the MLM-1DAE method; Figure 10 (g) is the estimation result using the A2SN method; Figure 10 (h) is the estimation result using the A2SAN method; Figure 10 (i) is the estimation result using the UNTR method; Figure 10 (j) is the estimation result using the UNTR-ds method;

[0052] Figure 11 The endmember optimization process of the proposed method on the Jasper Ridge dataset for embodiment eleven, wherein the abscissa represents Band and the ordinate Reflectance represents reflectance, Figure 11 (a) represents random initialization; Figure 11 (b) represents training 100 times; Figure 11 (c) represents training 200 times; Figure 11 (d) represents training 300 times; Figure 11 (e) represents training 400 times; Figure 11 (f) represents training 500 times; Figure 11 (g) represents training 600 times; Figure 11 (h) represents training 700 times; Figure 11 (i) represents training 800 times; Figure 11 (j) represents training 900 times;

[0053] Figure 12 The hyperspectral image reconstructed on the Urban dataset and its root mean square error diagram for embodiment eleven, the first row is a color image synthesized by 28, 18, 12 bands, and the second row is an error diagram of the reconstructed image and the original image, wherein, Figure 12 (a) represents the image and error diagram reconstructed by the linear mixing model, Figure 12 (b) represents the reconstructed image and error diagram of the first decoder under the fitting mode of the nonlinear mixing mechanism proposed in the present application, Figure 12 (c) represents the reconstructed image and error diagram of the second decoder under the fitting mode of the nonlinear mixing mechanism proposed in the present application, Figure 12 (d) represents the reconstructed image and error diagram of the third decoder under the fitting mode of the nonlinear mixing mechanism proposed in the present application. DETAILED DESCRIPTION

[0054] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application.

[0055] Embodiment one, see Figure 1 This embodiment is described. The blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism described in this embodiment, the method comprises:

[0056] Construct an abundance mapping module, extract features from hyperspectral images through the convolutional neural network in the abundance mapping module, and decode the features into an estimated abundance map;

[0057] Mapping abundance maps to abundance embeddings;

[0058] Construct a hybrid mechanism fitting Transformer module, which combines endmember query, reconstruction query and abundance embedding to obtain reconstruction query of endmember information and abundance information;

[0059] Reshape the original hyperspectral image according to the reconstruction query.

[0060] This embodiment proposes a blind hyperspectral unmixing method based on endmember query design and the Transformer attention mechanism. Building on the Transformer attention mechanism, it proposes the concepts of endmember query, reconstruction query, and mixing mechanism fitting. Endmembers are decomposed in the spectral dimension using endmember query, allowing accurate endmember estimation to be decomposed into the degree of attention of different queries in different bands, capturing the interrelationships between them and achieving training convergence and accurate estimation under random endmember initialization. Reconstructed queries are learned using a cross-attention mechanism, achieving fitting of the mixing mechanism inherent in the hyperspectral image itself from a data fitting perspective, without relying on any prior assumptions about the mixing mechanism. The method proposed in this embodiment can achieve "end-to-end" unsupervised blind hyperspectral unmixing, improving the accuracy of ground feature abundance and endmember estimation, and providing more accurate sub-pixel information for analysis and interpretation in subsequent applications.

[0061] The characteristics of the mixing mechanism in this embodiment are analyzed as follows:

[0062] Assumptions represents the hyperspectral image to be unmixed, where L, H, and W represent the number of bands, height, and width of the image, respectively. represents the end member matrix, represents the corresponding abundance, where R represents the number of end members. The commonly used simplified mixing mechanism - linear mixed model (LMM) can be expressed as:

[0063]

[0064] in, represents additive noise, and J represents a matrix of all 1s. Considering the physical meaning of abundance, These represent the abundance nonnegativity constraint (ANC) and the abundance sum unity constraint (ASC), respectively. The linear mixing model assumes that photons arriving at the sensor interact with only one material. However, in reality, interactions also occur between endmembers. Photons arriving at the sensor may pass through multiple different materials, not a simplified linear mixture. A more practical nonlinear mixing model can be expressed as follows:

[0065]

[0066] Here, φ represents the mixing mechanism present in the hyperspectral image and is a nonlinear function acting on the endmembers E and parameterized by the corresponding abundances M. Nonlinear mixing models such as PPNMM and MLM, developed on this basis, simplify highly complex problems. However, integrating them into deep learning methods may not fully exploit the advantages of data-driven approaches. The advantage of deep learning lies in its ability to fit the desired function using observed data. This application to blind hyperspectral unmixing allows for the fitting of the mixing mechanism φ in a data-driven manner.

[0067] The deep learning method commonly used for blind unmixing is mainly the autoencoder, which uses the encoder output as the abundance and the weights of the decoder's fully connected layer as the endmembers to construct a self-supervised learning process in a linear mixing manner. Introducing the above-mentioned mixing mechanism fitting method into the autoencoder is a deepening of the general simplified linear mixing model. The encoder output is still considered as the abundance, but the decoder parameters no longer represent the endmembers but the mixing mechanism. The encoder output abundance and the additional defined learnable endmember matrix are used as input to reconstruct the hyperspectral image to be unmixed. Specifically, it can be expressed as:

[0068]

[0069] Among them, ξ E represents the encoder, and ξ D represents the decoder and the mixing mechanism learned from the observed data. It is worth noting that this approach does not inherently associate abundances with endmembers or guarantee their physical meaning. Therefore, while constructing the loss function to reconstruct the input hyperspectral image, additional linear auxiliary constraints must be added to the constructed autoencoder:

[0070]

[0071] Among them, Θ E and Θ D are the encoder and decoder parameters respectively. L n It is used to reduce the difference between the input hyperspectral image and the hyperspectral image reconstructed by the autoencoder, ensuring that the decoder parameters effectively fit the inherent mixing mechanism of the observed data. lIt is a linear mixing constraint used to increase the similarity between the hyperspectral image reconstructed by the linear mixing mechanism and the original image, ensure the alignment of end members and abundances, maintain their physical meaning, and make the learning of the above nonlinear mixing mechanism practical.

[0072] The goal of endmember querying is to estimate the spectral curves of pure substances, or endmembers, in hyperspectral images. By decomposing accurate endmember estimation into the importance estimation of multiple endmember queries across different bands, this method addresses the current deep learning method's requirement for good endmember initialization and achieves accurate endmember estimation even with random initialization, making it more practical.

[0073] The purpose of the reconstruction query is to restore the hyperspectral image to achieve self-supervised learning of the entire unmixing process. By capturing the endmember information and optimizing the endmember query, the reconstruction query is enriched with endmember and abundance information, which can restore the input hyperspectral image. It represents the reconstructed query, initialized with 0 elements, and the subsequent cascaded Transformer decoder will gradually optimize it during the self-supervised learning process.

[0074] Traditional blind unmixing methods often rely on the accuracy of endmember initialization. Poor initialization can significantly impact unmixing performance. By leveraging an endmember query design module, this implementation dynamically queries and optimizes endmembers during the unmixing process, avoiding the limitations of traditional methods that rely on prior information. This not only reduces endmember initialization errors but also improves the flexibility and accuracy of blind unmixing.

[0075] Implementation method 2, see Figure 2 This embodiment further defines the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism described in Embodiment 1. The abundance mapping module is composed of a U-shaped structure, extracts features through a convolutional layer, and uses a Swish activation function for nonlinear transformation. After the convolutional layer, the input hyperspectral image is mapped into an abundance map according to two downsampling and upsampling operations, and the output features of the encoder are optimized toward the specified abundance space by using the abundance non-negative constraint ASC and the abundance sum is one ANC constraint:

[0076]

[0077] Among them, X (i,h,w) It represents the feature that needs to be activated by Softmax at the spatial position (h, w), e represents the natural base, and ω represents the scale factor.

[0078] Specifically, the abundance mapping module constructed in this embodiment serves two purposes. First, it uses an abundance decoder to convert the captured multi-scale features into an abundance map, mapping the input hyperspectral image to a latent abundance space. Second, it converts the estimated abundance into an abundance embedding with the same dimensionality as the input. This embedding contains the abundance information while maintaining the same dimensionality as the input, providing abundance information input for subsequent decoding.

[0079] like Figure 2 As shown. In this embodiment, a simplified U-shaped network structure is used to form an abundance mapping module, which includes two downsampling and upsampling operations to extract rich features to generate an abundance map. This convolution structure effectively integrates multi-scale local features and enhances feature representation. Considering the significant impact of activation function on feature extraction, the present invention uses the Swish activation function to activate the features after convolution:

[0080]

[0081] Where x represents the feature to be activated. Mapping the input hyperspectral image to an abundance map involves optimizing the encoder's output features toward a specified abundance space using ASC and ANC constraints. The Softmax activation function is well-suited to ASC and ANC constraints and is used to ensure abundance sparsity, so it is used in this implementation to generate the final abundance map:

[0082]

[0083] Among them, X (i,h,w) It represents the feature that needs to be activated by Softmax at the spatial position (h, w). Considering that abundance maps are often sparse in reality, a scaling factor ω is used here to achieve this abundance sparsity, with a default setting of 3.5.

[0084] This embodiment adopts a simplified U-Net structure, extracts the features of the hyperspectral image through the convolution layer, and then restores the spatial resolution through downsampling and upsampling operations, which can effectively retain the spatial information of the input image and help accurately generate the abundance map of the ground objects. U-Net combines the high-level features of the encoder stage with the low-level features of the decoder stage through jump connections, which can effectively combine the global information of the image with the local details and improve the accuracy of abundance decoding. In this method, the convolution layer is used to extract the features of the input hyperspectral image, and the Swish activation function is used to perform nonlinear transformations on these features. When modeling complex features in the hyperspectral image (such as the nonlinear relationship between different ground objects and mixed backgrounds), the performance of the unmixing results can be improved.

[0085] In this implementation, downsampling helps compress input data and extract high-level features, while upsampling restores image resolution, generates a detailed abundance map, and maintains spatial consistency. This operation, through the encoding and decoding process, helps the model capture the characteristic hierarchy of the hyperspectral image, thereby improving unmixing accuracy. Due to the skip connections in the U-Net structure, low-level and high-level features can be effectively combined after downsampling and upsampling operations, helping to preserve spatial detail during the unmixing process and avoid information loss.

[0086] During the unmixing process, the abundance of land features usually needs to satisfy non-negativity (i.e., the abundance value should be greater than or equal to zero) and the sum of the abundance of all land features must be one (i.e., the sum of the abundance of each land feature must be 1). This embodiment optimizes the unmixing process by adding an abundance non-negativity constraint (ASC) and an abundance sum-to-one constraint (ANC). The non-negative constraint ensures that the abundance estimation results of land features are physically reasonable and avoids the situation where the abundance has negative values. The abundance sum-to-one constraint ensures that the sum of the abundances of multiple land features is 1, further ensuring that the unmixing result is consistent with the physical characteristics of hyperspectral imaging data. These constraints help to optimize the network learning process, making the model more consistent with the abundance distribution of actual land features during training, avoiding model overfitting or learning unrealistic results, thereby improving the accuracy of unmixing.

[0087] In the final stage of the model, the Softmax activation function is used to activate the i-th feature at the spatial location to ensure that the abundance estimate is within a reasonable range. Softmax converts the network output into a probability distribution, ensuring that the abundance value of each feature falls within the interval [0, 1] and that the sum of the abundances of all features is 1. This processing approach is highly interpretable in practical applications, facilitating analysis and verification of the unmixing results. The output of the Softmax activation can clearly determine the abundance of different features at each pixel, especially in complex mixed backgrounds. This effectively reduces uncertainty and improves the accuracy and robustness of the unmixing process.

[0088] Implementation method three. This implementation method further limits the blind hyperspectral unmixing method based on end-member query design and Transformer attention mechanism described in implementation method one. The abundance map is mapped to an abundance embedding with the same size as the input hyperspectral image through a convolutional neural network. The abundance embedding with the same size is obtained by retaining the original spatial size or by using any one of the downsampling operations.

[0089] This embodiment is described in conjunction with embodiment 1. Since the self-attention and cross-attention mechanisms of the Transformer are used to fit the inherent mixing mechanism in the data after the abundance map is output. Since the cross-attention mechanism requires the dimensions of multiple input data to be consistent, the abundance map obtained by the abundance mapping module must be converted into a form suitable for subsequent Transformer decoding. In order to achieve this goal, the concept of abundance embedding is proposed in this embodiment, and the abundance is mapped to a feature with the same number of bands as the input image and containing abundance information through a simple convolutional neural network. Taking into account the sparsity and local similarity of the image, two methods are used to form pixel-by-pixel abundance embedding: one method retains the original spatial size, and the other is to use a downsampling operation, setting the step size to 2 during convolution to form a downsampled sparse abundance embedding. Subsequent experiments show that both methods perform well in blind hyperspectral unmixing. The key difference is that the downsampling method reduces the amount of floating-point operations. Therefore, when the amount of floating-point operations is not a priority, the present invention recommends using the first method. If the amount of operations needs to be reduced, the second downsampling method can be used.

[0090] Implementation method 4, see Figure 3 This embodiment further defines the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism described in Embodiment 1. The Transformer module fitted by the mixing mechanism includes:

[0091] Q represents the Query vector in the Transformer attention mechanism, K represents the Key vector in the Transformer attention mechanism, and V represents the Value vector in the Transformer attention mechanism. The output Z is expressed as:

[0092]

[0093] in, Used to control scale scaling;

[0094] Match the attention mechanism with the reconstructed query using a multi-head attention mechanism for optimization;

[0095] The Transformer module decoder is represented as:

[0096]

[0097] Among them, Z (l,1) represents the self-attention mechanism of the l-th decoder, Z (l,2) represents the cross attention mechanism of the l-th decoder;

[0098] The entire Transformer module uses a cascaded decoder structure to gradually optimize and reconstruct queries.

[0099] Specifically:

[0100] Assuming that Q, K, and V represent the query, key, and value vectors in the Transformer attention mechanism, the output Z can be expressed as:

[0101]

[0102] in, Used to control scale scaling. When the attention mechanism corresponds to the optimization of the present invention for reconstructing queries, a decoder in the proposed Transformer module can be expressed as:

[0103]

[0104] Among them, Z (l,1) and Z (l,2) They represent the self-attention mechanism and cross-attention mechanism of the l-th decoder respectively. The self-attention mechanism is used to initially combine the end-member query with the reconstruction query, while the cross-attention mechanism further fuses the end-member query with the abundance information to obtain an optimized reconstruction query. In actual use, the multi-head attention mechanism (MHA) is used to implement it. (l,1) and T (l,2) It represents the l-th decoder summation and layer normalization operation. When l is 1, Initialized with all 0 elements, after updating It can be expressed as:

[0105]

[0106] In the formula, MLP stands for multi-layer perceptron, which also stands for fully connected layer. In addition, the specific schematic diagram of a single decoder in the Transformer module is as follows Figure 3 To obtain sufficiently accurate reconstructed queries, the entire Transformer module adopts a cascaded decoder structure to gradually optimize the reconstructed queries.

[0107] Implementation method five, see Figure 4 This embodiment further defines the blind hyperspectral unmixing method based on endmember query design and the Transformer attention mechanism described in Embodiment 4. The structure of the cascade decoder is as follows: the optimized reconstruction query, endmember query, and abundance embedding of the previous layer are used as inputs to share the fully connected layer mapping and reshape it into a hyperspectral image, and the reconstructed image is gradually optimized from coarse to fine.

[0108] This embodiment is described in conjunction with the fourth embodiment. In order to obtain sufficiently accurate reconstructed queries, the entire Transformer module adopts a cascade decoder structure to gradually optimize the reconstructed query, such as Figure 4 This approach allows the reconstruction query generated by each decoder to restore multiple input hyperspectral images through a shared fully connected layer, achieving progressive optimization from coarse to fine. Therefore, the loss function can be composed of the reconstruction losses of multiple intermediate reconstruction queries and the reconstruction loss of the final reconstruction query, which differ only in the degree of refinement of the restored image.

[0109] Implementation 6. This implementation further limits the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism described in Implementation 4. The method also includes constructing a loss function including three perspectives: reconstruction, abundance, and endmember:

[0110]

[0111] L final =L cas +L aux ,

[0112] Among them, L cas represents the cascade loss, Indicates that the focus of the entire model is the final result, represents the reconstructed image corresponding to the lth decoder, I represents the input hyperspectral image, represents the estimated abundance matrix, represents the estimated end-member matrix, N represents the number of cascaded decoders, represents the auxiliary output hyperspectral image under the linear mixed model, represents the transpose of the estimated endmember matrix, L aux represents additional linear mixing auxiliary constraints.

[0113] Blind hyperspectral image unmixing does not rely on true value pair model training, and only uses a self-supervised approach, making the construction of the loss function crucial. In the present invention, the loss function is constructed from three perspectives: reconstruction, abundance, and endmembers. The reconstruction loss ensures the consistency between the input hyperspectral image and the output hyperspectral image, enabling the model to fully learn the hyperspectral image feature representation. The abundance constraint takes into account the sparsity of the abundance of ground objects, while the endmember constraint ensures that the estimated endmembers are numerically reasonable. Considering that the mean square error (MSE) captures numerical differences, while the spectral angular distance (SAD) captures scale invariance, the present invention simultaneously uses both to construct the basic reconstruction loss, as shown below:

[0114]

[0115] Where (h, w) represents the spatial position, ||·||2 represents the l2 norm, and α represents the weight of the spectral angular distance, which is set to 1 in the experiments of the present invention.

[0116] For abundance constraints, the Softmax activation function can ensure the ASC and ANC constraints. Currently, many studies generally believe that abundance is sparse. Therefore, this embodiment adopts l 1 / 2 Regularization to achieve abundance sparsity:

[0117]

[0118] Considering that endmember estimation in blind hyperspectral unmixing often leads to unreasonable values ​​in practical applications, the present invention proposes an endmember loss L endm To improve this problem, the endmember values ​​are constrained to be the average value of the input hyperspectral data:

[0119]

[0120] Then the loss function adopted can be expressed as:

[0121]

[0122] Where β and γ are the weights of the sparse constraint and the endmember constraint respectively. Figure 4 As shown in Figure 1, the decoding structure is carried out step by step from coarse to fine in the process of reconstructing the input hyperspectral image. In addition to the final reconstructed image, N-1 intermediate restored images are also generated. This makes the adopted loss function expandable to the cascade loss L cas :

[0123]

[0124] L cas With L n Corresponding. It represents the reconstructed image corresponding to the lth decoder. Since the focus of the entire model is the final result Therefore, the weight of the intermediate reconstruction results is reduced. In order to ensure the physical meaning of end members and abundances and to correspond the two, an additional linear mixing auxiliary constraint L aux It can be expressed as:

[0125]

[0126] Among them, L aux and L in formula (1.4) l Correspondingly, the final loss function L final It can be expressed as:

[0127] L final =Lcas +L aux .

[0128] Implementation 7: This implementation further limits the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism described in Implementation 1. The endmember acquisition method is:

[0129]

[0130] in, represents the i-th end member, It represents the scale factor that controls the number of endmember queries corresponding to each endmember. represents the endmember query; R represents the number of endmembers; L represents the number of bands of the image;

[0131] When K=1, the endmember query degenerates into the endmember.

[0132] In this implementation, endmember queries from i·K to (i+1)·K are used to obtain the i-th endmember. This indirect acquisition method can decompose a relatively accurate endmember estimate into the average of multiple endmember queries in different bands, reducing the need for well-initialized endmembers.

[0133] Implementation 8: The blind hyperspectral unmixing system based on endmember query design and Transformer attention mechanism described in this implementation includes:

[0134] An abundance map acquisition unit is used to construct an abundance mapping module, extract features from the hyperspectral image through a convolutional neural network in the abundance mapping module, and perform abundance decoding on the features to form an estimated abundance map;

[0135] A mapping unit for mapping the abundance map to an abundance embedding;

[0136] The information acquisition unit is used to construct a hybrid mechanism fitting Transformer module, which combines the endmember query, reconstruction query and abundance embedding through the hybrid mechanism fitting Transformer module to obtain the reconstructed query of endmember information and abundance information;

[0137] The hyperspectral image acquisition unit is used to reshape the original hyperspectral image according to the reconstruction query.

[0138] Implementation method nine. A computer device described in this implementation method includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the blind hyperspectral unmixing method based on end-member query design and Transformer attention mechanism according to any one of implementation methods one to seven.

[0139] Implementation method ten. A computer-readable storage medium described in this implementation method stores a computer program, which, when executed by a processor, executes the steps of the blind hyperspectral unmixing method based on end-member query design and Transformer attention mechanism as described in any one of implementation methods one to seven.

[0140] Implementation method 11, see Figures 5 to 10 This embodiment provides a specific example of the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism described in embodiment 1, and is also used to explain embodiments 2 to 7. Specifically:

[0141] The abundance mapping module and the hybrid mechanism fitting Transformer module proposed in this embodiment constitute a blind unmixing autoencoder network UNTR, which involves multiple parameter settings. The settings of general hyperparameters are shown in Table 1 below.

[0142] Table 1 General hyperparameter settings

[0143]

[0144] Among them, ω represents the weight of the Softmax activation function, α is the weight of the spectral angular distance loss, γ is the weight of the endmember loss, and β is the weight of the abundance sparsity.

[0145] The number of endmember queries and the number of cascaded decoders for different data sets are shown in Table 2 below.

[0146] Table 2 Endmember query and cascade decoder number settings of the proposed method on different datasets

[0147]

[0148]

[0149] In Table 2, UNTR-ds refers to the method of downsampling the abundance embedding. K and N represent the number of endmember queries and the number of cascaded decoders, respectively.

[0150] To quantitatively evaluate the performance of each blind hyperspectral image method, this implementation uses the root mean square error (RMSE) and spectral angular distance (SAD) as evaluation metrics for abundance estimation and endmember estimation for each feature class, respectively. Furthermore, the average root mean square error (aRMSE) and average spectral angular distance (aSAD) are used as global evaluation metrics, as shown below:

[0151]

[0152] The proposed UNTR and UNTR-ds algorithms are compared with the traditional methods FCLSU, SCLSU, and NUSAL; the pixel-level deep learning methods DAEU, SIDAEU, MLAEM, and MLM-1DAE; and the full-image deep learning methods DeepTrans, A2SN, and A2SAN. Except for the proposed method, A2SN, and S2SAN, which do not require endmember initialization, the remaining methods all use vertex component analysis (VCA) to initialize endmembers.

[0153] Table 3 Experimental results on the Jasper Ridge dataset

[0154]

[0155] Table 3 shows the unmixing results of different methods on the Jasper Ridge dataset. The UNTR method proposed in the present invention achieved the lowest aRMSE and aSAD, which were 0.0470 and 0.0360, respectively. In terms of abundance, the estimation error of each category was the lowest, which was significantly reduced compared with other methods. The most significant error reduction was observed in the water category, which was reduced by 65%. For end members, reliable results were obtained for each category. Even if it was not the best estimate, a suboptimal estimate was usually obtained, and the deviation from the best estimated end member was very small. On the other hand, the proposed UNTR method does not adopt any end member initialization method. Compared with A2SN and A2SAN, which also do not have good end member initialization, the high accuracy achieved by the proposed UNTR under random initialization shows the effectiveness of the method.

[0156] In addition, the abundance maps and endmember spectra estimated by different methods are as follows Figure 5 and Figure 6 As shown in Figure 2, the UNTR abundance map is visually very similar to the true value and maintains consistency in local details. For endmembers, the spectrum estimated by UNTR (blue curve) and the true spectrum (red curve) tend to change in a consistent manner, with significantly smaller deviations in the road category, while other methods often show large errors in this category.

[0157] Table 4 Experimental results on the Apex dataset

[0158]

[0159] Table 4 shows the unmixing results of different methods on the recently proposed Apex dataset. Quantitative analysis shows that DeepTrans performs best in terms of abundance and endmember estimation on this dataset, followed closely by the UNTR method proposed in this invention. For specific endmembers, recent methods such as DeepTrans, MLM-1DAE, and A2SN show high accuracy in certain categories, while the UNTR of this invention performs well in abundance estimation of roads and rooftops, and performs well in all endmember estimations. Considering that the proposed method has no endmember initialization, it has achieved a great improvement compared with A2SN and A2SAN without endmember initialization, which once again proves the feasibility and effectiveness of the proposed method. The abundance maps and endmember spectra estimated by different methods on the Apex dataset are shown in Figure 4. Figure 7 and Figure 8 shown.

[0160] Table 5 Experimental results on the Urban dataset with four end members

[0161]

[0162] Considering the comparison of different blind hyperspectral image unmixing methods on datasets with larger spatial range and more texture information, Table 5 shows the experimental results on the Urban dataset with four-endmember annotation. It can be seen that the proposed UNTR method achieved the lowest aRMSE and aSAD in all four categories. Among them, the aSAD is significantly lower than that of other methods, and is reduced by 61% compared with the second-best method A2SN. The abundance maps and endmember spectra estimated by different methods on the Urban dataset with four-endmember annotation are shown in Figure 5. Figure 9 and Figure 10 shown.

[0163] Furthermore, in order to more clearly show the convergence process of the proposed method of random endmember initialization during training and verify the feasibility of the proposed endmember query, Figure 11 The changes and convergence of end members during the training process are given.

[0164] from Figure 11 As can be seen in the figure, because the endmember query is randomly initialized, the resulting endmembers are also disordered. However, with increasing training times, the spectral curves of different endmembers gradually become separable and eventually stabilize. This endmember visualization during the training process further validates the effectiveness of the proposed method, which can obtain endmember spectra even with random initialization, making it a more practical deep learning estimation method.

[0165] In order to verify the nonlinear capability of the proposed UNTR method, Figure 12The reconstructed hyperspectral image and root mean square error are given under the hybrid mechanism of estimated abundance, endmember query and Transformer cascade decoder fitting.

[0166] It can be found that if the reconstructed hyperspectral image ( Figure 12 (a) obviously loses detail information and has a large reconstruction error. The proposed UNTR uses reconstruction queries with different optimization levels to restore hyperspectral images such as Figure 12 As shown in (b)-(d), the restored hyperspectral image maintains better detail information and has a lower RMSE. Furthermore, as the number of decoders increases, the reconstruction error of the restored hyperspectral image gradually decreases, demonstrating the effectiveness of the cascade structure. The error of the hyperspectral image restored by the first decoder is significantly lower than that restored by the linear mixture model. While the error of subsequent cascaded decoders decreases, the decrease is less significant, demonstrating the importance of decoding in the first layer. The structure of the subsequent cascaded decoders ensures the stability of the algorithm.

[0167] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure and are not intended to limit its scope of protection. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the disclosed claims.

Claims

1. A blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism, characterized by: The method comprises: Construct an abundance mapping module, extract features from hyperspectral images through the convolutional neural network in the abundance mapping module, and decode the features into an estimated abundance map; Mapping abundance maps to abundance embeddings; Construct a hybrid mechanism fitting Transformer module, which combines endmember query, reconstruction query and abundance embedding to obtain reconstruction query of endmember information and abundance information; reshape the original hyperspectral image according to the reconstruction query; The Transformer module of the hybrid mechanism fitting includes: Q Represents the Query vector in the Transformer attention mechanism, K Represents the Key vector in the Transformer attention mechanism, V Represents the Value vector in the Transformer attention mechanism, output Z Expressed as: in, Used to control scale scaling; Match the attention mechanism with the reconstructed query using a multi-head attention mechanism for optimization; The Transformer module decoder is represented as: in, Indicates the l The self-attention mechanism of the decoder, Indicates the l Cross-attention mechanism for decoders; represents the end member query, Representative l The decoder sums, Representative l decoder layer normalization operation; The entire Transformer module uses a cascaded decoder structure to gradually optimize and reconstruct the query; The structure of the cascade decoder is as follows: the reconstruction query, endmember query and abundance embedding optimized in the previous layer are taken as input to share the fully connected layer mapping and reshape it into a hyperspectral image, and the reconstructed image is gradually optimized from coarse to fine.

2. The blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism according to claim 1 is characterized in that The abundance mapping module is composed of a U-shaped structure, extracts features through a convolutional layer, and uses a Swish activation function for nonlinear transformation; After passing through the convolutional layer, the input hyperspectral image is mapped to an abundance map according to two downsampling and upsampling operations, and the output features of the encoder are optimized to the specified abundance space by using the abundance non-negative constraint ASC and the abundance sum is one ANC constraint: in, Indicates the spatial position ( h , w ) i Features that need to be activated by Softmax, represents the natural base, Represents the scale factor.

3. The blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism according to claim 1 is characterized in that The abundance map is mapped to an abundance embedding having the same size as the input hyperspectral image through a convolutional neural network. The abundance embedding having the same size is obtained by retaining the original spatial size or by using a downsampling operation.

4. The blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism according to claim 1 is characterized in that The method also includes constructing a loss function that includes three perspectives: reconstruction, abundance, and endmembers: ， ， ， in, represents the cascade loss, Represents the final result of the entire model, Indicates the l The reconstructed image corresponding to the decoder is represents the input hyperspectral image, represents the estimated abundance matrix, represents the estimated endmember matrix, Indicates the number of cascaded decoders, represents the auxiliary output hyperspectral image under the linear mixed model, represents the transpose of the estimated endmember matrix, represents additional linear mixing auxiliary constraints.

5. The blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism according to claim 1 is characterized in that The end member acquisition method is: in, Represents the i End members, It represents the scale factor that controls the number of endmember queries corresponding to each endmember. It represents the end member query; Indicates the number of end members; M Represents the number of bands of the image; when When , the endmember query degenerates into the endmember.

6. A blind hyperspectral unmixing system based on endmember query design and Transformer attention mechanism, characterized by: The system is implemented based on the method according to claim 1, and the system includes: An abundance map acquisition unit is used to construct an abundance mapping module, extract features from the hyperspectral image through a convolutional neural network in the abundance mapping module, and perform abundance decoding on the features to form an estimated abundance map; A mapping unit for mapping the abundance map to an abundance embedding; The information acquisition unit is used to construct a hybrid mechanism fitting Transformer module, which combines the endmember query, reconstruction query and abundance embedding through the hybrid mechanism fitting Transformer module to obtain the reconstructed query of endmember information and abundance information; The hyperspectral image acquisition unit is used to reshape the original hyperspectral image according to the reconstruction query.

7. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the blind hyperspectral unmixing method based on endmember query design and Transformer attention mechanism as described in any one of claims 1 to 5.

Images