Hyperspectral image unmixing method and device, electronic equipment and computer program product

Abstract

The application discloses a hyperspectral image unmixing method and device, electronic equipment and computer program product, and relates to the technical field of spectral image unmixing. The hyperspectral image data is input into a hyperspectral unmixing deep learning model for local spatial-spectral feature downsampling and global spectral information perception, so that the down-sampled local spatial-spectral feature and the global spectral feature map are obtained. The local spatial-spectral feature and the global spectral feature map are stacked and decoded to obtain a plurality of end-member spectral signals. The hyperspectral unmixing deep learning model comprises a local spatial-spectral feature downsampling branch and a global spectral information perception network branch. The local spatial-spectral feature downsampling branch is used for local spatial-spectral feature extraction and downsampling. The global spectral information perception network branch is used for calculating the correlation weight between different spectral channels and fusing the spatial attention weight from the convolution branch. The application can realize global spectral perception, significantly improve the hyperspectral information utilization rate, and improve the unmixing effect of the spectral image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of spectral image unmixing, and particularly relates to a hyperspectral image unmixing method and device, an electronic device and a computer program product. BACKGROUND

[0002] Due to the correlation between ground objects and ground objects and the correlation between spectral bands in the hyperspectral image, there is a large amount of redundancy in the spectral information contained in the hyperspectral image, so it is necessary to unmix the hyperspectral image.

[0003] At present, for the unmixing of the hyperspectral image, the commonly used way is to unmix through a single convolutional neural network or a sampling Transformer architecture model, however, the unmixing method using a single convolutional neural network is insufficient in utilizing spectral information, and it is difficult to capture long-range dependencies in the spectral dimension, which easily leads to loss of spectral details in the down-sampling process, and finally leads to poor spectral unmixing effect. The Transformer architecture model relies on the self-attention mechanism, and has a great advantage over the convolutional neural network in capturing long-range relationships, can assign different attention weights to the spectral sequence, and achieve the effect of global spectral perception, but it also has the disadvantage of insufficient ability to capture local spatial-spectral features, which also leads to poor spectral unmixing effect.

[0004] Therefore, how to provide an effective scheme to improve the unmixing effect of the spectral image has become a problem to be solved in the prior art. SUMMARY

[0005] The purpose of the present application is to provide a hyperspectral image unmixing method, device, electronic device and computer program product to solve the above problems existing in the prior art.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0007] In a first aspect, the present application provides a hyperspectral image unmixing method, comprising:

[0008] inputting the hyperspectral image data to be unmixing into a pre-trained hyperspectral unmixing deep learning model for local spatial-spectral feature down-sampling and global spectral information perception, to obtain down-sampled local spatial-spectral features Y and global spectral feature map Xout;

[0009] stacking the down-sampled local spatial-spectral features Y and the global spectral feature map Xout and then decoding to obtain a plurality of endmember spectral signals;

[0010] The hyperspectral unmixing deep learning model comprises a local spatial-spectral feature down-sampling branch based on a convolutional neural network and a global spectral information perception network branch based on a self-attention mechanism.

[0011] The local spatial-spectral feature downsampling branch based on the convolutional neural network is used for local spatial-spectral feature extraction and local spatial-spectral feature downsampling of the hyperspectral image data.

[0012] The global spectral information perception network branch based on the self-attention mechanism is used for extracting the self-attention SA between spectral channels in the hyperspectral image data, determining the spatial attention weight Xp of each spectral channel based on the local spatial-spectral feature of the hyperspectral image data, determining the value matrix Vspa after spatial attention is given based on the spatial attention weight Xp of each spectral channel and the feature matrix obtained by linear projection of the hyperspectral image data in the self-attention extraction process, and linearly combining the value matrix Vspa after spatial attention is given and the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

[0013] In a possible design, the local spatial-spectral feature downsampling branch based on the convolutional neural network includes a first convolutional layer of 1×1, a second convolutional layer of 3×3, and a third convolutional layer of 1×1, the first convolutional layer is used for feature extraction of the hyperspectral image data to obtain first features, the second convolutional layer is used for feature extraction of the first features to obtain the local spatial-spectral features, and the third convolutional layer is used for feature extraction of the local spatial-spectral features to obtain the down-sampled local spatial-spectral features.

[0014] In a possible design, the global spectral information perception network branch based on the self-attention mechanism includes a global spectral perception module and a convolutional network local spatial-spectral feature transmission module.

[0015] The convolutional network local spatial-spectral feature transmission module is used for determining the spatial attention weight Xp of each spectral channel based on the local spatial-spectral feature of the hyperspectral image data.

[0016] The global spectral perception module is used for extracting the self-attention SA between spectral channels in the hyperspectral image data, determining the value matrix Vspa after spatial attention is given based on the spatial attention weight Xp of each spectral channel and the feature matrix obtained by linear projection of the hyperspectral image data in the self-attention extraction process, and linearly combining the value matrix Vspa after spatial attention is given and the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

[0017] In a possible design, the convolutional network local spatial-spectral feature transmission module includes a fourth convolutional layer of 1×1 and a fifth convolutional layer of 5×5, and the convolutional network local spatial-spectral feature transmission module is used for determining the spatial attention weight Xp of each spectral channel in the following manner:

[0018] extracting features of the hyperspectral image data through the fourth convolutional layer to obtain second features;

[0019] extracting features of the second features through the fifth convolutional layer to obtain third features;

[0020] resolving the third features into features of multiple spectral channels to obtain spatial attention weights Xp of each spectral channel.

[0021] In one possible design, the global spectral perception module is configured to determine the global spectral feature map Xout in the following manner:

[0022] taking spectral image data of each spectral channel in the hyperspectral image data as a spectral data block to obtain a sequence of spectral data blocks;

[0023] linearly projecting the sequence of spectral data blocks through three different matrices to obtain a query matrix Q, a key matrix K and a value matrix V of each spectral channel;

[0024] dividing the query matrix Q, the key matrix K and the value matrix V of each spectral channel into multiple attention heads along a spectral band dimension;

[0025] calculating self-attention of each attention head in each spectral channel, and determining self-attention SA between spectral channels in the hyperspectral image data based on the self-attention of each attention head in each spectral channel;

[0026] element-wise multiplying the spatial attention weights Xp of each spectral channel and the value matrix V of each spectral channel to obtain a value matrix Vspa to which spatial attention is assigned;

[0027] linearly combining the value matrix Vspa to which spatial attention is assigned and the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

[0028] In one possible design, the query matrix Q, the key matrix K and the value matrix V of each spectral channel are as follows:

[0029] ;

[0030] wherein X represents a spectral data block in the sequence of spectral data blocks, 、 and all represent matrix parameters.

[0031] In one possible design, the decoding of the down-sampled local spatial-spectral features Y and the global spectral feature map Xout after stacking obtains multiple endmember spectral signals, including:

[0032] The downsampled local spatial spectral features Y and the global spectral feature map Xout are stacked and then extracted through a convolutional layer.

[0033] The features extracted from the convolutional layer are activated into an abundance map using a non-linear activation function.

[0034] The abundance map was decoded into a hyperspectral image by linear mapping, resulting in multiple endmember spectral signals.

[0035] In a second aspect, the present invention provides a hyperspectral image demixing apparatus, comprising:

[0036] The feature sampling and information perception unit is used to input the hyperspectral image data to be unmixed into a pre-trained hyperspectral unmixing deep learning model to perform local spatial spectral feature downsampling and global spectral information perception, and obtain the downsampled local spatial spectral feature Y and global spectral feature map Xout.

[0037] The stacking and decoding unit is used to stack the downsampled local spatial spectral feature Y with the global spectral feature map Xout and then decode them to obtain multiple endmember spectral signals.

[0038] The hyperspectral unmixing deep learning model includes a local spatial spectral feature downsampling branch based on a convolutional neural network and a global spectral information perception network branch based on a self-attention mechanism.

[0039] The local spatial spectral feature downsampling branch based on the convolutional neural network is used to extract and downsample the local spatial spectral features of the hyperspectral image data.

[0040] The global spectral information sensing network branch based on the self-attention mechanism is used to extract the self-attention SA between spectral channels in the hyperspectral image data, determine the spatial attention weight Xp of each spectral channel based on the local spatial spectral features of the hyperspectral image data, determine the value matrix Vspa after spatial attention is applied based on the spatial attention weight Xp of each spectral channel and the feature matrix obtained by linear projection of the hyperspectral image data during the self-attention extraction process, and linearly combine the value matrix Vspa after spatial attention with the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

[0041] Thirdly, the present invention provides an electronic device comprising a memory, a processor, and a transceiver connected in sequence and communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the hyperspectral image demixing method as described in the first aspect or any possible design of the first aspect.

[0042] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the hyperspectral image demixing method described in the first aspect or any possible design of the first aspect.

[0043] Fifthly, the present invention provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the hyperspectral image unmixing method as described in the first aspect or any possible design of the first aspect.

[0044] Beneficial effects:

[0045] This invention employs a hyperspectral unmixing deep learning model comprising a local spatial spectral feature downsampling branch based on a convolutional neural network and a global spectral information perception network branch based on a self-attention mechanism. The local spatial spectral feature downsampling branch based on the convolutional neural network is used to extract and downsample local spatial spectral features from hyperspectral image data. The global spectral information perception network branch based on the self-attention mechanism is used to extract the self-attention (SA) between spectral channels in the hyperspectral image data, determine the spatial attention weights (Xp) of each spectral channel based on the local spatial spectral features of the hyperspectral image data, determine the value matrix (Vspa) after spatial attention is applied based on the feature matrix obtained by linear projection of the hyperspectral image data during the self-attention extraction process using the spatial attention weights (Xp) of each spectral channel and the feature matrix obtained by linear projection of the hyperspectral image data during the self-attention extraction process, and linearly combine the value matrix (Vspa) after spatial attention is applied with the self-attention (SA) between spectral channels to obtain the global spectral feature map (Xout). Thus, when demixing hyperspectral image data, the local spatial spectral features can be extracted using the local spatial spectral feature downsampling branch of the convolutional neural network, and the correlation weights between different spectral channels can be dynamically calculated using the global spectral information perception network branch based on the self-attention mechanism. The spatial attention weights from the convolutional branch are also fused to achieve global spectral modeling with spatial perception capabilities. This ensures spectral details while achieving global spectral perception, significantly improving the utilization rate of hyperspectral information, enhancing the demixing effect of spectral images, and facilitating practical applications and promotion. Attached Figure Description

[0046] Figure 1 A flowchart of the hyperspectral image demixing method provided in the embodiments of this application;

[0047] Figure 2 This is a schematic diagram of the overall structure of the hyperspectral unmixing deep learning model provided in the embodiments of this application;

[0048] Figure 3 A schematic diagram of the structure of a local spatial spectrum feature downsampling branch based on a convolutional neural network provided in an embodiment of this application;

[0049] Figure 4 A schematic diagram of the structure of a global spectral information sensing network branch based on a self-attention mechanism provided in an embodiment of this application;

[0050] Figure 5 A block diagram of the hyperspectral image demixing device provided in the embodiments of this application;

[0051] Figure 6 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0053] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0054] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0055] To improve the demixing effect of spectral images, embodiments of this application provide a hyperspectral image demixing method, apparatus, electronic device, and computer program product. This hyperspectral image demixing method, apparatus, electronic device, and computer program product can significantly improve the utilization rate of hyperspectral information and enhance the demixing effect of spectral images.

[0056] like Figure 1 As shown, the first aspect of the present application provides a hyperspectral image demixing method, which may include, but is not limited to, the following steps S101-S102.

[0057] Step S101. Input the hyperspectral image data to be unmixed into a pre-trained hyperspectral unmixing deep learning model to perform local spatial spectral feature downsampling and global spectral information perception, and obtain the downsampled local spatial spectral feature Y and global spectral feature map Xout.

[0058] Step S102. Stack the downsampled local spatial spectral feature Y with the global spectral feature map Xout and then decode to obtain multiple endmember spectral signals.

[0059] Please see Figure 2 The hyperspectral unmixing deep learning model includes a local spatial spectral feature downsampling branch based on a convolutional neural network and a global spectral information perception network branch based on a self-attention mechanism. The local spatial spectral feature downsampling branch based on the convolutional neural network is used to extract and downsample local spatial spectral features from the hyperspectral image data. The global spectral information perception network branch based on the self-attention mechanism is used to extract the self-attention (SA) between spectral channels in the hyperspectral image data, determine the spatial attention weights (Xp) of each spectral channel based on the local spatial spectral features of the hyperspectral image data, determine the value matrix (Vspa) after applying spatial attention based on the spatial attention weights (Xp) and the feature matrix obtained by linear projection of the hyperspectral image data during the self-attention extraction process, and linearly combine the value matrix (Vspa) with the self-attention (SA) between spectral channels to obtain the global spectral feature map (Xout).

[0060] The local spatial spectral feature downsampling branch based on a convolutional neural network is used to extract and downsample local spatial spectral features from the hyperspectral image data. Please refer to [link / reference]. Figure 3 The local spatial spectrum feature downsampling branch based on convolutional neural network includes a 1×1 first convolutional layer, a 3×3 second convolutional layer, and a 1×1 third convolutional layer. The first convolutional layer is used to extract features from hyperspectral image data to obtain the first feature. The second convolutional layer is used to extract features from the first feature to obtain local spatial spectrum features. The third convolutional layer is used to extract features from the local spatial spectrum features to obtain downsampled local spatial spectrum features.

[0061] When demixing hyperspectral images, you can input the hyperspectral image data X to be demixed. R H×W×CHere, H and W represent the height and width of the hyperspectral image data, respectively, and C represents the number of bands. Feature extraction is then performed on the hyperspectral image data using a 1×1 first convolutional layer. This 1×1 first convolutional layer upscales the output features by N times, followed by batch normalization and Leaky ReLU activation to obtain the first feature. Next, a 3×3 second convolutional layer extracts features from the first feature, maintaining the same output feature map size, followed by batch normalization and Leaky ReLU activation to obtain local spatial spectral features. Simultaneously, a third convolutional layer can be used to extract features from the local spatial spectral features, followed by batch normalization and Leaky ReLU activation to obtain downsampled local spatial spectral features.

[0062] In this embodiment of the application, the first feature can be represented as: Where X represents the hyperspectral image data to be unmixed. This indicates that the hyperspectral image data is processed by convolution through a 1×1 first convolutional layer, and BN represents batch normalization. express Activation function. Local spatial spectral features can be represented as... ,in This indicates that the first feature is processed by a 3×3 second convolutional layer, and the downsampled local spatial spectral features can be represented as follows: ,in This indicates that the local spatial spectral features are processed by convolution through a 1×1 third convolutional layer.

[0063] Please see Figure 4 The global spectral information perception network branch based on the self-attention mechanism includes a global spectral perception module and a convolutional network local spatial spectral feature transfer module. The convolutional network local spatial spectral feature transfer module determines the spatial attention weights Xp of each spectral channel based on the local spatial spectral features of the hyperspectral image data. The global spectral perception module extracts the self-attention SA between spectral channels in the hyperspectral image data, determines the value matrix Vspa after applying spatial attention based on the spatial attention weights Xp of each spectral channel and the feature matrix obtained by linearly projecting the hyperspectral image data during the self-attention extraction process, and linearly combines the value matrix Vspa with the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

[0064] Specifically, the local spatial spectral feature transfer module of the convolutional network includes a 1×1 fourth convolutional layer and a 5×5 fifth convolutional layer. This module can be used to determine the spatial attention weights Xp of each spectral channel as follows:

[0065] The fourth convolutional layer extracts local spatial spectral features from the hyperspectral image data to obtain the second feature.

[0066] The third feature is obtained by extracting features from the second feature through the fifth convolutional layer;

[0067] The third feature is reconstructed into features of multiple spectral channels, and the spatial attention weights Xp of each spectral channel are obtained.

[0068] In this embodiment of the application, the spatial attention weight of each spectral channel can be expressed as: in This indicates batch normalization processing.

[0069] The global spectral sensing module can be used to determine the global spectral feature map Xout in the following manner:

[0070] The spectral image data of each spectral channel in the hyperspectral image data is taken as a spectral data block to obtain a spectral data block sequence;

[0071] The spectral data block sequence is linearly projected using three different matrices to obtain the query matrix Q, key matrix K, and value matrix V for each spectral channel;

[0072] The query matrix Q, key matrix K, and value matrix V of each spectral channel are all divided into multiple attention heads along the spectral band dimension;

[0073] Calculate the self-attention of each attention head in each spectral channel, and determine the self-attention (SA) between spectral channels in the hyperspectral image data based on the self-attention of each attention head in each spectral channel;

[0074] The spatial attention weights Xp of each spectral channel are multiplied element-wise with the value matrices V of each spectral channel to obtain the value matrix Vspa after spatial attention is applied.

[0075] The global spectral feature map Xout is obtained by linearly combining the value matrix Vspa after spatial attention is applied with the self-attention SA between the spectral channels.

[0076] Specifically, the hyperspectral image data can be X-rayed first. R H×W×C The spectral image data of each spectral channel is taken as a spectral data block, resulting in a spectral data block sequence X. R HW×C This operation specifically involves unfolding the two-dimensional hyperspectral image data into one-dimensional data within each channel. Subsequently, the sequence of spectral data blocks is linearly projected using three different learnable matrices to generate the query matrix Q, key matrix K, and value matrix V for each spectral channel required by the attention mechanism. The query matrix Q, key matrix K, and value matrix V for each spectral channel can be represented as:

[0077] ;

[0078] Wherein, X represents a spectral data block in the spectral data block sequence. , and Both represent matrix parameters.

[0079] After linear projection, the information contained in each spectral data block in the input sequence is linearly projected into a set of vector features. The feature vectors of different spectral data blocks are used to represent their positions in the entire feature distribution space. Therefore, the relationship between spectral data blocks is described through the relationship between feature vectors. To make the features more diverse and expressive, the query matrix Q, key matrix K, and value matrix V can be divided into multiple attention heads along the spectral channel (spectral band) dimension. Each attention head has a feature vector dimension of C / N. Self-attention is calculated for each head j according to the self-attention calculation method. Then, based on the self-attention of each attention head in each spectral channel, the self-attention SA between spectral channels in the hyperspectral image data is determined. The spatial attention weight Xp of each spectral channel is then element-wise multiplied with the value matrix V of each spectral channel to obtain the value matrix Vspa after spatial attention is applied. Finally, the value matrix Vspa after spatial attention is applied is linearly combined with the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

[0080] The self-attention of each head j can be represented as: , ,in The attention weights between spectral data block i and spectral data block j in the spectral data block sequence are represented by softmax, which represents a non-linear activation function. K j This represents the j-th attention head in the key matrix K. This indicates the j-th attention head in the query matrix Q. This represents the j-th attention head in the value matrix V.

[0081] The value matrix Vspa after applying spatial attention can be represented as ,in This indicates element-wise multiplication.

[0082] In this embodiment of the application, the downsampled local spatial spectral feature Y is stacked with the global spectral feature map Xout and then decoded to obtain multiple endmember spectral signals. This may include, but is not limited to, the following steps:

[0083] The downsampled local spatial spectral features Y and the global spectral feature map Xout are stacked and then extracted through a convolutional layer.

[0084] The features extracted from the convolutional layer are activated into an abundance map using a non-linear activation function.

[0085] The abundance map was decoded into a hyperspectral image by linear mapping, resulting in multiple endmember spectral signals.

[0086] This invention employs a hyperspectral unmixing deep learning model comprising a local spatial spectral feature downsampling branch based on a convolutional neural network and a global spectral information perception network branch based on a self-attention mechanism. The local spatial spectral feature downsampling branch based on the convolutional neural network is used to extract and downsample local spatial spectral features from hyperspectral image data. The global spectral information perception network branch based on the self-attention mechanism is used to extract the self-attention (SA) between spectral channels in the hyperspectral image data, determine the spatial attention weights (Xp) of each spectral channel based on the local spatial spectral features of the hyperspectral image data, determine the value matrix (Vspa) after spatial attention is applied based on the feature matrix obtained by linear projection of the hyperspectral image data during the self-attention extraction process using the spatial attention weights (Xp) of each spectral channel and the feature matrix obtained by linear projection of the hyperspectral image data during the self-attention extraction process, and linearly combine the value matrix (Vspa) after spatial attention is applied with the self-attention (SA) between spectral channels to obtain the global spectral feature map (Xout). Thus, when demixing hyperspectral image data, the local spatial spectral features can be extracted using the local spatial spectral feature downsampling branch of the convolutional neural network, and the correlation weights between different spectral channels can be dynamically calculated using the global spectral information perception network branch based on the self-attention mechanism. The spatial attention weights from the convolutional branch are also fused to achieve global spectral modeling with spatial perception capabilities. This ensures spectral details while achieving global spectral perception, significantly improving the utilization rate of hyperspectral information, enhancing the demixing effect of spectral images, and facilitating practical applications and promotion.

[0087] Please see Figure 5 The second aspect of this application provides a hyperspectral image demixing apparatus, which includes:

[0088] The feature sampling and information perception unit is used to input the hyperspectral image data to be unmixed into a pre-trained hyperspectral unmixing deep learning model to perform local spatial spectral feature downsampling and global spectral information perception, and obtain the downsampled local spatial spectral feature Y and global spectral feature map Xout.

[0089] The stacking and decoding unit is used to stack the downsampled local spatial spectral feature Y with the global spectral feature map Xout and then decode them to obtain multiple endmember spectral signals.

[0090] The hyperspectral unmixing deep learning model includes a local spatial spectral feature downsampling branch based on a convolutional neural network and a global spectral information perception network branch based on a self-attention mechanism.

[0091] The local spatial spectral feature downsampling branch based on the convolutional neural network is used to extract and downsample the local spatial spectral features of the hyperspectral image data.

[0092] The global spectral information sensing network branch based on the self-attention mechanism is used to extract the self-attention SA between spectral channels in the hyperspectral image data, determine the spatial attention weight Xp of each spectral channel based on the local spatial spectral features of the hyperspectral image data, determine the value matrix Vspa after spatial attention is applied based on the spatial attention weight Xp of each spectral channel and the feature matrix obtained by linear projection of the hyperspectral image data during the self-attention extraction process, and linearly combine the value matrix Vspa after spatial attention with the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

[0093] The working process, working details and technical effects of the hyperspectral image demixing device provided in the second aspect of this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0094] like Figure 6 As shown, a third aspect of this application provides an electronic device, including a memory, a processor, and a transceiver that are sequentially and communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the hyperspectral image demixing method as described in the first aspect of the embodiment.

[0095] Specifically, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or last-in-first-out (FILO) memory, etc.; the processor may not be limited to microprocessors of the STM32F105 series, ARM (Advanced RISC Machines), x86 architecture processors, or processors with integrated NPU (neural-network processing units); the transceiver may be, but is not limited to, WiFi (Wireless Fidelity) wireless transceivers, Bluetooth wireless transceivers, General Packet Radio Service (GPRS) wireless transceivers, ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard), 3G transceivers, 4G transceivers, and / or 5G transceivers, etc.

[0096] This fourth aspect of the embodiment provides a computer-readable storage medium storing instructions comprising the hyperspectral image demixing method described in the first aspect of the embodiment. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the hyperspectral image demixing method as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0097] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the hyperspectral image demixing method as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0098] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for unmixing hyperspectral imagery, the method comprising: The application relates to a hyperspectral unmixing method based on a deep learning model. The hyperspectral image data to be unmixing is input into a pre-trained hyperspectral unmixing deep learning model to perform local space-spectral feature downsampling and global spectral information sensing, so as to obtain down-sampled local space-spectral features Y and global spectral feature maps Xout. The down-sampled local space-spectral features Y and the global spectral feature maps Xout are stacked and decoded to obtain a plurality of end-member spectral signals. The hyperspectral unmixing deep learning model comprises a local space-spectral feature downsampling branch based on a convolutional neural network and a global spectral information sensing network branch based on a self-attention mechanism. The local space-spectral feature downsampling branch based on the convolutional neural network is used for local space-spectral feature extraction and local space-spectral feature downsampling of the hyperspectral image data. The global spectral information sensing network branch based on the self-attention mechanism is used for extracting self-attention SA between spectral channels of the hyperspectral image data, determining spatial attention weights Xp of each spectral channel based on local space-spectral features of the hyperspectral image data, determining a value matrix Vspa after spatial attention is given based on the spatial attention weights Xp of each spectral channel and a feature matrix obtained by linear projection of the hyperspectral image data in the self-attention extraction process, and linearly combining the value matrix Vspa after spatial attention is given and the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

2. The hyperspectral image unmixing method of claim 1, wherein, The local space-spectral feature downsampling branch based on the convolutional neural network comprises a first convolutional layer with a size of 1*1, a second convolutional layer with a size of 3*3 and a third convolutional layer with a size of 1*1, the first convolutional layer is used for feature extraction of the hyperspectral image data to obtain first features, the second convolutional layer is used for feature extraction of the first features to obtain the local space-spectral features, and the third convolutional layer is used for feature extraction of the local space-spectral features to obtain the down-sampled local space-spectral features Y.

3. The hyperspectral image unmixing method of claim 1, wherein, The global spectral information sensing network branch based on the self-attention mechanism comprises a global spectral sensing module and a convolutional network local space-spectral feature transmission module. The convolutional network local space-spectral feature transmission module is used for determining the spatial attention weights Xp of each spectral channel based on the local space-spectral features of the hyperspectral image data. The global spectral sensing module is used for extracting the self-attention SA between spectral channels of the hyperspectral image data, determining the value matrix Vspa after spatial attention is given based on the spatial attention weights Xp of each spectral channel and the feature matrix obtained by linear projection of the hyperspectral image data in the self-attention extraction process, and linearly combining the value matrix Vspa after spatial attention is given and the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

4. The hyperspectral image unmixing method of claim 3, wherein, The convolutional network local space-spectral feature transmission module comprises a fourth convolutional layer with a size of 1*1 and a fifth convolutional layer with a size of 5*5, and the convolutional network local space-spectral feature transmission module is used for determining the spatial attention weights Xp of each spectral channel in the following manner: The fourth convolutional layer is used for feature extraction of the local space-spectral features of the hyperspectral image data to obtain second features. The second feature is subjected to feature extraction by the fifth convolutional layer to obtain a third feature; The third feature is reshaped into features of multiple spectral channels to obtain spatial attention weights Xp of each spectral channel.

5. The hyperspectral image unmixing method of claim 3, wherein, The global spectral perception module is configured to determine the global spectral feature map Xout in the following manner: Spectral image data in each spectral channel of the hyperspectral image data is taken as a spectral data block to obtain a sequence of spectral data blocks; The sequence of spectral data blocks is linearly projected by three different matrices to obtain a query matrix Q, a key matrix K and a value matrix V of each spectral channel; The query matrix Q, the key matrix K and the value matrix V of each spectral channel are all divided into multiple attention heads along the spectral band dimension; Self-attention of each attention head in each spectral channel is calculated, and self-attention SA between spectral channels in the hyperspectral image data is determined based on the self-attention of each attention head in each spectral channel; The spatial attention weights Xp of each spectral channel are element-wise multiplied with the value matrix V of each spectral channel to obtain a value matrix Vspa to which spatial attention is assigned; The value matrix Vspa to which spatial attention is assigned is linearly combined with the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

6. The hyperspectral image unmixing method of claim 5, wherein, The query matrix Q, the key matrix K and the value matrix V of each spectral channel are as follows: ； wherein X denotes a spectral data block of the sequence of spectral data blocks, , and each denote a matrix parameter.

7. The hyperspectral image unmixing method of claim 1, wherein, The decoded multiple endmember spectral signals include: The down-sampled local spectral features Y and the global spectral feature map Xout are stacked and subjected to feature extraction by a convolutional layer; Features after the convolutional layer feature extraction are activated into abundance maps by a nonlinear activation function; The abundance maps are decoded into hyperspectral images by linear mapping to obtain the multiple endmember spectral signals.

8. A hyperspectral image unmixing device, characterized by, The method comprises: A feature sampling and information perception unit is configured to input the hyperspectral image data to be un-mixed into a pre-trained hyperspectral un-mixing deep learning model to perform local spectral feature down-sampling and global spectral information perception to obtain down-sampled local spectral features Y and a global spectral feature map Xout; A stacking and decoding unit is configured to stack and decode the down-sampled local spectral features Y and the global spectral feature map Xout to obtain multiple endmember spectral signals; The hyperspectral un-mixing deep learning model comprises a local spectral feature down-sampling branch based on a convolutional neural network and a global spectral information perception network branch based on a self-attention mechanism; The local spectral feature down-sampling branch based on the convolutional neural network is configured to perform local spectral feature extraction and local spectral feature down-sampling on the hyperspectral image data; The global spectral information perception network branch based on self-attention mechanism is used for extracting self-attention SA between spectral channels in the hyperspectral image data, determining spatial attention weight Xp of each spectral channel based on local space-spectrum features of the hyperspectral image data, determining value matrix Vspa after spatial attention is given based on spatial attention weight Xp of each spectral channel and a feature matrix obtained by linear projection of the hyperspectral image data in the self-attention extraction process, and linearly combining the value matrix Vspa after spatial attention is given and the self-attention SA between spectral channels to obtain the global spectral feature map Xout.

9. An electronic device, comprising: The memory, the processor and the transceiver are connected in sequence and communicate with each other, the memory is used for storing a computer program, the transceiver is used for transmitting and receiving messages, and the processor is used for reading the computer program and executing the hyperspectral image unmixing method according to any one of claims 1-7.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or the instruction realizes the hyperspectral image unmixing method according to any one of claims 1-7 when executed by a computer.

Images