Self-supervised hyperspectral image anomaly detection method based on limited attention mechanism

Abstract

The present application relates to a kind of self-supervised hyperspectral image anomaly detection methods based on limited attention mechanism, comprising the following steps: obtaining the color space data of hyperspectral image;Spectral channel is selected by clustering-based spectral band selection method, and generate hyperspectral cube;Construct spatial limited self-attention layer FSAL:5 limited self-attention module FSAB is constructed;Self-supervised background spectrum reconstruction neural network is constructed;Generate hyperspectral image;Obtain stable hyperspectral background reconstruction model;Using self-supervised background spectrum reconstruction neural network, cooperate anomaly detection algorithm, carry out anomaly detection to hyperspectral data.

Description

Technical Field

[0001] This invention relates to a self-supervised hyperspectral image anomaly detection method. Background Technology

[0002] Hyperspectral imagery is a form of optical imagery with extremely high spectral resolution and an important medium for information dissemination in remote sensing. Hyperspectral imagery possesses rich spectral and detailed spatial information, giving it the potential to distinguish different materials and target appearances, making it widely applicable in fields such as environmental monitoring and resource surveys. With the increasing number of application scenarios and images, the demand for automatic detection of anomalous targets is becoming increasingly strong, making hyperspectral anomaly detection one of the research hotspots in the field. Hyperspectral anomaly detection aims to locate anomalous targets that differ significantly from background pixels in both spectral and appearance, and the anomalous target categories are not predefined during training. Under these constraints, hyperspectral anomaly detection cannot directly use mature supervised classification models. Furthermore, the presence of complex background content and other interference factors presents significant challenges to the construction of hyperspectral anomaly detection models, making anomaly detection algorithms highly valuable for research.

[0003] In recent years, many ingeniously designed hyperspectral anomaly detection algorithms have been proposed, driving progress in the field. Among them, the Reed-Xiaoli (RX) algorithm... [1] The RX algorithm is a well-known classic hyperspectral anomaly detection algorithm. It assumes that background pixels follow a Gaussian model, while anomalous targets do not. Therefore, the Mahalanobis distance between the two distributions can be used to detect anomalous targets. The RX algorithm is a simple and effective hyperspectral anomaly detection algorithm, showing good performance on many datasets, but its ability to handle complex backgrounds is poor. Subsequently, local-RX... [2] (Local Reed-Xiaoli, LRX), Kernelized-RX [3] (Kernel Reed-Xiaoli, KRX) and Weighted-RX Detector [4] Improved algorithms for anomaly detection, such as Weighted Reed-Xiaoli Detector (W-RXD), have been proposed to further enhance accuracy and robustness. RX and similar algorithms have driven the development of anomaly detection tasks from a model perspective. In addition to classical algorithms, many others have improved anomaly detection from the perspective of feature learning. Among these algorithms, sparse representation algorithms, feature transformation algorithms, and feature decomposition algorithms have been well studied. These algorithms incorporate background prior knowledge into the feature learning process, thereby identifying anomalous targets through feature-level differences. Recently, deep learning-based anomaly detection algorithms have developed rapidly. Generative anomaly detection algorithms have been extensively explored, for example, by Jiang et al.[5] A discriminative reconstruction constrained generative adversarial network (HADGAN) anomaly detection algorithm was proposed. The algorithm designed a novel generative adversarial network and proposed unsupervised discriminative reconstruction constraints. Finally, it used spatial energy and spectral distance to jointly detect anomalous targets. After that, a series of generative adversarial network-based methods were proposed. Reference [6] proposed a semi-supervised generative adversarial network model, which reconstructed the spectral information of background seed points to explore the distribution law of background spectrum. On the other hand, deep autoencoder network structures have also been well explored. Liu et al. [7] A dual-frequency autoencoder network structure was proposed, and anomalous targets in images were detected by fusing the outputs of the two networks. In addition, a series of hyperspectral anomaly detection algorithms based on generative network structures have been subsequently proposed. [8][9]

[10]

[11] The detection performance of the algorithm can be improved by designing more reasonable constraints and network structures.

[0004] Analysis reveals that due to the lack of semantic labels, effective supervised learning methods cannot be established. Previous hyperspectral anomaly detection algorithms are mostly unsupervised or semi-supervised learning methods. In the construction of hyperspectral anomaly detection models, previous methods transformed artificial prior knowledge of the background into statistical distribution models or feature learning constraints, thereby obtaining a spectral description model of the normal background and realizing the detection and localization of abnormal targets. Therefore, previous algorithms are mostly prior-driven rather than data-driven methods, lacking guidance from high-level semantic information, resulting in weak model discrimination ability. In recent years, self-supervised learning has developed rapidly, solving the problem of difficult semantic label acquisition in various application tasks. In the field of natural image analysis, scholars have proposed self-supervised image anomaly detection algorithms. These algorithms construct auxiliary learning tasks suitable for anomaly detection by mining the inherent attributes of objects in natural scenes. For example, Reference

[12] mined the structural distribution law of normal objects through content erasure and paste operations, and regarded those that violated the law as abnormal samples. Reference

[13] explored rotation and coloring tasks, and by constructing attribute transformation tasks of normal samples, the model mastered the attribute transformation relationship of normal objects, and then screened out abnormal samples that did not meet this relationship. Therefore, constructing the mapping process between attributes is the starting point for building self-supervised anomaly detection algorithms. An appropriate mapping process establishes a supervised relationship between the attributes of normal objects, realizing a self-supervised learning process. Related research has shown that self-supervised learning is an effective means of capturing high-level semantic information and has achieved good results in natural image anomaly detection tasks. It is worth noting that in addition to the structural and pose attributes frequently used in natural images, hyperspectral images also contain spectral attributes, giving them a unique advantage in constructing self-supervised learning methods. However, in the field of hyperspectral anomaly detection, there are currently no self-supervised anomaly detection algorithms, and their specific construction methods require further exploration.

[0005] In addition, background pixels occupy most of the image area in hyperspectral remote sensing images, and previous studies have also demonstrated that background pixels have low-rank characteristics. Therefore, there is a close relationship between the spectral and spatial dimensions in hyperspectral images, which also means that background features should have long-distance correlation and globality. However, due to the lack of training samples and the interference of anomalous pixels during training, most existing deep hyperspectral anomaly detection methods use pixel-level spectral reconstruction as the training objective of the network. Under this objective setting, the spatial receptive field of the network is extremely small, the model cannot grasp the intrinsic relationship between spectral and spatial information, cannot aggregate global background features in a data-driven manner, and therefore cannot learn the spatial distribution characteristics of the background. The model's utilization of spatial information mostly relies on additional background prior knowledge, which limits the applicability of the algorithm to the introduced artificial priors, resulting in limited model universality.

[0006] Related literature

[0007] [1]Reed I S,Yu X.Adaptive multiple-band CFAR detection of an opticalpattern with unknown spectral distribution[J].IEEE Transactions on AcousticsSpeech&Signal Processing,1990,38(10):1760-1770.

[0008] [2]MOLERO,J.M.,GARZON,E.M.,GARCIA,I.,et al.Analysis and Optimizationsof Global and Local Versions of the RX Algorithm for Anomaly Detection inHyperspectral Data[J].IEEE journal of selected topics in applied earthobservations and remote sensing,2013,6(2Part3):801-814.

[0009] [3]Kwon H,Nasrabadi N M.Kernel RX-algorithm:a nonlinear anomalydetector for hyperspectral imagery[J].IEEE Transactions on Geoscience&RemoteSensing,2005,43(2):388-397.

[0010] [4]Guo Q,Zhang B,Ran Q,et al.Weighted-RXD and Linear Filter-BasedRXD:Improving Background Statistics Estimation for Anomaly Detection inHyperspectral Imagery[J].IEEE Journal of Selected Topics in Applied EarthObservations&Remote Sensing,2014,7(6):2351-2366.

[0011] [5]Jiang T,Li Y,Xie W,et al.Discriminative Reconstruction ConstrainedGenerative Adversarial Network for Hyperspectral Anomaly Detection[J].IEEETransactions on Geoscience and Remote Sensing,2020,58(7):4666-4679.

[0012] [6]Jiang K,Xie W,Li Y,et al.Semisupervised Spectral Learning withGenerative Adversarial Network for Hyperspectral Anomaly Detection[J].IEEETransactions on Geoscience and Remote Sensing,2020,58(7):5224-5236.

[0013] [7]Y.Liu,W.Xie,Y.Li,et al.Dual-Frequency Autoencoder for AnomalyDetection in Transformed Hyperspectral Imagery[J].IEEE Transactions onGeoscience and Remote Sensing,2022,60():1-13,DOI:10.1109 / TGRS.2022.3152263.

[0014] [8]Xie W,Liu B,Li Y,et al.Spectral Adversarial Feature Learning forAnomaly Detection in Hyperspectral Imagery[J].IEEE Transactions on Geoscienceand Remote Sensing,2019,58(4):1-14.

[0015] [9]T.Jiang,W.Xie,Y.Li,et al.Weakly Supervised Discriminative LearningWith Spectral Constrained Generative Adversarial Network for HyperspectralAnomaly Detection[J].IEEE Transactions on Neural Networks and LearningSystems,2021,accepted,DOI:10.1109 / TNNLS.2021.3082158.

[0016]

[10] Jiang T,Li Y,Xie W,et al.Discriminative ReconstructionConstrained Generative Adversarial Network for Hyperspectral AnomalyDetection[J].IEEE Transactions on Geoscience and Remote Sensing,2020,58(7):4666-4679.

[0017]

[11] Zhong J,Xie W,Li Y,et al.Characterization of Background-AnomalySeparability with Generative Adversarial Network for Hyperspectral AnomalyDetection[J].IEEE Transactions on Geoscience and Remote Sensing,2021,59(7):6017-6028.

[0018]

[12] Li C L,Sohn K,Yoon J,et al.CutPaste:Self-Supervised Learning forAnomaly Detection and Localization[C] / / 2021 IEEE / CVF Conference on ComputerVision and Pattern Recognition (CVPR),June 19-25,2021,Online,2021:9659-9669.

[0019]

[13] Fei Y,Huang C,Cao J,et al.Attribute Restoration Framework forAnomaly Detection[J].IEEE Transactions on Multimedia,2022,24():116-127,DOI:10.1109 / TMM.2020.3046884.

[0020]

[14] Wang Q,Zhang F,Li X.Optimal clustering framework forhyperspectral band selection[J].IEEE Transactions on Geoscience and RemoteSensing,2018,56(10):5910-5922.

[0021]

[15] Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,LlionJones,Aidan N Gomez, Kaiser,and Illia Polosukhin.Attention is all youneed.In NeurIPS,2017.2

[0022]

[16] Li W,Du Q.Collaborative Representation for Hyperspectral AnomalyDetection[J].IEEE Transactions on Geoscience and Remote Sensing,2015,53(3):1463-1474.

[0023]

[17] Tao R,Zhao X,Li W,et al.Hyperspectral Anomaly Detection byFractional Fourier Entropy[J].IEEE Journal of Selected Topics in AppliedEarth Observations and Remote Sensing,2019,12(12):4920-4929.

[0024]

[18] Li S, Zhang K, Hao Q, et al. Hyperspectral Anomaly Detection with Multiscale Attribute and Edge-Preserving Filters[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(10): 1605-1609. Summary of the Invention

[0025] The present invention proposes a self-supervised hyperspectral image anomaly detection method based on a restricted attention mechanism. The proposed method conducts hyperspectral anomaly detection from a self-supervised perspective. A self-supervised mapping process from visible light to hyperspectral is designed according to the unique spectral attributes of the image, and then an internal connection between appearance attributes and spectral attributes is established, solving the problem of difficult acquisition of semantic information in hyperspectral anomaly detection. The technical solutions are as follows.

[0026] A self-supervised hyperspectral image anomaly detection method based on a restricted attention mechanism, comprising the following steps:

[0027] First, for the RGB color image corresponding to the hyperspectral image Use a data augmentation method with a k×k sliding window to obtain color space data where W and H respectively represent the number of pixels in the horizontal and vertical directions of the hyperspectral image, k represents the block size of the color cube, i∈1…n represents the cube index number, and n is the number of color cubes. The block size needs to satisfy k<W and k<H;

[0028] Second, for the original hyperspectral image data where C represents the total number of spectral bands of the hyperspectral image data; select the top n most representative spectral channels through a spectral band selection method based on clustering, and generate a hyperspectral cube corresponding to the color space data through a sliding window operation s

[0029] Third step: Construct a spatially restricted self-attention layer FSAL. In this layer, construct a restricted spatial self-attention mechanism, model the mutual relationship of different position features in the spatial dimension, and generate corresponding spatial features according to this relationship The method is as follows:

[0030] (1) The color space data generated in the first step After being extracted by the convolutional layer, generate a three-dimensional feature matrix And use it as the input of FSAL. First, expand this three-dimensional feature matrix into a vector group X FSA =[x​​F1 ,x F2 ...,x Fm ]′, where x Fi (i∈[1,m]) are row vectors and m=k×k, each row vector x Fi Contains n s Column data, that is, to obtain m n s 3D eigenvectors;

[0031] (2) Calculate the three linear projection matrices of the vector group in step (1) to obtain the query, key, and value required for calculating self-attention, Q = XW Q K = XW K V = XW V ,in, These represent the query, key, and value matrices, respectively. It is an optimizable linear transformation matrix;

[0032] (3) Cluster the query matrix from step (2) to obtain n c Each category and its corresponding cluster center vector Mining background features;

[0033] (4) Obtain the augmented query matrix and augmented key matrix by row concatenation. Calculate the distribution of attention Among them, K' T This represents the transpose of the augmented bond matrix. It is a scaling factor, representing the distribution of attention, A. spa middle Represents the global attention relationship between features, where A spa_t =A spa [1:m;1:m]; This represents the attention relationship between features and cluster classes, where A spa_c =A spa [1:m;m+1:m+n] c ];

[0034] (5) Design a confidence-based learning strategy, the method is as follows: In A spa_c The probability p of each pixel belonging to the background is calculated using a clustering algorithm in the matrix. i And based on the p of all pixels i The first r percentages are selected to generate a background weight map M; the local energy of the attention distribution is measured by calculating the local mean of the attention scores, and a local attention energy map E is constructed. i,j ; Calculate the weight of a pixel belonging to the foreground in Let M be the conjugate matrix of the background weight map. The larger the pixel value, the greater the probability that the pixel belongs to the foreground.

[0035] (6) Take A obtained in step three spa_t and A spa_c Substitution In the process, spatial features are obtained. in For A spa_c The vector in the i-th row of the matrix, For A spa_c In the matrix, the vector in the j-th row, I, represents the index of a feature in its respective cluster feature space. This indicates the correlation between image position i and image position j, v j Let j be the value matrix vector;

[0036] Step 4: Using the Spatially Constrained Self-Attention Layer (FSAL) proposed in Step 3, construct five constrained self-attention modules (FSABs): FSAB1, FSAB2, FSAB3, FSAB4, and FSAB5. The feature matrix of each constrained self-attention module (FSAB) is extracted through a convolutional layer. The input is fed into a spatially restricted self-attention layer (FSAL); after processing by the FSAL layer, spatial features are obtained. Spatial features are input into the spectral spatial self-attention layer (SAL) to obtain the final aggregated features. Among them, the spectral spatial self-attention layer (SAL) uses the MSA module in the transform neural network to extract the global correlation of hyperspectral data;

[0037] The SAL processing method is consistent for each Restricted Self-Attention Module (FSAB), with differences only in the processing of convolutional layers. Specifically, in FSAB1, the convolutional layers do not change the feature space size. In FSAB2 and FSAB3, the convolutional layers are used for downsampling the input features, compressing the feature space to half its length and width, and doubling the number of channels. In FSAB4 and FSAB5, the opposite is true: they are used for upsampling the input features, making the feature space twice the size of the input, and halving the number of channels.

[0038] The fifth step involves constructing a self-supervised background spectral reconstruction neural network using the restricted self-attention module FSAB built in the fourth step. The self-supervised background spectral reconstruction neural network adopts an autoencoder network structure, which consists of an encoder composed of two restricted self-attention modules, FSAB1 and FSAB2, and a decoder composed of two restricted self-attention modules, FSAB4 and FSAB5, connected in series. A bottleneck layer is used to form the latent space, which contains a third restricted self-attention module, FSAB3.

[0039] Step 6: Transfer color space data The data is input into the self-supervised background spectral reconstruction neural network constructed in step 5; after the data passes through the first restricted self-attention module FSAB1, feature learning and feature smoothing are performed through convolutional layers, and the number of channels is expanded to n. s Become In the second restricted self-attention module FSAB2, spatial downsampling is implemented to obtain the image feature map. In the third restricted self-attention module FSAB3, spatial downsampling is implemented to obtain the image feature map. Spatial upsampling is implemented and image feature maps are obtained in the fourth restricted self-attention module FSAB4. In the fifth restricted self-attention module FSAB5, spatial upsampling is implemented to obtain the image feature map. Then, the image features After performing convolutional smoothing and normalization operations, the hyperspectral image is finally generated.

[0040] Step 7: During training, the recovery quality of the confidence background region is defined as the loss function, and the root mean square error is used to measure the hyperspectral image generated during training. and true data The quality difference between them; and use this index to guide the training of the hyperspectral background reconstruction model. When the loss function shows a stable convergence state, the training process ends, thereby obtaining a stable hyperspectral background reconstruction model.

[0041] The eighth step involves using the self-supervised background spectral reconstruction neural network proposed in the previous seven steps, combined with an anomaly detection algorithm, to perform anomaly detection on the hyperspectral data.

[0042] Furthermore, the processing method for the spectral spatial self-attention layer (SAL) is as follows: The output features of the spatially confined self-attention layer (FSAL) are processed... Consider as k×k n s 3D spectral features, for each n s The spectral feature is treated as a label, and self-interest is calculated along the spectral dimension. It is then expanded along the spectral dimension to form the spectral channel feature vector. The final aggregated features are obtained by using the MSA module, along with normalization operations and residual linking.

[0043] This invention introduces self-supervised learning into hyperspectral anomaly detection. It utilizes the unique spectral properties of hyperspectral images to construct a self-supervised mapping process from visible light to the hyperspectral spectrum, establishing an intrinsic connection between appearance and spectral attributes. This self-supervised learning approach enables the acquisition and utilization of high-level semantic information, solving the problem of insufficient semantic information in previous hyperspectral anomaly detection algorithms. Furthermore, this invention constructs and designs a spatially constrained self-attention mechanism and proposes a confidence spectral learning strategy, enabling the network to better capture high-level semantic information of normal targets. This results in a stronger ability to represent normal targets and produces differentiated recovery effects for normal and anomalous targets. The performance of the algorithm is validated on six internationally available datasets, demonstrating that the proposed algorithm outperforms current mainstream anomaly detection methods. Attached Figure Description

[0044] Appendix Figure 1 Overall framework of anomaly detection algorithms

[0045] Appendix Figure 2 Schematic diagram of spatially constrained attention layer structure

[0046] Appendix Figure 3 Anomaly detection results on six datasets Detailed Implementation

[0047] This invention introduces transform neural networks into the field of hyperspectral anomaly detection. A self-supervised background spectral reconstruction neural network is constructed, which can simultaneously extract spatial and spectral information from images and establish a global dependency relationship between the spectral and spatial dimensions, achieving an end-to-end anomaly detection process. This invention designs a constrained spatial self-attention mechanism and proposes a confidence-based background learning strategy. This learning strategy can mine the cluster structure of the background spectrum in a data-driven manner, and proposes confidence-based background weight calculation methods and confidence-based foreground weight calculation methods based on the cluster structure. The confidence-based background weight can prevent interference from anomalous targets during the training process; the confidence-based foreground weight, on the other hand, focuses on the spatial distribution characteristics of the background and targets, increasing the discriminability between the background and anomalous targets, thereby improving the performance of the anomaly detection model. The overall framework of this invention is attached. Figure 1 As shown, it mainly consists of three parts: color space data construction, self-supervised learning objective construction, and backbone network (self-supervised background spectrum reconstruction neural network).

[0048] The self-supervised hyperspectral image anomaly detection method based on the restricted attention mechanism of the present invention includes the following steps:

[0049] The first step is to analyze the RGB color images corresponding to the hyperspectral images. Color space data was obtained using a k×k sliding window data augmentation technique. To improve the ability of the entire network to mine spectral and spatial information, where W and H represent the number of pixels in the horizontal and vertical directions of the spectral image respectively, k represents the block size of the color cube, i ∈ 1…n represents the cube index number, and n is the number of color cubes. The block size needs to satisfy k < W and k < H.

[0050] In the second step, for the original hyperspectral image data where C represents the total number of bands of the hyperspectral image data. Through the spectral band selection method (OCF) based on the optimal clustering framework

[14] select the top n s spectral channels with the most representative ability, and generate the hyperspectral cube corresponding to X i through a sliding window operation to guide the network to complete the self-supervised learning process, avoiding the interference of redundant information on the learning effect of the model, and enabling the model to better master the mapping relationship of normal samples from the appearance space and the spectral space. Among them, OCF selects spectral bands with lower consistency and higher discrimination according to the results of clustering and sorting, and can find the optimal solution for the hyperspectral data set under reasonable constraints. And OCF is a general framework, that is, once different types of objective functions meet a specific form, they can be optimized through the same program.

[0051] In the third step: Construct a spatially constrained self-attention layer (FSAL), and construct a constrained spatial self-attention mechanism in this layer to model the mutual relationship of different position features in the spatial dimension, and generate corresponding spatial features according to this relationship Specifically:

[0052] (1) Pass the color space data generated in the first step through a convolutional layer to extract and generate a three-dimensional feature matrix and use it as the input of FSAL. First, expand this three-dimensional feature matrix into a vector group X FSA =[x F1 ,x F2 ···,x Fm ′, where x Fi (i ∈ [1, m]) is a row vector and m = k × k. Each row vector x Fi contains n s column data, that is, obtain m n s -dimensional feature vectors.

[0053] (2) Calculate the three groups of linear projection matrices of the vector group in step (1) of the third step to obtain the query (Query), key (Key), and value (Value) required for calculating self-attention, Q = XW Q , K = XW K , V = XW V,in, These represent the query, key, and value matrices, respectively. It is an optimizable linear transformation matrix.

[0054] (3) Cluster the query matrix from step (2) to obtain n c Each category and its corresponding cluster center vector Discover background features.

[0055] (4) Obtain the augmented query matrix and augmented key matrix by row concatenation. And calculate the distribution of attention. Among them, K' T This represents the transpose of the augmented bond matrix. This is the scaling factor. The distribution of attention, A. spa middle Represents the global attention relationship between features, where A spa_t =A spa [1:m;1:m]. This represents the attention relationship between features and cluster classes, where A spa_c =A spa [1:m;m+1:m+n] c ].

[0056] (5) To reduce the impact of abnormal targets on the optimization process, a confidence background learning strategy is designed. Specifically: in A spa_c The probability p of each pixel belonging to the background is calculated using a clustering algorithm in the matrix. i And based on the p of all pixels i The first r percentages are selected to generate a background weight map M; the local energy of the attention distribution is measured by calculating the local mean of the attention scores, and a local attention energy map E is constructed. i,j Finally, the weights of pixels belonging to the foreground are calculated. in Let M be the conjugate matrix of the background weight map. The larger the pixel value, the greater the probability that the pixel belongs to the foreground.

[0057] (6) Take A obtained in step (4) and (5) spa_t and A spa_c Substitution In the process, spatial features are obtained. in For A spa_c The vector in the i-th row of the matrix, For A spa_c In the matrix, the vector in the j-th row, I, represents the index of a feature in its respective cluster feature space. This indicates the correlation between image position i and image position j, vj Let be the j-th value matrix vector.

[0058] Step 4: Using the Spatially Constrained Self-Attention Layer (FSAL) proposed in Step 3, construct five Constrained Self-Attention Modules (FSABs): FSAB1, FSAB2, FSAB3, FSAB4, and FSAB5. In FSAB1, the convolutional layers do not change the feature space size; they only serve the functions of feature learning and smoothing. The feature matrix extracted after the convolutional layers... Input is fed into FSAL. After processing by FSAL, spatial features are obtained. Spatial features are input into a spectral spatial self-attention layer (SAL) to obtain the final aggregated features. Among them, the spectral spatial self-attention layer (SAL) utilizes the Multi-head Self Attention (MSA) module in the transform neural network.

[15] It excels at capturing nonlocal similarity and long-range dependencies, extracting global correlations from hyperspectral data. It utilizes the output features of spatially constrained attention units. Consider as k×k n s The spectral features are treated as a label, and self-interest is calculated along the spectral dimension. Specifically, the spectral features are expanded along the spectral dimension to form a spectral channel feature vector. The final aggregated features are obtained by using the MSA module, along with normalization operations and residual linking. In the other FSABs, the processing of FSAL and SAL is consistent with that in FSAB1, differing only in the convolutional layer processing. Specifically, in FSAB2 and FSAB3, the convolutional layers are used for input feature downsampling, compressing the feature space's width and height to half and doubling the number of channels; FSAB4 and FSAB5, conversely, are used for input feature upsampling, doubling the feature space's width and height and halving the number of channels. As data flows sequentially through these five FSABs, its size changes sequentially as follows:

[0059] The fifth step involves constructing a self-supervised background spectral reconstruction neural network using the restricted self-attention module FSAB built in the fourth step. This network employs an autoencoder network structure. Specifically, the network structure consists of an encoder composed of two restricted self-attention modules, FSAB1 and FSAB2, cascaded together; a decoder composed of two restricted self-attention modules, FSAB4 and FSAB5, cascaded together; and a bottleneck layer forming the latent space, which contains a third restricted self-attention module, FSAB3.

[0060] Step 6: Transfer color space data The data is input into the self-supervised background spectrum reconstruction neural network constructed in step 5. After the data passes through the first restricted self-attention module FSAB1, it undergoes feature learning and smoothing through convolutional layers, and its channel number is expanded to n. s Become In the second restricted self-attention module FSAB2, spatial downsampling is implemented to obtain the image feature map. In the third restricted self-attention module FSAB3, spatial downsampling is implemented to obtain the image feature map. Spatial upsampling is implemented and image feature maps are obtained in the fourth restricted self-attention module (FSAB4). In the fifth restricted self-attention module (FSAB5), spatial upsampling is implemented to obtain the image feature map. Then, the image features After performing convolutional smoothing and normalization operations, the hyperspectral image is finally generated.

[0061] Step 7: During training, the recovery quality of the confidence background region is defined as the loss function, and the root mean square error is used to measure the quality of the images generated during training. and true data The quality difference between the two is used to guide the training of the hyperspectral background reconstruction model. When the loss function shows a stable convergence state, the training process ends, thus obtaining a stable hyperspectral background reconstruction model.

[0062] Step 8: Using the self-supervised background spectral reconstruction neural network proposed in the previous seven steps, and in conjunction with an anomaly detection algorithm, anomaly detection is performed on the hyperspectral data. Specifically:

[0063] (1) The hyperspectral image data to be detected is segmented into non-intersecting spatial cube blocks. The background confidence strategy proposed in step 3 is used to analyze the hyperspectral data to be detected, and the weight F of each pixel belonging to the foreground is calculated. i,j .

[0064] (2) The hyperspectral data to be detected S i,j The RGB data is input into the detection network, and the corresponding hyperspectral reconstruction S is performed using the hyperspectral background reconstruction model obtained in steps five and six. i,j The reconstruction results are compared with the hyperspectral data to be detected, S. i,j Compare and calculate the reconstruction difference S i,j -S i,j .

[0065] (3) Obtain the anomaly score by using the spatial distribution characteristics calculated in step 8 (1) and the reconstruction difference calculated in step 8 (2), and then calculate the anomaly score Ano(S). i,j )=||S i,j -Si,j ||2 + α·F i,j to determine whether the image is abnormal, where α is a weight coefficient used to balance the relationship between the two terms. The calculated anomaly score is finally obtained after smoothing and non-linear (Sigmoid) mapping.

[0066] The experimental data in this invention mainly comes from 6 public datasets (Gulfport, Pavia, Texas Coast - 2, Los Angeles - 1, San Diego - 1, and San Diego - 2). To make the technical solution of this invention clearer, the following further describes the specific implementation manner of this invention.

[0067] The first step: Construction of color space data

[0068] For the RGB color image corresponding to the hyperspectral image Use the data enhancement method of a k×k sliding window to obtain color space data to improve the ability of the entire network to mine spectral information and spatial information, where W and H respectively represent the number of pixels in the horizontal and vertical directions of the spectral image, k represents the block size of the color cube, i ∈ 1…n represents the cube index number, and n is the number of color cubes. The block size needs to satisfy k < W and k < H.

[0069] The second step: Construction of self-supervised learning objectives

[0070] For the original hyperspectral image data where C represents the total number of spectral bands of the hyperspectral image data. Through the spectral band selection method (OCF) based on the optimal clustering framework

[14] select the top n s most representative spectral channels, and generate a hyperspectral cube corresponding to X i through a sliding window operation to guide the network to complete the self-supervised learning process, avoiding the interference of redundant information on the model learning effect, and enabling the model to better master the mapping relationship of normal samples from the appearance space and the spectral space. Among them, OCF selects spectral bands with lower consistency and higher discrimination based on the results of clustering and sorting, and can find the optimal solution for the hyperspectral dataset under reasonable constraints. And OCF is a general framework, that is, once different types of objective functions meet a specific form, they can be optimized through the same program.

[0071] The third step: Construction of a spatially constrained self-attention layer (FSAL)

[0072] This invention constructs a Spatial Constrained Self-Attention Layer (FSAL), in which a constrained spatial self-attention mechanism is built. By modeling the interrelationships of features at different locations along the spatial dimension, corresponding spatial aggregation features are generated. The composition structure of the Spatial Constrained Self-Attention Layer is as follows: Figure 2 As shown. The specific method is as follows:

[0073] (1) The color space data generated in the first step A three-dimensional feature matrix is ​​generated after convolutional layer extraction. And using this as input to FSAL, the three-dimensional feature matrix is ​​first expanded into a vector group X. FSA =[x F1 ,x F2 ...,x Fm ]′, where x Fi (i∈[1,m]) are row vectors and m=k×k, each row vector x Fi Contains n s Column data, that is, to obtain m n s 3D eigenvectors.

[0074] (2) Calculate the three linear projection matrices of the vector group in step (1) to obtain the query, key, and value required for calculating self-attention. The calculation method is shown in formula (1):

[0075] Q = XW Q K = XW K V = XW V (1)

[0076] in, These represent the query, key, and value matrices, respectively. It is an optimizable linear transformation matrix.

[0077] (3) To utilize the low-rank characteristic of the normal background and mine background features, the query matrix Q from step (2) is clustered to obtain n c Each category and its corresponding cluster center vector This enables the mining of background features.

[0078] (4) After obtaining the cluster centers, the augmented query matrix and augmented key matrix are obtained by row concatenation. And calculate the distribution of attention A. spa As shown in formula (2):

[0079]

[0080] Among them, A spa The value of K' is between [0,1]. TThis represents the transpose of the augmented bond matrix. This is the scaling factor. The distribution of attention, A. spa middle Represents the global attention relationship between features, where A spa_t =A spa [1:m;1:m]. This represents the attention relationship between features and cluster classes, where A spa_c =A spa [1:m;m+1:m+n c ].

[0081] (5) To reduce the impact of abnormal targets on the optimization process, a confidence background learning strategy is designed. Specifically: in A spa_c The probability p of each pixel belonging to the background is calculated using a clustering algorithm in the matrix. i And based on the p of all pixels i The first r percentages are selected to generate a background weight map M. In M, pixels with a value of "0" are considered foreground regions, and the remaining pixels are considered background regions. The closer a pixel value is to "1", the higher its probability of belonging to the background. The local energy of the attention distribution is measured by calculating the local mean of the attention scores, thus constructing a local attention energy map E. i,j As shown in formula (3).

[0082] Based on this, the weight of a pixel belonging to the foreground is calculated. in Let M be the conjugate matrix of the background weight map. The larger the pixel value, the greater the probability that the pixel belongs to the foreground.

[0083]

[0084] Where z∈[i±w,j±w] represents the spatial local neighborhood of position (i,j), and the neighborhood contains a total of N W 1 pixel This represents the correlation between the pixel at image position (i,j) and the neighboring pixels.

[0085] (6) Take A obtained in step (4) and (5) spa_t and A spa_c Substitution In the process, spatial features are obtained. in For A spa_c The vector in the i-th row of the matrix, For A spa_c In the matrix, the vector in the j-th row, I, represents the index of a feature in its respective cluster feature space. This indicates the correlation between image position i and image position j, v jLet be the j-th value matrix vector.

[0086] Step 4: Construct Restricted Self-Attention Modules (FSAB)

[0087] Using the Spatially Restricted Self-Attention Layer (FSAL) proposed in the third step, five restricted self-attention modules (FSABs) are constructed: FSAB1, FSAB2, FSAB3, FSAB4, and FSAB5. In FSAB1, the convolutional layers do not change the feature space size; they only serve the functions of feature learning and feature smoothing. The feature matrix extracted after the convolutional layers... Input is fed into FSAL. After processing by FSAL, spatial features are obtained. Spatial features are input into a spectral spatial self-attention layer (SAL). The spectral spatial self-attention layer (SAL) utilizes the Multi-head Self-Attention (MSA) module in a transform neural network.

[15] It excels at capturing nonlocal similarity and long-range dependencies, extracting global correlations from hyperspectral data. It utilizes the output features of spatially constrained attention units. Consider as k×k n s The spectral features are treated as a label, and self-interest is calculated along the spectral dimension. Specifically, the spectral features are expanded along the spectral dimension to form a spectral channel feature vector. The final aggregated features are obtained by using the MSA module, along with normalization operations and residual linking. In the other FSABs, the processing of FSAL and SAL is consistent with that in FSAB1, differing only in the convolutional layer processing. Specifically, in FSAB2 and FSAB3, the convolutional layers are used for input feature downsampling, compressing the feature space's width and height to half and doubling the number of channels; FSAB4 and FSAB5, conversely, are used for input feature upsampling, doubling the feature space's width and height and halving the number of channels. As data flows sequentially through these five FSABs, its size changes sequentially as follows:

[0088] Step 5: Construct a self-supervised background spectrum reconstruction neural network

[0089] A self-supervised background spectral reconstruction neural network is constructed using the restricted self-attention module FSAB built in step four. This network adopts an autoencoder network structure. Specifically, the network structure is as follows: two restricted self-attention modules, FSAB1 and FSAB2, are cascaded to form an encoder, and two restricted self-attention modules, FSAB4 and FSAB5, are cascaded to form a decoder. A bottleneck layer is used to form the latent space, which contains a third restricted self-attention module, FSAB3.

[0090] Step 6: Network Training

[0091] Color space data The data is input into the self-supervised background spectrum reconstruction neural network constructed in step 5. After the data passes through the first restricted self-attention module FSAB1, it undergoes feature learning and smoothing through convolutional layers, and its channel number is expanded to n. s Become In the second restricted self-attention module FSAB2, spatial downsampling is implemented to obtain the image feature map. In the third restricted self-attention module FSAB3, spatial downsampling is implemented to obtain the image feature map. Spatial upsampling is implemented and image feature maps are obtained in the fourth restricted self-attention module (FSAB4). In the fifth restricted self-attention module (FSAB5), spatial upsampling is implemented to obtain the image feature map. Then, the image features After performing convolutional smoothing and normalization operations, the hyperspectral image is finally generated.

[0092] Step 7: Obtain a stable hyperspectral background reconstruction model

[0093] During training, the recovery quality of the confidence background region is defined as the loss function, and the root mean square error is used to measure the quality of the generated images during training. and true data The quality difference between the two is used to guide the training of the hyperspectral background reconstruction model. When the loss function shows a stable convergence state, the training process ends, thus obtaining a stable hyperspectral background reconstruction model.

[0094] Step 8: Anomaly Detection

[0095] The self-supervised background spectral reconstruction neural network proposed in the first seven steps is used in conjunction with an anomaly detection algorithm to detect anomalies in hyperspectral data. Specifically:

[0096] (1) The hyperspectral image data to be detected is segmented into non-intersecting spatial cube blocks. The background confidence strategy proposed in step 3 is used to analyze the hyperspectral data to be detected, and the weight F of each pixel belonging to the foreground is calculated. i,j .

[0097] (2) The hyperspectral data to be detected S i,j The RGB data is input into the detection network, and the corresponding hyperspectral reconstruction S is performed using the hyperspectral background reconstruction model obtained in steps five and six. i,j The reconstruction results are compared with the hyperspectral data to be detected, S. i,j Compare and calculate the reconstruction difference S i,j -S i,j .

[0098] (3) Obtain the anomaly score by using the spatial distribution characteristics calculated in step 8 (1) and the reconstruction difference calculated in step 8 (2), and then calculate the anomaly score Ano(S). i,j )=||S i,j -S i,j ||2+α·F i,j The magnitude of the anomaly score determines whether an image is abnormal, where α is a weighting coefficient used to balance the relationship between the two terms. The calculated anomaly score is then smoothed and mapped using a sigmoid algorithm to obtain the final result.

[0099] Experimental Analysis

[0100] The method proposed in this invention was developed in a workstation environment with an Intel Core i7-10700K CPU, an NVIDIA 2080Ti GPU, and 32GB of memory. To ensure algorithm consistency, the color image data for each dataset was generated using the Hyperspectral Imaging Library in the Image Processing Toolbox of MATLAB 2022. The algorithm was implemented in Python 3.8 and based on the PyTorch deep learning framework (PyTorch 1.8). During training, the batch size was set to 4, the number of iterations to 200, and the initial learning rate to 4 × 10⁻⁶. -4 The cosine annealing learning strategy was used during the process, with a minimum learning rate set to 1×10⁻⁶. -6 .

[0101] To verify the effectiveness and advancement of the proposed algorithm, a systematic comparative experiment was conducted on six publicly available datasets (Gulfport, Pavia, Texas Coast-2, Los Angeles-1, San Diego-1, and San Diego-2). The experiments compared classic algorithms and cutting-edge achievements in the field of hyperspectral anomaly detection in recent years, specifically the Reed-Xiaoli algorithm (RX) based on global statistics. [1] The Local Reed-Xiaoli algorithm (LRX) based on local statistics. [2] An anomaly detection algorithm based on collaborative representation (CRD)

[16] An anomaly detection algorithm based on fractional Fourier entropy (FrFE)

[17] An anomaly detection algorithm based on attributes and edge-preserving filtering (AED)

[18] A discriminative reconstruction constrained generative adversarial network for hyperspectral anomaly detection algorithm (HADGAN) [5] A semi-supervised spectral learning algorithm for hyperspectral anomaly detection based on generative adversarial networks (WGAN). [6] A hyperspectral anomaly detection algorithm based on the characterization of background-anomaly separability with generative adversarial network (BASGAN).

[11] Among them, RX, LRX, and AED are representative algorithms of statistical modeling methods. The RX algorithm is a classic algorithm based on global background distribution statistics and performs well in scenes with simple background distribution. The LRX algorithm is an improvement on the RX algorithm and is suitable for anomaly detection in local scenes. The AED algorithm is an efficient anomaly detection algorithm based on spatial attributes and has excellent detection performance on multiple real hyperspectral datasets. CRD and FrFE are classic algorithms in discriminative representation methods. The CRD algorithm is an algorithm based on cooperative representation. This algorithm assumes that each pixel in the background can be represented as a linear combination of its spatial neighborhood pixels, while anomalies do not have this representational characteristic. Based on this assumption, this algorithm can effectively detect anomalous targets in complex environments. The FrFE algorithm introduces fractional Fourier transform and has signal enhancement capabilities, making it a highly competitive hyperspectral anomaly detection algorithm. HADGAN, WGAN, and BASGAN represent cutting-edge achievements in deep learning anomaly detection algorithms. The HADGAN algorithm focuses on reconstructing the background to improve detection performance by introducing adversarial learning in the latent space and conditional constraints. The WGAN algorithm eliminates the requirement for distribution assumptions in the background distribution estimation process, effectively improving the accuracy of background modeling and detection performance. The BASGAN algorithm introduces latent representation and image reconstruction adversarial loss, and adds pseudo-anomaly samples to the training to improve the separability between the background and anomalies.

[0102] Table 1 shows the evaluation results of this invention and eight comparative algorithms on six public datasets. The values ​​in the table are the AUC values ​​of each algorithm, and the performance index of the best algorithm is shown in bold. Through comparison, it can be found that this invention achieves good detection results on the six datasets and can effectively identify anomalies in hyperspectral images. Specifically, for the Gulfport dataset, the AUC of this invention reaches 99.18%, the highest detection accuracy among all algorithms. For the Pavia dataset, as shown in Table 1, the top three algorithms are the algorithm proposed in this invention, WGAN, and AED, with AUC indices of 98.27%, 97.96%, and 97.92%, respectively. The first-ranked algorithm improved the AUC by 0.31% compared to the second-ranked algorithm, while the second-ranked algorithm improved by 0.04% compared to the third-ranked algorithm. That is, the performance improvement of this algorithm compared to the second-ranked algorithm is 7.75 times that of the second-ranked algorithm compared to the third-ranked algorithm. For the Texas Coast-2 dataset, this invention achieves the third-highest AUC. For the Los Angles-1 dataset, this invention achieves the best AUC. Compared to other algorithms, the detection rate is improved by an average of 9.60%. For the San Diego-1 dataset, this invention achieves the best detection accuracy on San Diego-1, with an AUC of 99.62%, the highest among all algorithms. The detection accuracy is 0.21% higher than the second-best AED algorithm and 11.29% higher than the lowest-performing LRX algorithm. For the San Diego-2 dataset, this invention achieves the best detection accuracy on San Diego-2, with an AUC of 99.85%, the highest among all algorithms. The detection accuracy is improved by an average of 1.74% compared to other algorithms. The combined performance on six datasets demonstrates the excellent detection performance of this invention, proving its effectiveness and advancement.

[0103] Table 1. AUC values ​​of nine anomaly detection algorithms on six datasets.

[0104]

[0105] Figure 3 The diagrams show the anomaly detection results of this invention in six datasets, demonstrating that anomalies are accurately located.

Claims

1. A self-supervised hyperspectral image anomaly detection method based on a restricted attention mechanism, comprising the following steps: First step, for the RGB color image corresponding to the hyperspectral image Use the data augmentation method of a k×k sliding window to obtain color space data Where W and H respectively represent the number of pixels in the horizontal and vertical directions of the hyperspectral image, k represents the block size of the color cube, i∈1…n represents the cube index number, n is the number of color cubes, and the block size needs to satisfy k<W and k<H; The second step involves analyzing the raw hyperspectral image data. Where C represents the total number of bands in the hyperspectral image data; the top n bands with the best representativeness are selected using a cluster-based spectral band selection method. s Each spectral channel is used to generate color space data through a sliding window operation. The corresponding hyperspectral cube Step 3: Construct a Spatial Constrained Self-Attention Layer (FSAL). In this layer, a constrained spatial self-attention mechanism is built to model the relationships between features at different locations in the spatial dimension, and corresponding spatial features are generated based on these relationships. The method is as follows: (1) The color space data generated in the first step A three-dimensional feature matrix is ​​generated after convolutional layer extraction. And using this as input to FSAL, the three-dimensional feature matrix is ​​first expanded into a vector group X. FSA =[x F1 ,x F2 …,x Fm ]′, where x Fi (i∈[1,m]) are row vectors and m=k×k, each row vector x Fi Contains n s Column data, that is, to obtain m n s 3D eigenvectors; (2) Calculate the three sets of linear projection matrices of the vector group in step (1) to obtain the query, key, and value required for calculating self-attention, Q = XW Q K = XW K V = XW V ,in, These represent the query, key, and value matrices, respectively. It is an optimizable linear transformation matrix; (3) Cluster the query matrix from step (2) to obtain n c Each category and its corresponding cluster center vector Mining background features; (4) Obtain the augmented query matrix and augmented key matrix by row concatenation. Calculate the distribution of attention in, This represents the transpose of the augmented bond matrix. It is a scaling factor, representing the distribution of attention, A. spa middle Represents the global attention relationship between features, where A spa_t =A spa [1:m;1:m]; This represents the attention relationship between features and cluster classes, where A spa_c =A spa [1:m;m+1:m+n c ]; (5) Design a confidence-based learning strategy, the method is as follows: In A spa_c The probability p of each pixel belonging to the background is calculated using a clustering algorithm in the matrix. i And based on the p of all pixels i The first r percentages are selected to generate a background weight map M; the local energy of the attention distribution is measured by calculating the local mean of the attention scores, and a local attention energy map E is constructed. i,j ; Calculate the weight of a pixel belonging to the foreground in Let M be the conjugate matrix of the background weight map. The larger the pixel value, the greater the probability that the pixel belongs to the foreground. (6) Take A obtained in step 3 spa_t and A spa_c Substitution In the process, spatial features are obtained. in For A spa_c The vector in the i-th row of the matrix, For A spa_c In the matrix, the vector in the j-th row, I, represents the index of a feature in its respective cluster feature space. This indicates the correlation between image position i and image position j, v j Let j be the value matrix vector; Step 4: Using the Spatially Constrained Self-Attention Layer (FSAL) proposed in Step 3, construct five constrained self-attention modules (FSABs): FSAB1, FSAB2, FSAB3, FSAB4, and FSAB5. The feature matrix of each constrained self-attention module (FSAB) is extracted through a convolutional layer. The input is fed into a spatially restricted self-attention layer (FSAL); after processing by the FSAL layer, spatial features are obtained. Spatial features are input into the spectral spatial self-attention layer (SAL) to obtain the final aggregated features. Among them, the spectral spatial self-attention layer (SAL) uses the MSA module in the transform neural network to extract the global correlation of hyperspectral data; The SAL processing method is consistent for each Restricted Self-Attention Module (FSAB), with differences only in the processing of convolutional layers. Specifically, in FSAB1, the convolutional layers do not change the feature space size. In FSAB2 and FSAB3, the convolutional layers are used for downsampling the input features, compressing the feature space to half its length and width, and doubling the number of channels. In FSAB4 and FSAB5, the opposite is true: they are used for upsampling the input features, making the feature space twice the size of the input, and halving the number of channels. The fifth step involves constructing a self-supervised background spectral reconstruction neural network using the restricted self-attention module FSAB built in the fourth step. The self-supervised background spectral reconstruction neural network adopts an autoencoder network structure, which consists of an encoder composed of two restricted self-attention modules, FSAB1 and FSAB2, and a decoder composed of two restricted self-attention modules, FSAB4 and FSAB5, connected in series. A bottleneck layer is used to form the latent space, which contains a third restricted self-attention module, FSAB3. Step 6: Transfer color space data The data is input into the self-supervised background spectral reconstruction neural network constructed in step 5; after the data passes through the first restricted self-attention module FSAB1, feature learning and feature smoothing are performed through convolutional layers, and the number of channels is expanded to n. s Become In the second restricted self-attention module FSAB2, spatial downsampling is implemented to obtain the image feature map. In the third restricted self-attention module FSAB3, spatial downsampling is implemented to obtain the image feature map. Spatial upsampling is implemented and image feature maps are obtained in the fourth restricted self-attention module FSAB4. In the fifth restricted self-attention module FSAB5, spatial upsampling is implemented to obtain the image feature map. Then, the image features After performing convolutional smoothing and normalization operations, the hyperspectral image is finally generated. Step 7: During training, the recovery quality of the confidence background region is defined as the loss function, and the root mean square error is used to measure the hyperspectral image generated during training. and true data The quality difference between them; and use this index to guide the training of the hyperspectral background reconstruction model. When the loss function shows a stable convergence state, the training process ends, thereby obtaining a stable hyperspectral background reconstruction model. The eighth step involves using the self-supervised background spectral reconstruction neural network proposed in the previous seven steps, combined with an anomaly detection algorithm, to perform anomaly detection on the hyperspectral data.

2. The detection method according to claim 1, characterized in that, The processing method for the spectral spatial self-attention layer (SAL) is as follows: The output features of the spatially confined self-attention layer (FSAL) are processed... Consider as k×k n s 3D spectral features, for each n s The spectral feature is treated as a label, and self-interest is calculated along the spectral dimension. It is then expanded along the spectral dimension to form the spectral channel feature vector. The final aggregated features are obtained by using the MSA module, along with normalization operations and residual linking.

3. The detection method according to claim 1, characterized in that, The method for step eight is as follows: (1) The hyperspectral image data to be detected is segmented into non-intersecting spatial cube blocks. The background confidence strategy proposed in step 3 is used to analyze the hyperspectral data to be detected, and the weight F of each pixel belonging to the foreground is calculated. i,j ; (2) The hyperspectral data to be detected S i,j The RGB data is input into the detection network, and the corresponding hyperspectral reconstruction is performed using the hyperspectral background reconstruction model obtained in step seven. The reconstruction results are compared with the hyperspectral data to be detected, S. i,j Compare and calculate reconstruction differences (3) Obtain the anomaly score by using the weights of each pixel belonging to the foreground calculated in step 8 (1) and the reconstruction difference calculated in step 8 (2). The magnitude of the anomalous value determines whether an image is abnormal, where α is a weighting coefficient used to balance the relationship between the two terms; the calculated anomalous score is then smoothed and nonlinearly mapped to obtain the final result.

Images