A hyperspectral cross-domain robust anomaly detection method, system, device, and medium based on view transformation combined with generative adversarial networks for continuous learning.

Abstract

A robust hyperspectral cross-domain anomaly detection method, system, device, and medium based on view transform combined with generative adversarial networks (GANs) for continuous learning are disclosed. The method includes: constructing continuous learning tasks; performing spatial spectral background filtering preprocessing on the data from each task to obtain a background set; using the background set from the first task as the training set; replaying the data from other tasks and using the replay sets and background sets as training sets; alternately training the ViT GAN model using the training sets; simultaneously constructing F-norm regularization loss terms for the generators of the current and previous tasks; finally, inputting the data from before the current task as the test set into the trained generator; after obtaining the final test result, incrementing the task count by 1 and returning to perform preprocessing. The system, device, and medium are used to implement a robust hyperspectral cross-domain anomaly detection method based on view transform combined with GANs for continuous learning; it features continuous detection of anomalies in cross-domain hyperspectral images.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology, and specifically relates to a hyperspectral cross-domain robust anomaly detection method, system, device, and medium based on visual transformation combined with generative adversarial networks for continuous learning. Background Technology

[0002] Hyperspectral images are high-dimensional images acquired by spectral imagers, containing hundreds of spectral channels. Each pixel represents a continuous, high-dimensional spectral curve, allowing for the selection or extraction of specific bands to highlight target features. Hyperspectral imagers simultaneously detect both the two-dimensional geometric spatial information and the one-dimensional spectral information of a target; therefore, hyperspectral data possesses an "image cube" structure, demonstrating the characteristics and advantages of "image-spectrum integration." Currently, hyperspectral images are widely used in agriculture, military, astronomy, and other fields.

[0003] The most prominent feature of hyperspectral remote sensing images is their ability to effectively fuse spatial image-dimensional information with spectral information. Therefore, targeted spatial-spectral feature extraction, based on the characteristics of hyperspectral remote sensing data and application needs, can more effectively improve the performance of hyperspectral remote sensing technology in practical applications. Furthermore, in geological exploration and disaster response applications, hyperspectral remote sensing images require extensive field visits by relevant experts for calibration, resulting in a very limited number of labeled samples available for practical applications. Thus, the high-dimensional spectral features, the limited number of labeled samples, and the highly correlated spatial characteristics present a series of challenges for hyperspectral remote sensing image classification and anomaly detection.

[0004] For hyperspectral anomaly detection tasks, due to the very limited prior knowledge of anomalous samples and the extreme imbalance between the number of background and anomalous samples, many existing anomaly detection methods rely on identifying features or patterns in hyperspectral data that differ from normal samples. These anomalies may represent potential malfunctions, unusual events, or other unusual phenomena.

[0005] Patent application [CN114005044A], entitled "A Method for Anomaly Detection of Hyperspectral Images Based on Superpixels and Progressive Low-Rank Representation," provides a method for anomaly detection of hyperspectral images based on superpixels and progressive low-rank representation. It primarily addresses the problem of low purity and incompleteness of the background dictionary constructed during existing low-rank representation processes, leading to poor detection results. The method includes: segmenting the hyperspectral image under test using a superpixel segmentation method based on orthogonal projection divergence to obtain a set of homogeneous regions; constructing a background dictionary using the centroids of these homogeneous regions as atoms, and performing low-rank representation on the image; calculating the detection result for the obtained anomalous parts; finally, using this detection result as a benchmark to select pure homogeneous regions, constructing a new background dictionary using their centroids as atoms, and continuing the low-rank representation operation; repeating the above process until the detection result remains unchanged, thus obtaining the final detection result. However, because this method uses low-rank representation of the image, it may cause problems such as large data redundancy and loss of image information. At the same time, the input image is the entire hyperspectral image data, which has a large amount of computation and high computational complexity. Due to the characteristics of hyperspectral images, using only spectral information will result in a lack of knowledge, thus affecting the detection accuracy. Moreover, this model can only detect anomalies in one image at a time, which will lead to a waste of time and resources in practical applications. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention aims to provide a robust hyperspectral cross-domain anomaly detection method, system, device, and medium based on view transformation combined with generative adversarial networks for continuous learning. This method obtains feature-enhanced data through spatial spectral background screening, acquires replay sample sets for each task using a clustering algorithm, constructs a loss term with F-norm regularization, and establishes a model combining ViT and generative adversarial networks. While reconstructing the background of the image to be tested and capturing anomalies, it also prevents the forgetting of knowledge from previous hyperspectral images, ensuring the accuracy of anomaly detection in both the current hyperspectral image to be tested and the previously measured hyperspectral images, and features continuous detection of hyperspectral image anomalies.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A robust hyperspectral cross-domain anomaly detection method based on view transformation combined with generative adversarial networks for continuous learning includes the following steps:

[0009] Step 1: Construct t tasks that satisfy continuous learning of hyperspectral images;

[0010] Step 2: Perform spatial spectral background filtering preprocessing on each data point in the task constructed in Step 1 to obtain the background set;

[0011] Step 3: When t=1, use the background set obtained in Step 2 as the training set and execute Step 4;

[0012] Otherwise, use a clustering algorithm to replay the t-1 task data to obtain a replay sample set, and use the union of the background set obtained in step 2 and the replay sample set obtained in step 3 as the training set, and then execute step 4.

[0013] Step 4: Construct a ViT generative adversarial network model with a cascaded generator and discriminator structure;

[0014] Step 5: When t=1, use the training set from Step 3 to alternately train the generator and discriminator of the ViT Generative Adversarial Network model from Step 4 to obtain the trained ViT Generative Adversarial Network model, and then execute Step 6.

[0015] Otherwise, the generator and discriminator of the ViT generative adversarial network model in step 4 are trained alternately using the training set in step 3, while constructing F-norm regularization loss terms for the generators of the current task and the previous task, and then step 6 is executed.

[0016] Step 6: Use the data from t tasks as the test set, input the test set into the ViT generative adversarial network model trained in Step 5, test and obtain the final result, then let t = t + 1, and return to execute Step 2.

[0017] Step 1 specifically includes:

[0018] Construct t tasks, whose data are denoted as Y1, Y2, ..., Y... m ...Y t , where Y m (m∈1,2,…,t) represents the dataset of the m-th training scene in the open scene.

[0019] Step 2 specifically includes the following steps:

[0020] Step 2.1: Define a hyperspectral image (HSI) containing M×N pixels, each pixel having O channels. Use PCA to reduce the channel dimension to C dimensions (C≤O), represented as... in, Y represents m The spectral vector with coordinates (i, j), Y m It can be divided into anomaly sample set A m and background set, i.e. Y m =[A m B m A m ∪B m =Y m , After capturing representative features of the background set, utilize The reconstructed background sample is obtained, so that The distribution of the background set is similar to that of the background set, where G represents the generator in the view transform generative adversarial network, B represents the background set, and m represents the number of current tasks, m∈1,2,...,t;

[0021] Step 2.2, regarding Y in Step 2.1 m For (m∈1,2,...,t), cosine similarity is used to calculate the distance between adjacent pixels within a local region. The calculation of cosine similarity is expressed as: Pixels with a similarity greater than a set threshold are retained as background samples, while other samples are marked as anomalous samples. Obtain the coarse background mask matrix Where τ is the threshold for background pixel similarity;

[0022] Step 2.3, regarding Y in Step 2.1 m Block-based spatial spectral feature fusion is performed on (m∈1,2,...,t) to enhance spatial information, and the average vector of a local region of size w×w is calculated: Where r is the index of the spectral vector in each w×w local region, the average vector is concatenated with the spectral vector to enhance spatial information: in, This represents a vector containing the original spectral information and local spatial information of the coordinates (i, j). The concatenation operation yields the spatial spectral feature matrix of the HSI:

[0023] Step 2.4: Using the coarse background mask matrix K from step 2.2 m m∈1,2,...,t are indices, for the spatial spectral feature matrix F in step 2.3 m Let m∈1,2,...,t divide the abnormal sample set A m and background set, abnormal sample set A m Represented as: The background set is represented as: Among them, a i Represents the abnormal sample set A m The i-th sample in n a Represents the abnormal sample set A m The number of samples in b i Let n represent the i-th sample in the background set. b Let B represent the number of samples in the background set, and m represent the number of current tasks, where m∈1,2,...,t.

[0024] Step 3 specifically includes the following steps:

[0025] Step 3.1, Current task data Ym There is N m =M×N pixels, using K-means to analyze Y m The clusters are divided into n groups, and each group contains N... i，m Given (i = 1, 2, ..., n) pixels, select the pixel closest to the cluster center from each group. Each sample is used as a representative background sample in the replay sample pool. The replay sample is represented as follows:

[0026]

[0027] Where P represents the current task data Y m The number of replay samples selected in KM i This represents the i-th group in K-means clustering, where the sample set e is replayed when a new task is input. m m∈1,2,...,t according to e m ←e m-1 ∪s(Y m ), (e at t=1) m-1 =φ) is continuously updated to construct the replay sample set as: e m ={s(Y1), s(Y2), ..., s(Y m This is used to preserve representative background samples from historical tasks;

[0028] Step 3.2: Determine if the currently delivered task is the first task, i.e., Y. t =Whether Y1 holds true; if true, the training set becomes the background set, i.e., X_train_AD1 = B1; if false, the replay sample set e from step 3.1 is changed. m The union of the background set and the training set is used as the training set, i.e., X_train_AD. m =B m ∪e m , m∈1,2,...,t.

[0029] Step 4 specifically includes the following steps:

[0030] Step 4.1: Build the generator: The generator consists of encoder 1, decoder, and encoder 2 connected in series. Encoder 1 and encoder 2 have the same structure. Encoder 1 and encoder 2 are used to assist in training the decoder.

[0031] Step 4.1.1: Encoder 1 is constructed using three fully connected layers and one depthwise separable convolutional layer. Encoder 1 is used to map the training set data from Step 3 into the latent space.

[0032] Step 4.1.2: Use four fully connected layers to form a decoder, and use the decoder to map the data in the latent space in step 4.1.1 to a spectral vector of the same size as the training set in step 3;

[0033] Step 4.1.3: Encoder 2 is constructed using three fully connected layers and one depthwise separable convolutional layer. Encoder 2 is used to map the spectral vector in step 4.1.2 to obtain the latent spatial vector.

[0034] Step 4.2: Construct a discriminator with a ViT network structure: The discriminator includes multi-scale convolutional layers, view transformation layers, and an output layer arranged sequentially according to the data logic processing order. The discriminator is used to assist in training the generator.

[0035] Step 4.2.1: The multi-scale convolutional layer consists of three parallel one-dimensional convolutional layers with different kernel sizes and one one-dimensional convolutional layer. The multi-scale convolutional layer is used to obtain an intermediate vector that fuses multi-level features.

[0036] Step 4.2.2: The view transform layer includes L2 multi-head self-attention, a normalization layer, and a residual connection structure. The view transform layer maps the intermediate vector from step 4.2.1 to an attention vector containing key features. The view transform layer can be represented as:

[0037]

[0038]

[0039] Where LN is the layer normalization function, expressed as: μ and σ represent the mean and standard deviation of the neuron’s output in the batch, respectively; ∈ is a small constant; i represents one of the neurons in the layer; and L2_MHSA represents L2 multi-head self-attention.

[0040] Step 4.2.3: Use a Sigmoid layer to form the output layer, and use the output layer to map the attention vector in step 4.2.2 to obtain the true / false judgment result;

[0041] Step 4.3: Concatenate the generator built in Step 4.1 with the discriminator built in Step 4.2 to obtain the ViT generative adversarial network model.

[0042] Step 5 specifically includes the following steps:

[0043] Step 5.1, Construct the generator loss function: When t=1, the generator loss function is expressed as:

[0044]

[0045]

[0046] When t≠1, the generator's loss function is expressed as:

[0047]

[0048] in, In this context, E represents the mathematical expectation, p(X_train_AD) m E1(X_train_AD) represents the data distribution of the training set. m ) represents the output of encoder 1, G(E1(X_train_AD) m )) represents the reconstructed image output by the decoder, Diff(G(E1(X_train_AD) m ))) indicates that differentiable data augmentation is performed on the reconstructed image output by the decoder to obtain an augmented reconstruction sample set, E2(G(E1(X_train_AD) m ))) represents the latent features obtained by encoder 2 for the reconstructed image, D(Diff(G(E1(X_train_AD)). m )))) represents the discriminator's output prediction of the true probability of the reconstructed sample, and MSE represents the mean squared error loss function.

[0049] Formula L Gt L in f The F-norm regularization loss forces the reconstructed image of task (t-1) to be closer to that of task (t), thus reducing the magnitude of parameter updates. For task (t), it is assumed that the replay sample is in the current generator G. t The generated reconstructed sample is represented as Z. t The output for each data point is represented as The reconstructed sample generated by the generator in the (t-1)th task is represented as Z. t-1 According to the output representation Z t The covariance matrix of the reconstructed samples generated by the current generator is calculated as follows:

[0050]

[0051] Where b represents batch_size;

[0052] According to Z t-1 The position (i, j) where the middle pixel is zero is in Z. t The pixel matrix in Z is denoted as Z t [mask], the number of corresponding pixels is denoted as num, and the category-related covariance matrix is ​​calculated:

[0053] The updated cumulative covariance matrix is ​​as follows:

[0054] P_W+=P_X+P_C-I,

[0055] I represents the identity matrix with the same dimensions as P_C.

[0056] Finally, the F-norm regularization loss for each task t is obtained as:

[0057] L f =λ*||P_W-I|| F ;

[0058] Step 5.2: Construct the discriminator loss function: The discriminator loss function L D Represented as:

[0059]

[0060] in, In this context, E represents the mathematical expectation, p(X_train_AD) m E1(X_train_AD) represents the data distribution of the background sample training set. m ) represents the output of encoder 1, G(E1(X_train_AD) m )) represents the reconstructed image output by the decoder, Diff(G(E1(X_train_AD) m ))) represents performing differentiable data augmentation on the reconstructed image output by the decoder to obtain an augmented reconstructed sample set, D(Diff(X_train_AD) m )) represents the prediction result obtained after the enhanced background sample set, which is obtained by performing differentiable data augmentation on the background sample set, is fed into the discriminator;

[0061] Step 5.3: Alternately train the generator and discriminator using the loss function of the generator in Step 5.1 and the loss function of the discriminator in Step 5.2:

[0062] Step 5.3.1: Train the generator: Input the training set into the generator for non-linear mapping to generate reconstructed background samples G(E1(X_train_AD)). m The discriminator is used to reconstruct the background sample G(E1(X_train_AD). m A nonlinear mapping is performed, and the prediction result of whether the reconstructed background sample is true or false is output, denoted as true_G(E1(X_train_AD). m The loss value L is calculated using the mean squared error loss function to determine the prediction result of whether the reconstructed background sample is true or false. G Using the loss value L G The generator is trained using backpropagation.

[0063] Step 5.3.2: Training the discriminator: The discriminator performs a non-linear mapping on the training set, outputting the prediction result of whether the training set samples are true or false, denoted as true_x. The mean squared error loss is used to calculate the loss value L of the prediction result of the training set samples. D_true The discriminator performs a nonlinear mapping on the reconstructed background samples, outputting the prediction result for the reconstructed background samples, denoted as false_x. The mean squared error loss is used to calculate the loss value L of the true and false prediction results on the training set. D_false , will L D_true and L D_false The values ​​are added together to form the total discriminator loss L. D Using L D The discriminator is trained via backpropagation; the model training is now complete.

[0064] Step 6 specifically includes the following steps:

[0065] Step 6.1: Construct the test set Y_test1 = Y1, Y_test2 = Y2, ..., Y_test m =Y m m = 1, 2, ..., t, including all task data prior to the current task; constructing the background set T_B1 = B1, T_B2 = B2, ..., T_B m =B m m = 1, 2, ..., t;

[0066] Step 6.2: Using the trained generator, process the background set T_B of the test set from Step 6.1. m Reconstruct the background sample, and use the mean squared error loss to calculate the reconstructed background sample and Y_test. m The loss value of the data is used to obtain the anomaly probability of each pixel in the reconstructed image. The expression for the anomaly probability of each pixel is: M(i,j). m =MSE(Y_test(i,j)) m G m (E1(B(i,j)) m ))), (0≤i≤M,0≤j≤N), where G m Y_test(i, j) represents the generator trained for the m-th task, which is then tested sequentially on the first m-1 tasks. m This represents the pixel in the hyperspectral image to be tested for the m-th task;

[0067] Step 6.3: Increment the value of m by 1 and return to step 2.

[0068] A hyperspectral cross-domain robust anomaly detection system based on view transformation combined with generative adversarial networks for continuous learning includes:

[0069] Continuous learning task building module: Constructs t tasks that satisfy continuous learning of hyperspectral images;

[0070] Spatial-spectral feature fusion module: performs spatial-spectral feature fusion on the task data constructed in the continuous learning task construction module;

[0071] Coarse background mask matrix construction module: performs background filtering preprocessing on the results of the spatial spectral feature fusion module;

[0072] Sample replay module: Using the data in the spatial spectrum feature fusion module, a clustering algorithm is used to obtain the replay sample set, which is then combined with the background set in the coarse background mask matrix construction module and fed into the generator module and discriminator module models.

[0073] Generator module: Construct encoder 1, decoder and encoder 2 connected in series to form a generator;

[0074] Discriminator module: Construct a discriminator including multi-scale convolutional layers, view transformation layers, and output layers to assist in the training of the generator;

[0075] Generator and discriminator loss modules: Train the generator and discriminator modules to obtain the trained ViT generative adversarial network model;

[0076] Test module: Test the ViT generative adversarial network model trained in the generator and discriminator loss modules.

[0077] A hyperspectral cross-domain robust anomaly detection device based on view transformation combined with generative adversarial networks for continuous learning includes:

[0078] Memory: Used to store the computer program that implements the hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with generative adversarial networks for continuous learning;

[0079] Processor: Used to implement the described hyperspectral cross-domain robust anomaly detection method based on continuous learning using view transformation combined with generative adversarial networks when executing the computer program.

[0080] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with generative adversarial networks for continuous learning.

[0081] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0082] 1. The present invention provides a hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial networks for continuous learning. It utilizes historical data to obtain a replay sample set, enhances the model's memory of old knowledge, and effectively solves the catastrophic forgetting problem of the model.

[0083] 2. The present invention provides a hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial networks for continuous learning. An F-norm regularization loss term is designed to enable the model to learn new knowledge while retaining previous knowledge information. The model can reduce the correlation between the currently learned features and the previous features, thereby reducing the conflict between samples and enhancing the model's ability to learn new knowledge.

[0084] 3. The present invention provides a robust cross-domain anomaly detection method for hyperspectral images based on view transformation combined with generative adversarial networks (GANs) for continuous learning. A ViT combined with a GAN model is designed, which can capture deeper sample features by utilizing the L2 multi-head self-attention in ViT. The generator and discriminator in the GAN are trained alternately to improve the generator's ability to reconstruct background samples. This model can improve the learning performance of hyperspectral images in cross-scene tasks and promote the practical application of deep learning-based structures in hyperspectral interpretation.

[0085] In summary, this invention utilizes historical data to obtain a replay sample set, designs an F-norm regularization loss term, and designs a ViT combined with a generative adversarial network (GAN) model. This model can capture deeper sample features by leveraging L2 multi-head self-attention in ViT, and improves the generator's ability to reconstruct background samples by alternating training of the generator and discriminator in the GAN. It enhances the model's memory of old knowledge and its ability to learn new knowledge, improves the learning performance of hyperspectral images across different scenarios, and promotes the practical application of deep learning-based structures in hyperspectral interpretation. Attached Figure Description

[0086] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0087] Figure 2 This is a diagram illustrating the process of detecting persistent hyperspectral anomalies using the method of this invention.

[0088] Figure 3 A model diagram of visual transformation generative adversarial network is constructed for the hyperspectral persistent anomaly detection method of the present invention. Detailed Implementation

[0089] See Figure 1 A robust hyperspectral cross-domain anomaly detection method based on view transformation combined with generative adversarial networks for continuous learning includes the following steps:

[0090] Step 1: Construct t tasks that satisfy continuous learning of hyperspectral images;

[0091] Step 2: Perform spatial spectral background filtering preprocessing on each data point in the task constructed in Step 1 to obtain the background set;

[0092] Step 3: When t=1, use the background set obtained in Step 2 as the training set and execute Step 4;

[0093] Otherwise, use a clustering algorithm to replay the t-1 task data to obtain a replay sample set, and use the union of the background set obtained in step 2 and the replay sample set obtained in step 3 as the training set, and then execute step 4.

[0094] Step 4: Construct a ViT generative adversarial network model with a cascaded generator and discriminator structure;

[0095] Step 5: When t=1, use the training set from Step 3 to alternately train the generator and discriminator of the ViT Generative Adversarial Network model from Step 4 to obtain the trained ViT Generative Adversarial Network model, i.e., the generator, and then execute Step 6.

[0096] Otherwise, the generator and discriminator of the ViT generative adversarial network model in step 4 are trained alternately using the training set in step 3, while constructing F-norm regularization loss terms for the generators of the current task and the previous task to obtain the trained generator, and then step 6 is executed.

[0097] Step 6: Use the data from t tasks as the test set, input the test set into the generator trained in Step 5, test and obtain the final result, then let t = t + 1, and return to execute Step 2.

[0098] Step 1 specifically includes:

[0099] Five tasks are constructed, and their data are denoted as Y1, Y2, ..., Y... m ...Y5, where Y m (m∈1,2,...,5) represents the dataset of the m-th training scene in the open scene.

[0100] Step 2 specifically includes the following steps:

[0101] Step 2.1: Define a hyperspectral image (HSI) containing M×N pixels, each pixel having O channels. Use PCA to reduce the channel dimension to C dimensions (C≤O), where M represents the number of rows and N represents the number of columns in the hyperspectral image, denoted as... in, Y represents m The spectral vector with coordinates (i, j), Y mIt can be divided into anomaly sample set A m and background set, i.e. Y m =[A m B m A m ∪B m =Y m , After capturing representative features of the background set, utilize The reconstructed background sample is obtained, so that The distribution of the background set is similar to that of the background set, where G represents the generator in the view transform generative adversarial network, B represents the background set, and m represents the number of current tasks, m∈1,2,...,5;

[0102] For example, the data of the hyperspectral image abu-urban-4 is a data cube of size (100, 100, 205), where the two 100s represent the number of rows and columns of the abu-urban-4 data, respectively, and 205 represents the number of channels of the abu-urban-4 data. After dimensionality reduction to 188 dimensions using PCA, the size of the abu-urban-4 data is updated to (100, 100, 376).

[0103] Step 2.2, regarding Y in Step 2.1 m (m∈1,2,...,5) The distance between adjacent pixels in a local region is calculated using cosine similarity. The calculation of cosine similarity is expressed as: Pixels with a similarity greater than a set threshold are retained as background samples, while other samples are marked as anomalous samples. Obtain the coarse background mask matrix Where τ is the threshold for background pixel similarity;

[0104] Specifically, in this embodiment, the value of τ is 0.99. If the cosine similarity is less than 0.99, the pixel is considered to have a large difference from its neighboring pixels and is therefore considered an abnormal sample. If the cosine similarity is greater than or equal to 0.99, the pixel is considered a background sample. Assuming there are P samples in the background set, then... So, for example, for the hyperspectral image abu-urban-4, its background set

[0105] Step 2.3, regarding Y in Step 2.1 m Block-based spatial spectral feature fusion is performed on (m∈1,2,...,5) to enhance spatial information, and the average vector of a local region of size w×w is calculated: Where r is the index of the spectral vector in each w×w local region, and w is 3. The average vector and the spectral vector are concatenated to enhance spatial information. in This represents a vector containing the original spectral information and local spatial information of the coordinates (i, j). The concatenation operation yields the spatial spectral feature matrix of the HSI:

[0106] For example, for a local region of the hyperspectral image abu-urban-4, a mean vector of size (1, 1, 188) is calculated. Concatenating the mean vector with the spectral vector (1, 1, 188) yields an f vector of size (1, 1, 376). i，j Perform this operation on each pixel in the hyperspectral image to obtain the spatial-spectral feature matrix F, which fuses the original spectral information with the local spatial information;

[0107] Step 2.4: Using the coarse background mask matrix K from step 2.2 m m∈1,2,...,5 are indices for the spatial spectral feature matrix F from step 2.3. m Let m ∈ 1, 2, ..., 5 to partition the abnormal sample set A. m and background set, abnormal sample set A m Represented as: The background set is represented as: Among them, a i Represents the abnormal sample set A m The i-th sample in n a Represents the abnormal sample set A m The number of samples in b i Let n represent the i-th sample in the background set. b Let B represent the number of samples in the background set, and m represent the number of current tasks, where m∈1,2,...,5.

[0108] Step 3 specifically includes the following steps:

[0109] Step 3.1, see Figure 2 , Figure 2 This is a flowchart illustrating the processing steps of the hyperspectral persistent anomaly detection method invented by the present invention. As shown in the figure, the current task's data Y... m There is N m =M×N pixels, using K-means to analyze Y m The clusters are divided into 3 groups, each containing N. i，m (i = 1, 2, 3) pixels, select the pixel closest to the cluster center in each group. Each sample is used as a representative background sample in the replay sample pool. The replay sample is represented as follows:

[0110]

[0111] Where P represents the current task data Ym The number of replay samples selected in KM i This represents the i-th group in K-means clustering, where the sample set e is replayed when a new task is input. m m∈1,2,...,5 according to e m ←e m-1 ∪s(Y m ), (e at t=1) m-1 =φ) is continuously updated to construct the replay sample set as: e m ={s(Y1), s(Y2), ..., s(Y m ), ..., s(Y5)} to store representative background samples from historical tasks;

[0112] Step 3.2: Determine if the currently delivered task is the first task, i.e., Y. t =Whether Y1 holds true; if true, the training set becomes the background set, i.e., X_train_AD1 = B1; if false, the replay sample set e from step 3.1 is changed. m The union of the background set and the training set is used as the training set, i.e., X_train_AD. m =B m ∪e m m∈1,2,...,5.

[0113] Step 4 specifically includes the following steps:

[0114] See Figure 3 , Figure 3 The figure shows a model diagram of the hyperspectral persistent anomaly detection and visual transform generative adversarial network (GAN) invented by the present invention:

[0115] Step 4.1: Build the generator: The generator consists of encoder 1, decoder, and encoder 2 connected in series. Encoder 1 and encoder 2 have the same structure. Encoder 1 and encoder 2 are used to assist in training the decoder.

[0116] Step 4.1.1: Encoder 1 is constructed using three fully connected layers and one depthwise separable convolutional layer, and the training set data X_train_AD from step 3 is used. m Input encoder 1, and input the training set data X_train_AD m Mapped to a 100-dimensional latent vector, denoted as E1(X_train_AD) m );

[0117] Step 4.1.2: A decoder is constructed using four fully connected layers. The decoder maps the data in the latent space from Step 4.1.1 to spectral vectors of the same size as the training set from Step 3. After mapping by the encoder, the decoder reconstructs the latent vectors into a 2C-dimensional reconstructed image, denoted as G(E1(X_train_AD). m ));

[0118] Step 4.1.3: Encoder 2 is constructed using three fully connected layers and one depthwise separable convolutional layer. Encoder 2 is used to map the spectral vector in step 4.1.2 to obtain the latent space vector, denoted as E2(G(E1(X_train_AD1))).

[0119] Step 4.2: Construct a discriminator with a ViT network structure: The discriminator includes multi-scale convolutional layers, view transformation layers, and an output layer arranged sequentially according to the data logical processing order. The discriminator is used to assist in training the generator. The reconstructed image G(E1(X_train_AD) output by the generator is then used. m After differentiable data augmentation, the augmented reconstructed image is denoted as Diff(G(E1(X_train_AD)). m The discriminator outputs the prediction result of the true probability of the enhanced reconstructed image, denoted as D(Diff(G(E1(X_train_AD)). m ))));

[0120] Step 4.2.1: The multi-scale convolutional layer consists of three parallel one-dimensional convolutional layers with different kernel sizes and one one-dimensional convolutional layer. The multi-scale convolutional layer is used to obtain an intermediate vector that fuses multi-level features, denoted as d. i ;

[0121] Step 4.2.2: The view transformation layer includes L2 multi-head self-attention, a normalization layer, and a residual connection structure. The view transformation layer is used to transform the intermediate vector d from step 4.2.1... i Mapped to an attention vector containing key features; the view transformation layer can be represented as:

[0122]

[0123]

[0124] Where LN is the layer normalization function, expressed as: μ and σ represent the mean and standard deviation of the neuron's output in the batch, respectively. ∈ is a small constant. i represents one of the neurons in the layer. L2_MHSA represents L2 multi-head self-attention. The batch size is 64.

[0125] Step 4.2.3: Use a Sigmoid layer to form the output layer. Use the output layer to map the attention vector in step 4.2.2. The result is predicted as real, and predicted as false.

[0126] Step 4.3: Concatenate the generator built in Step 4.1 with the discriminator built in Step 4.2 to obtain the ViT generative adversarial network model.

[0127] Step 5 specifically includes the following steps:

[0128] See Figure 3 , Figure 3 The figure shows a model diagram of the hyperspectral persistent anomaly detection and visual transform generative adversarial network (GAN) invented by the present invention:

[0129] Step 5.1, Construct the generator loss function: When t=1, the generator loss function is expressed as:

[0130]

[0131]

[0132] When t≠1, the generator's loss function is expressed as:

[0133]

[0134] in, In this context, E represents the mathematical expectation, p(X_train_AD) m E1(X_train_AD) represents the data distribution of the training set. m ) represents the output of encoder 1, G(E1(X_train_AD) m )) represents the reconstructed image output by the decoder, Diff(G(E1(X_train_AD) m ))) indicates that differentiable data augmentation is performed on the reconstructed image output by the decoder to obtain an augmented reconstruction sample set, E2(G(E1(X_train_AD) m ))) represents the latent features obtained by encoder 2 for the reconstructed image, D(Diff(G(E1(X_train_AD)). m )))) represents the discriminator's output prediction of the true probability of the reconstructed sample, and MSE represents the mean squared error loss function.

[0135] Formula L Gt L in f The F-norm regularization loss forces the reconstructed image of task (t-1) to be closer to that of task (t), thus reducing the magnitude of parameter updates. For task (t), it is assumed that the replay sample is in the current generator G.t The generated reconstructed sample is represented as Z. t The output for each data point is represented as The reconstructed sample generated by the generator in the (t-1)th task is represented as Z. t-1 According to the output representation Z t The covariance matrix of the reconstructed samples generated by the current generator is calculated as follows:

[0136]

[0137] Where b represents batch_size, which is 64;

[0138] According to Z t-1 The position (i, j) where the middle pixel is zero is in Z. t The pixel matrix in Z is denoted as Z t [mask], the number of corresponding pixels is denoted as num, and the category-related covariance matrix is ​​calculated:

[0139] The updated cumulative covariance matrix is ​​as follows:

[0140] P_W+=P_X+P_C-I,

[0141] I represents the identity matrix with the same dimensions as P_C.

[0142] Finally, the F-norm regularization loss for each task t is obtained as:

[0143] L f =λ*||P_W-I|| F ,

[0144] The loss of the final generator is denoted as L. G ;

[0145] Step 5.2: Construct the discriminator loss function: The discriminator loss function L D Represented as:

[0146]

[0147] in, In this context, E represents the mathematical expectation, p(X_train_AD) m E1(X_train_AD) represents the data distribution of the background sample training set. m ) represents the output of encoder 1, G(E1(X_train_AD) m )) represents the reconstructed image output by the decoder, Diff(G(E1(X_train_AD) m))) represents performing differentiable data augmentation on the reconstructed image output by the decoder to obtain an augmented reconstructed sample set, D(Diff(X_train_AD) m )) represents the prediction result obtained after the enhanced background sample set, which is obtained by performing differentiable data augmentation on the background sample set, is fed into the discriminator;

[0148] Step 5.3: Alternately train the generator and discriminator using the loss function of the generator in Step 5.1 and the loss function of the discriminator in Step 5.2, finally obtaining the trained generator:

[0149] Step 5.3.1: Train the generator: Input the training set into the generator for non-linear mapping to generate reconstructed background samples G(E1(X_train_AD)). m The discriminator is used to reconstruct the background sample G(E1(X_train_AD). m A nonlinear mapping is performed, and the prediction result of whether the reconstructed background sample is true or false is output, denoted as true_G(E1(X_train_AD). m The loss value L is calculated using the mean squared error loss function to determine the prediction result of whether the reconstructed background sample is true or false. G Using the loss value L G The generator is trained using backpropagation.

[0150] Step 5.3.2: Training the discriminator: The discriminator performs a non-linear mapping on the training set, outputting the prediction result of whether the training set samples are true or false, denoted as true_x. The mean squared error loss is used to calculate the loss value L of the prediction result of the training set samples. D_true The discriminator performs a nonlinear mapping on the reconstructed background samples, outputting the prediction result for the reconstructed background samples, denoted as false_x. The mean squared error loss is used to calculate the loss value L of the true and false prediction results on the training set. D_false , will L D_true and L D_false The values ​​are added together to form the total discriminator loss L. D Using L D The discriminator is trained by backpropagation; the model is trained iteratively 5000 times, and the model training is now complete.

[0151] Step 6 specifically includes the following steps:

[0152] Step 6.1: Construct the test set Y_test1 = Y1, Y_test2 = Y2, ..., m = 1, 2, ..., 5, including all task data prior to the current task; the background set for constructing the test set is T_B1 = B1, T_B2 = B2, ..., T_B m =B mm = 1, 2, ..., 5;

[0153] Step 6.2: Using the trained generator, process the background set T_B of the test set from Step 6.1. m Reconstruct the background sample, and use the mean squared error loss to calculate the reconstructed background sample and Y_test. m The loss value of the data is used to obtain the anomaly probability of each pixel in the reconstructed image. The expression for the anomaly probability of each pixel is: M(i,j). m =MSE(Y_test(i,j)) m G m (E1(B(i,j)) m ))), (0≤i≤M,0≤j≤N), where G m Y_test(i, j) represents the generator trained for the m-th task, which is then tested sequentially on the first m-1 tasks. m This represents the pixel in the hyperspectral image to be tested for the m-th task;

[0154] Specifically, for example, m=3, the dataset for the current task is abu-beach-3, the dataset for the second task is abu-urban-5, and the dataset for the first task is abu-urban-4. Step 2 is performed on abu-beach-3 to obtain the background set, denoted as T_B3. This background set is then fed into the trained generator to obtain the reconstructed image G3(T_B3). The mean squared error loss is used to obtain the anomaly probability M(i,j)3 for each pixel (0≤i≤100, 0≤j≤100). Then, step 2 is performed on abu-urban-5... Step 2 obtains the background set, denoted as T_B2. The background set is fed into the trained generator to obtain the reconstructed image G2(T_B2). The anomaly probability M(i,j)2 of each pixel is obtained using the mean squared error loss (0≤i≤100, 0≤j≤100). Finally, Step 2 is performed on abu-urban-4 to obtain the background set, denoted as T_B1. The background set is fed into the trained generator to obtain the reconstructed image G3(T_B1). The anomaly probability M(i,j)1 of each pixel is obtained using the mean squared error loss (0≤i≤100, 0≤j≤100).

[0155] Step 6.3: Increment the value of m by 1 and return to step 2.

[0156] A hyperspectral cross-domain robust anomaly detection system based on view transformation combined with generative adversarial networks for continuous learning includes:

[0157] Continuous learning task construction module: Construct t tasks that satisfy continuous learning of hyperspectral images, and implement step 1 of a hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial network for continuous learning;

[0158] Spatial-spectral feature fusion module: Performs spatial-spectral feature fusion on the task data constructed in the continuous learning task construction module, which is used to implement step 2 of a hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with generative adversarial network for continuous learning.

[0159] Rough background mask matrix construction module: performs background filtering preprocessing on the results of the spatial-spectral feature fusion module, which is used to implement step 2 of a hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with generative adversarial network for continuous learning;

[0160] Sample replay module: Using the data in the spatial-spectral feature fusion module, the replay sample set is obtained by clustering algorithm, and then the union of the replay sample set with the background set in the coarse background mask matrix construction module is taken and fed into the generator module and discriminator module model to implement step 3 of a hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial network for continuous learning.

[0161] Generator module: Construct encoder 1, decoder and encoder 2 connected in series to form a generator, which is used to implement step 4 of a hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial network for continuous learning;

[0162] Discriminator module: Construct a discriminator including multi-scale convolutional layers, view transformation layers and output layers to assist in the training of the generator, and to implement step 4 of a hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial network for continuous learning;

[0163] Generator and discriminator loss module: Train the generator module and discriminator module to obtain the trained ViT generative adversarial network model, which is used to implement step 5 of a hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial network for continuous learning;

[0164] Test module: Test the ViT generative adversarial network model trained in the generator and discriminator loss modules. This is step 6 of implementing a hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial network for continuous learning.

[0165] A hyperspectral cross-domain robust anomaly detection device based on view transformation combined with generative adversarial networks for continuous learning includes:

[0166] Memory: Used to store the computer program that implements the hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with generative adversarial networks for continuous learning;

[0167] Processor: Used to implement the described hyperspectral cross-domain robust anomaly detection method based on continuous learning using view transformation combined with generative adversarial networks when executing the computer program.

[0168] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or any conventional processor. The processor is the control center of the device for a hyperspectral cross-domain robust anomaly detection method based on view transform combined with generative adversarial networks for continuous learning, and it connects various parts of the device using various interfaces and lines.

[0169] When the processor executes the computer program, it implements the steps of the above-mentioned hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial network for continuous learning, thus realizing the hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial network for continuous learning.

[0170] Alternatively, when the processor executes the computer program, it implements the functions of each module in the above system, for example: a continuous learning task construction module: constructing t tasks that satisfy continuous learning of hyperspectral images; a spatial-spectral feature fusion module: performing spatial-spectral feature fusion on the task data constructed in the continuous learning task construction module; a coarse background mask matrix construction module: performing background filtering preprocessing on the results of the spatial-spectral feature fusion module; and a sample replay module: using the data in the spatial-spectral feature fusion module, obtaining a replay sample set using a clustering algorithm, then taking the union of this set with the background set in the coarse background mask matrix construction module, and sending the result to the generator module and the discriminator. The model consists of the following modules: a generator module (comprised of encoder 1, decoder 2, and encoder 2 connected in series to form the generator); a discriminator module (comprised of multi-scale convolutional layers, view transformation layers, and output layers to assist in generator training); a generator and discriminator loss module (training the generator and discriminator modules to obtain a trained ViT generative adversarial network model); a testing module (testing the ViT generative adversarial network model trained in the generator and discriminator loss modules); and the output yields the results of the hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial networks for continuous learning.

[0171] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing preset functions. These instruction segments describe the execution process of the computer program in the device of the hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial networks for continuous learning. For example, the computer program can be divided into modules, with the following modules and specific functions: Continuous learning task construction module: constructing t tasks to satisfy continuous learning of hyperspectral images; Spatial-spectral feature fusion module: performing spatial-spectral feature fusion on the task data constructed in the continuous learning task construction module; Coarse background mask matrix construction module: performing background screening preprocessing on the results of the spatial-spectral feature fusion module; Sample replay module: using the data in the spatial-spectral feature fusion module, obtaining a replay sample set using a clustering algorithm, then taking the union with the background set in the coarse background mask matrix construction module, and sending it to the generator module and discriminator module. The model comprises: a generator module consisting of encoder 1, decoder 2, and encoder 2 connected in series; a discriminator module consisting of a discriminator including multi-scale convolutional layers, view transformation layers, and an output layer, used to assist in the training of the generator; a generator and discriminator loss module for training the generator and discriminator modules to obtain a trained ViT generative adversarial network model; and a testing module for testing the trained ViT generative adversarial network model using the generator and discriminator loss modules, outputting the results of the proposed hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial networks for continuous learning.

[0172] The device for the hyperspectral cross-domain robust anomaly detection method based on view transform combined with generative adversarial networks for continuous learning can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This device may include, but is not limited to, processors and memory. Those skilled in the art will understand that the above is an example of a device for a hyperspectral cross-domain robust anomaly detection method based on view transform combined with generative adversarial networks for continuous learning, and does not constitute a limitation on the device. It may include more components than described above, or combine certain components, or use different components. For example, the device may also include input / output devices, network access devices, buses, etc.

[0173] The memory can be used to store the computer program and / or modules. The processor implements various functions of the device for a hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with generative adversarial networks for continuous learning by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory.

[0174] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0175] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with generative adversarial networks for continuous learning.

[0176] If the system integration module / unit of the hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with generative adversarial networks for continuous learning is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0177] This invention implements all or part of the process of the aforementioned hyperspectral cross-domain robust anomaly detection method based on view transform combined with generative adversarial networks for continuous learning. It can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program implements the steps of the aforementioned hyperspectral cross-domain robust anomaly detection method based on view transform combined with generative adversarial networks for continuous learning. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or a preset intermediate form, etc.

[0178] The computer-readable storage medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0179] It should be noted that the content contained in the computer-readable storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0180] It should be noted that embodiments of the present invention can be implemented using hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated hardware.

[0181] Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry of semiconductors such as very large-scale integrated circuits or gate arrays, logic chips, transistors, etc., or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0182] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

[0183] Simulation Experiment

[0184] This simulation experiment verifies and demonstrates the effectiveness of the view transformation generation method for robust anomaly detection across hyperspectral domains against continuous learning using a specific implementation.

[0185] 1. Experimental conditions

[0186] In the simulation experiment, for the experiment of continuous anomaly detection in hyperspectral images, the five hyperspectral image datasets used for experimental verification were BU (Beach-Urban), namely two scenes: urban and beach. The urban scene has two hyperspectral images, and the beach scene has three hyperspectral images. In the experiment of continuous anomaly detection in hyperspectral images, the five tasks appeared in sequence: (1) abu-urban-4, (2) abu-urban-5, (3) abu-beach-3, (4) abu-beach-2, and (5) abu-beach-1.

[0187] Table 1. Detection data of persistent anomalies in hyperspectral images

[0188] Dataset Number of lines Number of columns Number of spectral bands abu-urban-4 100 100 205 abu-urban-5 100 100 205 abu-beach-3 100 100 188 abu-beach-2 100 100 193 abu-beach-1 150 150 188

[0189] This invention utilizes a hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial networks for continuous learning. In the task of persistent anomaly detection in hyperspectral images, it is compared with pure model (Trans GAN), fine-tuning-based (FT-Trans GAN), regularization-based (D-Trans GAN), replay-based (R-Trans GAN), and joint learning-based (J-Trans GAN) methods in cross-scene anomaly detection. Detection is performed under the same experimental conditions, and the AUC and BWT values ​​of the detection results are analyzed.

[0190] The AUC (Area Under Curve) value is defined as the area under the ROC curve and the coordinate axis. Since the ROC curve is generally above the line y = x, the AUC value ranges between 0.5 and 1. The closer the AUC is to 1, the higher the realism of the detection method; when it equals 0.5, the realism is the lowest, and it has no application value. The BWT (Backward Transfer) value is defined as the backward transfer index, used to evaluate the impact of new tasks on historical tasks. The larger the BWT index, the better the proposed continuous learning training strategy performs in mitigating catastrophic forgetting. ACC represents the average of the sum of the AUC values ​​of the model trained on the current task t for previous tasks; the larger the value, the stronger the ability to continuously learn new knowledge.

[0191] The formulas for calculating ACC and BWT values ​​are as follows:

[0192]

[0193]

[0194] Among them, AUC t,iThis represents the AUC value of the current model on task i after training for t tasks.

[0195] 2. Analysis of Experimental Results

[0196] See Table 2, which shows the ACC score and BWT value for continuous abnormality detection results under each detection setting.

[0197] Table 2 shows the ACC scores and BWT values ​​for persistent anomaly detection and classification under various detection settings.

[0198]

[0199] Referring to Table 2, Tasks 1-2 indicate that Abu-Urban-4 is performed first, followed by Abu-Urban-5; Tasks 1-3 indicate that Abu-Urban-4 is performed first, followed by Abu-Urban-5, and finally Abu-Beach-3; Tasks 1-4 indicate that Abu-Urban-4 is performed first, followed by Abu-Urban-5, followed by Abu-Beach-3, and finally Abu-Beach-2; Tasks 1-5 indicate that Abu-Urban-4 is performed first, followed by Abu-Urban-5, followed by Abu-Beach-3, followed by Abu-Beach-2, and finally Abu-Beach-1.

[0200] In the experimental setup, Trans GAN and J-Trans GAN lack memory capabilities and cannot mitigate catastrophic forgetting. The fine-tuning method FT-Trans GAN exhibits performance instability as the number of tasks increases, and its BWT value is negative, indicating that when new tasks are introduced for training, the model is more inclined to remember knowledge from new tasks while forgetting knowledge from old tasks, meaning the model suffers from severe catastrophic forgetting. The regularization-based method D-TransGAN uses an F-norm regularization loss term to narrow the gap between the parameters of the old and new models, thereby preventing forgetting. However, specific experiments show that the regularization loss term suffers from forgetting as the number of tasks increases. The replay-based R-TransGAN method uses representative samples from previous tasks as auxiliary samples for training the network. Therefore, its ACC value decreases less compared to other models as the number of tasks increases, indicating that the model can remember previously learned knowledge. However, the model suffers significant forgetting in the first 1-3 and 1-4 tasks, while its performance improves in the first 1-5 tasks, demonstrating that the replay-based method increases the network's plasticity. The CL-TransGAN method, utilizing replay and F-norm regularization loss terms, shows relatively stable performance among all compared algorithms. Furthermore, as the number of tasks increases, it rivals joint learning methods, even surpassing them in all ACC evaluation metrics. Experiments show that the proposed adaptive sample replay strategy and F-norm regularization loss term effectively mitigate catastrophic forgetting, and the view transform generative adversarial network can play a role in solving the hyperspectral anomaly detection problem.

Claims

1. A hyperspectral cross-domain robust anomaly detection method based on view transformation combined with generative adversarial network for continuous learning, characterized in that, The method comprises the following steps: Step 1, constructing t tasks meeting the continuous learning of hyperspectral images; Step 2, performing empty spectral background screening preprocessing on each data in the task constructed in step 1 to obtain a background set, and the specific steps are as follows: Step 2.1: Define a hyperspectral image HSI containing M×N pixels, each pixel having O channels. Use PCA to reduce the channel dimension to C dimensions (C≤O), represented as... in, Y represents m The spectral vector with coordinates (i,j) in the middle, Y m Divided into abnormal sample set A m and background set B m , i.e. Y m =[A m B m A m ∪B m =Y m , After capturing representative features of the background set, utilize The reconstructed background sample is obtained, so that The distribution of the background set is similar to that of the background set, where G represents the generator in the view transform generative adversarial network, B represents the background set, and m represents the number of current tasks, m∈1,2,…,t; Step 2.2, Y in step 2.1 is calculated using cosine similarity m The distance between adjacent pixels in the local region is calculated using cosine similarity, which is expressed as: Pixels with similarity greater than a set threshold are retained as background samples, and other samples are marked as abnormal samples, i.e. A rough background mask matrix is obtained Where τ is the threshold of the similarity of background pixels; Step 2.

3. Block-based spectral-spatial feature fusion is performed on Y m where r is the index of the spectral vector in each w x w local region, and the average vector is concatenated with the spectral vector to enhance the spatial information: where r is the index of the spectral vector in each w x w local region, and the average vector is concatenated with the spectral vector to enhance the spatial information: where denotes the vector containing the original spectral information and local spatial information of the (i,j) coordinate, denotes the concatenation operation, and the spectral-spatial feature matrix of HSI is obtained: Step 2.4.4, the coarse background mask matrix K of step 2.2 m , m e 1, 2, …, t is an index, the empty spectral feature matrix F of step 2.3 m , m e 1, 2, …, t divides the anomaly sample set A m and the background set B m , the anomaly sample set A m is expressed as: The background set is expressed as: Wherein, a i represents the i-th sample in the anomaly sample set A m , n a represents the number of samples in the anomaly sample set A m , b i represents the i-th sample in the background set, n b represents the number of samples in the background set, B represents the background set, and m represents the current task number, m e 1, 2, …, t. Step 3, when t = 1, taking the background set obtained in step 2 as a training set, and performing step 4; Otherwise, replaying the data of t-1 tasks by using a clustering algorithm to obtain a replay sample set, and taking the union of the background set obtained in step 2 and the replay sample set obtained in step 3 as a training set, and performing step 4; Step 4, building a ViT generative adversarial network model with a cascaded generator and discriminator structure, and the specific steps are as follows: Step 4.1, building a generator: the generator comprises an encoder 1, a decoder and an encoder 2 connected in series, the encoder 1 and the encoder 2 have the same structure, and the decoder is trained with the aid of the encoder 1 and the encoder 2: Step 4.1.1, adopting three fully connected layers and a depth separable convolution layer to constitute the encoder 1, and mapping the data of the training set in step 3 into a latent space by using the encoder 1; Step 4.1.2, adopting four fully connected layers to constitute the decoder, and mapping the data in the latent space in step 4.1.1 to a spectral vector with the same size as the training set in step 3 by using the decoder; Step 4.1.3, adopting three fully connected layers and a depth separable convolution layer to constitute the encoder 2, and mapping the spectral vector in step 4.1.2 to obtain a latent space vector by using the encoder 2; Step 4.2, building a discriminator with a ViT network structure: the discriminator comprises a multi-scale convolution layer, a view transformation layer and an output layer arranged in sequence according to the data logic processing order, and the generator is trained with the aid of the discriminator: Step 4.2.1, the multi-scale convolution layer comprises three one-dimensional convolution layers with different convolution kernel sizes connected in parallel and one one-dimensional convolution layer, and the multi-scale convolution layer is used to obtain an intermediate vector fused with multi-level features; Step 4.2.2, the view transformation layer comprises an L2 multi-head self-attention, a normalization layer and a residual connection structure, and the view transformation layer is used to map the intermediate vector in step 4.2.1 to an attention vector containing key features; the view transformation layer can be expressed as: where LN is a layer normalization function, denoted as: i denotes one of the neurons of the layer, μ and σ denote the mean and standard deviation of the output of the neuron in the batch, respectively, ∈ is a constant, and L2_MHSA denotes L2 multi-head self-attention. Step 4.2.3, adopting a Sigmoid layer to constitute the output layer, and mapping the attention vector in step 4.2.2 by using the output layer to obtain a true or false judgment result; Step 4.3, cascading the generator built in step 4.1 and the discriminator built in step 4.2 to obtain a ViT generative adversarial network model; Step 5, when t = 1, alternately training the generator and the discriminator of the ViT generative adversarial network model in step 4 by using the training set in step 3 to obtain a trained ViT generative adversarial network model, and performing step 6; Otherwise, alternately training the generator and the discriminator of the ViT generative adversarial network model in step 4 by using the training set in step 3, and constructing an F-norm regularization loss term for the generator of the current task and the last task, and performing step 6; Definition L f denotes the F-norm regularization loss, which forces the reconstructed image of the t-1th task to be close to the distance of the tth task, for the tth task, assuming the replay sample is in the current generator G t The generated reconstructed sample is denoted as Z t The output of each data point is denoted as The reconstructed sample generated by the generator at the T-1th task is denoted as Z t-1 According to the output denoted as Z t The covariance matrix of the reconstructed sample generated by the current generator is calculated, denoted as: Wherein, b represents batch_size; According to Z t-1 The position (i, J) in which the middle pixel is zero in Z t The pixel matrix in Z t [mask] is recorded as Z The updated cumulative covariance matrix is: P_W+ = P_X + P_C - I, I represents a unit matrix of the same dimension as P_C; Finally, the F-norm regularization loss of each task t is obtained as: L f = λ * ||P_W - I|| F ; Step 6, the data under t tasks are taken as a test set, the test set is input into the ViT generative adversarial network model trained in step 5, testing is performed to obtain a final result, t is set to t+1, and step 2 is returned to be executed.

2. The hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with continual learning of a generative adversarial network according to claim 1, characterized in that, The step 1 is specifically: t tasks are constructed, and the data of the tasks are respectively denoted as Y1, Y2, …, Yt m , …, Yt t , wherein Y m (m∈1, 2, …, t) represents a data set of the mth training scene in the open scene.

3. The hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with continual learning of a generative adversarial network according to claim 1, characterized in that, The step 3 specifically includes the following steps: Step 3.1, data Y of the current task m N in Y m = M x N pixels, Y is clustered into n groups by K-means, each group contains N m pixels, and the pixel closest to the cluster center in each group is selected as a representative background sample in the replay sample pool. The replay sample is represented as: i,m (i = 1, 2, …, n) samples in each group are selected as representative background samples in the replay sample pool. The replay sample is represented as: where P represents the number of replay samples selected from the current task data Y m KM i represents the i-th group of K-means clustering, when a new task is input, the replay sample set e m ,m∈1,2,…,t is updated according to e m ←e m-1 ∪s(Y m ),(t=1 when e m-1 =φ) is constantly updated, and the replay sample set e m ={s(Y1),s(Y2),…,s(Y m )} is constructed to save representative background samples in historical tasks; Step 3.2, judge whether the current arrived task is the first task, i.e. Y t = Y1 is true; if true, the training set is the background set, i.e. X_train_AD1 = B1; if not true, the union of the replay sample set e m and the background set as the training set, i.e. X_train_AD m = B m ∪e m , m e 1, 2, …, t.

4. The hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with continual learning of a generative adversarial network according to claim 1, characterized in that, The step 5 specifically includes the following steps: Step 5.1, constructing a generator loss function: when t = 1, the loss function of the generator is represented as: When t≠1, the loss function of the generator is represented as: wherein, E represents the mathematical expectation, p(X_train_AD m ) represents the data distribution of the training set, E1(X_train_AD m ) represents the output of the encoder 1, G(E1(X_train_AD m )) represents the reconstructed image output by the decoder, Diff(G(E1(X_train_AD m ))) represents the enhanced reconstructed sample set obtained by performing the differentiable data augmentation on the reconstructed image output by the decoder, E2(G(E1(X_train_AD m ))) represents the latent feature obtained by passing the reconstructed image through the encoder 2, D(Diff(G(E1(X_train_AD m )))) represents the prediction result of the discriminator output on the real probability of the reconstructed sample, and MSE represents the mean square error loss function. Step 5.2, Constructing the discriminator loss function: the loss function L of the discriminator D is represented as: wherein, E denotes the mathematical expectation, p(X_train_AD m ) denotes the data distribution of the background sample training set, E1(X_train_AD m ) denotes the output of the encoder 1, G(E1(X_train_AD m )) denotes the reconstructed image output by the decoder, Diff(G(E1(X_train_AD m ))) denotes the enhanced reconstructed sample set obtained by performing differentiable data augmentation on the reconstructed image output by the decoder, and D(Diff(X_train_AD m )) denotes the prediction result obtained by sending the enhanced background sample set obtained by performing differentiable data augmentation on the background sample set into the discriminator. Step 5.3, alternately training the generator and the discriminator using the loss function of the generator in step 5.1 and the loss function of the discriminator in step 5.2: Step 5.3.1, training the generator: input the training set into the generator for nonlinear mapping to generate the reconstructed background sample G(E1(X_train_AD m )), and use the discriminator to perform nonlinear mapping on the reconstructed background sample G(E1(X_train_AD m )) to output the prediction result of the true or false of the reconstructed background sample, denoted as true_G(E1(X_train_AD m )); use the mean square error loss function to calculate the loss value L G of the prediction result of the true or false of the reconstructed background sample; and use the loss value L G to perform backpropagation training on the generator; Step 5.3.2, training discriminator: using the discriminator to perform nonlinear mapping on the training set, output the prediction result of the true and false of the training set sample, recorded as true_x, using the mean square error loss to calculate the prediction result loss value L of the true and false of the training set sample D_true ; using the discriminator to perform nonlinear mapping on the reconstructed background sample, output the prediction result of the reconstructed background sample, recorded as false_x, using the mean square error loss to calculate the prediction result loss value L of the true and false of the training set D_false , add the values of L D_true and L D_false together as the total discriminator loss L D , use L D to train the discriminator by back propagation; at this time, the model training is completed.

5. The hyperspectral cross-domain robust anomaly detection method based on visual transformation combined with continual learning of a generative adversarial network according to claim 1, characterized in that, The step 6 specifically includes the following steps: Step 6.1, Construct test set Y_test1 = Y1, Y_test2 = Y2,..., Y_test m = Y m , m = 1, 2,..., t, including all task data before the current task; Construct the background set of the test set T_B1 = B1, T_B2 = B2,..., T_B m = B m , m = 1, 2,..., t; Step 6.

2. Using the trained generator, the background set T_B of the test set of step 6.1 is processed m The background samples are reconstructed, and the loss value of the reconstructed background samples and the data in Y_test is calculated using the mean square error loss to obtain the anomaly probability of each pixel point in the reconstructed image, and the anomaly probability expression of each pixel point is: M(i,j) m m m m (E1(B(i,j) m ))),(0≤i≤M,0≤j≤N), wherein G m represents the generator trained for the mth task, and the first m-1 tasks are tested in turn, Y_test(i,j) m represents the pixel point in the hyperspectral image to be tested for the mth task.​​​ Step 6.3, increasing the value of m by 1, and returning to execute step 2.

6. A hyperspectral cross-domain robust anomaly detection system based on view transformation combined with a generative adversarial network for continual learning, characterized in that, Including: Continuous learning task construction module: constructing t tasks that meet the continuous learning of hyperspectral images; Spectral feature fusion module: performing spectral feature fusion on the task data constructed in the continuous learning task construction module; Coarse background mask matrix construction module: performing background screening preprocessing on the results of the spectral feature fusion module; The implementation process of the spectral feature fusion module and the coarse background mask matrix construction module is as follows: Given a hyperspectral image HSI containing M×N pixels, each pixel having O channels, use PCA to reduce the channel dimension to C dimensions (C≤O), represented as follows: in, Y represents m The spectral vector with coordinates (i,j) in the middle, Y m Divided into abnormal sample set A m and background set B m , i.e. Y m =[A m B m A m ∪B m =Y m , After capturing representative features of the background set, utilize The reconstructed background sample is obtained, so that The distribution of the background set is similar to that of the background set, where G represents the generator in the view transform generative adversarial network, B represents the background set, and m represents the number of current tasks, m∈1,2,…,t; For Y m The distance between adjacent pixels in the local region is calculated using the cosine similarity, which is expressed as: Pixels with a similarity greater than a set threshold are retained as background samples, and other samples are marked as abnormal samples, i.e. A rough background mask matrix is obtained Where τ is the threshold of the similarity of background pixels. Y m The block-based spectral-spatial feature fusion is performed on m e 1, 2,..., t to enhance spatial information, and an average vector of a local region with a size of w x w is calculated: wherein r is an index of a spectral vector in each w x w local region, and the average vector is spliced with the spectral vector to enhance spatial information: wherein, denotes a vector containing original spectral information and local spatial information of the (i, j) coordinate, denotes a splicing operation, and an HSI spectral-spatial feature matrix is obtained: with a coarse background mask matrix K m , m e 1, 2, …, t as an index, the empty spectrum feature matrix F m , m e 1, 2, …, t is divided into an anomaly sample set A m and a background set B m , the anomaly sample set A m is expressed as: A m = The background set is expressed as: Where a i represents the i-th sample in the anomaly sample set A m , n a represents the number of samples in the anomaly sample set A m , b i represents the i-th sample in the background set, n b represents the number of samples in the background set, B represents the background set, and m represents the current task number, m e 1, 2, …, t. Sample replay module: using the data in the spectral feature fusion module, obtaining a replay sample set using a clustering algorithm, and taking the union of the replay sample set and the background set in the coarse background mask matrix construction module, and sending it into the model of the generator module and the discriminator module; Generator module: building an encoder 1, a decoder and an encoder 2 connected in series to form a generator; Discriminator module: building a discriminator including a multi-scale convolutional layer, a view transformation layer and an output layer to assist the training of the generator; The specific implementation process of the generator module and the discriminator module is as follows: Building a generator: the generator includes an encoder 1, a decoder and an encoder 2 connected in series, the encoder 1 and the encoder 2 have the same structure, and the decoder is trained with the aid of the encoder 1 and the encoder 2: First, an encoder 1 is constructed using three fully connected layers and a depth separable convolutional layer, and the data of the training set in step 3 is mapped into a latent space using the encoder 1; Then, a decoder is constructed using four fully connected layers, and the data in the latent space is mapped into a spectral vector of the same size as the preprocessed background set using the decoder; Then, an encoder 2 is constructed using three fully connected layers and a depth separable convolutional layer, and the spectral vector is mapped to obtain a latent space vector using the encoder 2; Building a discriminator with a ViT network structure: the discriminator includes a multi-scale convolutional layer, a view transformation layer and an output layer arranged in order according to the data logic processing sequence, and the discriminator is used to assist the training of the generator: First, the multi-scale convolutional layer includes three one-dimensional convolutional layers with different convolution kernel sizes connected in parallel and one one-dimensional convolutional layer, and the multi-scale convolutional layer is used to obtain an intermediate vector fused with multi-level features; The view transformation layer includes an L2 multi-head self-attention, a normalization layer and a residual connection structure, and the intermediate vector is mapped to an attention vector containing key features by using the view transformation layer; the view transformation layer can be represented as: where LN is a layer normalization function, denoted as: i denotes one of the neurons of the layer, μ and σ denote the mean and standard deviation of the output of the neuron in the batch, respectively, ∈ is a small constant, and L2_MHSA denotes L2 multi-head self-attention. Then, an output layer is formed by using a Sigmoid layer, the attention vector is mapped by using the output layer, and a true or false judgment result is obtained; Finally, the generator and the discriminator are cascaded to obtain a ViT generative adversarial network model; The generator and the discriminator loss module: the generator module and the discriminator module are trained to obtain a trained ViT generative adversarial network model; The specific implementation process of the generator and the discriminator loss module is as follows: When t = 1, the generator and the discriminator of the ViT generative adversarial network model are alternately trained by using the preprocessed background set, a trained ViT generative adversarial network model is obtained, and a test module is executed; Otherwise, the generator and the discriminator of the ViT generative adversarial network model are alternately trained by using the preprocessed background set, and the F-norm regularization loss term is constructed for the generator of the current task and the last task, and the test module is executed; Definition L f denotes the F-norm regularization loss, which forces the reconstructed image of the t-1th task to be close to the distance of the tth task, for the tth task, assuming the replay sample is in the current generator G t The generated reconstructed sample is denoted as Z t The output of each data point is denoted as The generated reconstructed sample of the generator at the T-1th task is denoted as Z t-1 According to the output denoted as Z t The covariance matrix of the reconstructed sample generated by the current generator is calculated, denoted as: Wherein, b represents batch_size; According to Z t-1 The position (i, J) in which the middle pixel is zero in Z t The pixel matrix in Z t [mask] is recorded as num, and the corresponding pixel point number is calculated. The category-related covariance matrix is: The cumulative covariance matrix is updated as: P_W+ = P_X+ P_C-I, I represents a unit matrix with the same dimension as P_C; Finally, the F-norm regularization loss of each task t is obtained as: L f = λ * ||P_W - I|| F ; The test module: the trained ViT generative adversarial network model in the generator and the discriminator loss module is used for testing.

7. A hyperspectral cross-domain robust anomaly detection device based on view transformation combined with a generative adversarial network for continual learning, characterized in that, Including: Memory: for storing a computer program for implementing the method for performing continuous learning of hyperspectral cross-domain robust anomaly detection based on view transformation combined with a generative adversarial network according to any one of claims 1-5; Processor: for executing the computer program to implement the method for performing continuous learning of hyperspectral cross-domain robust anomaly detection based on view transformation combined with a generative adversarial network according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the method for performing continuous learning of hyperspectral cross-domain robust anomaly detection based on view transformation combined with a generative adversarial network according to claims 1-5.

Images