A hyperspectral anomaly detection method and system based on capsule differential adversarial network

Abstract

The present invention relates to a hyperspectral anomaly detection method and system based on a capsule differential adversarial network. The method comprises: obtaining a hyperspectral image to be tested; inputting the hyperspectral image to be tested into a pre-trained generative adversarial network to obtain a reconstructed image corresponding to the hyperspectral image to be tested and the anomaly probability of each pixel in the reconstructed image; the generative adversarial network is trained based on a background sample training set, and the generative adversarial network includes a cascaded generator and a discriminator, the generator is used to generate reconstructed image data with a distribution similar to its input image data, and the discriminator is used to determine the authenticity of its input data; the generator and the discriminator are both one-dimensional capsule network structures. The method of the present invention utilizes a pre-trained generative adversarial network to detect anomalies in the hyperspectral image to be tested. Compared with traditional convolutional neural networks, the generative adversarial network can effectively alleviate the problem of overfitting and is better suited for hyperspectral anomaly detection scenarios with unbalanced data.

Description

Technical Field

[0001] The present invention belongs to the technical field of image information processing, and in particular relates to a hyperspectral anomaly detection method and system based on a capsule differential adversarial network. Background Art

[0002] Hyperspectral images, acquired by spectral imagers, are high-dimensional images containing hundreds of spectral channels. Therefore, each pixel represents a continuous spectral curve, allowing specific wavelengths to be selected or extracted as needed to highlight target features. Hyperspectral imagers simultaneously detect both the two-dimensional geometric spatial information and the one-dimensional spectral information of a target. As a result, hyperspectral data has an "image cube" structure, embodying the characteristics and advantages of "image-spectrum integration."

[0003] Due to the unique physical and chemical properties of different regions on the Earth's surface, which can emit, reflect, and absorb different electromagnetic radiation, the spectral discrimination information in hyperspectral images can be used to identify different materials. With its rich spectral, spatial, and temporal information, hyperspectral images have been widely used in many fields, including mineral exploration, environmental monitoring, precision agriculture, and national defense. Hyperspectral anomaly detection aims to identify objects that differ significantly from their surroundings in space or spectrum without any prior knowledge of the background or target, and is one of the most active research areas in hyperspectral imaging.

[0004] In recent years, deep learning algorithms have gained increasing attention in hyperspectral image processing. Convolutional neural networks have demonstrated excellent performance in deep learning models. However, these approaches, directly based on convolutional neural networks, have limited ability to model image feature locations. If the input data is rotated, tilted, or changes in any other direction, the convolutional neural network may exhibit poor recognition performance, resulting in poor hyperspectral anomaly detection results. Summary of the Invention

[0005] To address the above-mentioned problems in the prior art, the present invention provides a hyperspectral anomaly detection method and system based on a capsule differential adversarial network. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a hyperspectral anomaly detection method based on a capsule differential adversarial network, comprising:

[0007] Acquire the hyperspectral image to be measured;

[0008] Inputting the hyperspectral image to be tested into a pre-trained generative adversarial network to obtain a reconstructed image corresponding to the hyperspectral image to be tested and the abnormality probability of each pixel in the reconstructed image;

[0009] Among them, the generative adversarial network is obtained by training based on a background sample training set. The generative adversarial network includes a cascaded generator and a discriminator. The generator is used to generate reconstructed image data with a distribution similar to its input image data, and the discriminator is used to judge the authenticity of its input data; both the generator and the discriminator are one-dimensional capsule network structures.

[0010] In one embodiment of the present invention, the generator includes a cascaded encoder and decoder, the encoder is used to map the input raw data into a latent space, and the decoder is used to map the data in the latent space into a spectral vector of the same size as the raw data; wherein,

[0011] The encoder includes a plurality of first fully connected layers, a first primary capsule layer, and a first dense capsule layer cascaded in sequence, wherein the first primary capsule layer and the first dense capsule layer are connected via a dynamic routing strategy;

[0012] The decoder includes several cascaded second fully connected layers.

[0013] In one embodiment of the present invention, the discriminator includes a plurality of one-dimensional convolutional layers, a second primary capsule layer, and a second dense capsule layer cascaded in sequence, and the second primary capsule layer and the second dense capsule layer are connected via a dynamic routing strategy.

[0014] In one embodiment of the present invention, the training method of the generative adversarial network includes:

[0015] Obtain background sample training set;

[0016] Inputting the background sample training set into the generative adversarial network for training and calculating the loss function;

[0017] The generative adversarial network is optimized and updated according to the loss function to obtain a trained generative adversarial network.

[0018] In one embodiment of the present invention, obtaining a background sample training set includes:

[0019] Acquire a hyperspectral anomaly detection dataset, where the hyperspectral anomaly detection dataset includes a plurality of hyperspectral images;

[0020] Calculate the cosine similarity between each pixel and its adjacent pixels in the hyperspectral image, and record the coordinate set of pixels with cosine similarity greater than or equal to 0.99 as the background sample coordinate set X index ;

[0021] Performing spatial-spectral mean fusion processing on each pixel in the hyperspectral image to obtain a fused data set dataSpeSpa;

[0022] Extract coordinates from the fused dataset dataSpeSpa in the background sample coordinate set X index The pixels in the background sample training set X BKG .

[0023] In one embodiment of the present invention, the background sample training set is input into the generative adversarial network for training, and the loss function is calculated, including:

[0024] The background sample training set is input into the generator of the generative adversarial network for training, and the loss function L of the generator is calculated. G , the generator outputs a reconstructed image set corresponding to its input, and at the same time, performs differentiable data enhancement processing on the reconstructed image set to obtain an enhanced reconstructed image set;

[0025] Performing differentiable data enhancement processing on the background sample training set to obtain an enhanced background sample set;

[0026] The enhanced background sample set and the enhanced reconstructed image set are input into the discriminator of the generative adversarial network for training, and the loss function L of the discriminator is calculated. D .

[0027] In one embodiment of the present invention, the loss function L of the generator is G for:

[0028]

[0029] in, Where E represents the mathematical expectation, x represents the background sample in the background sample training set, and p(X BKG ) represents the data distribution of the background sample training set, E(x) represents the output of the encoder, G(E(x)) represents the reconstructed image output by the decoder, Diff(G(E(x)) represents the enhanced reconstructed image after differentiable data enhancement processing is performed on the reconstructed image, D(Diff(G(E(x)))) represents the prediction result of the true probability of the input sample Diff(G(E(x))) output by the discriminator, and MSE represents the mean square error loss function.

[0030] In one embodiment of the present invention, the loss function L of the discriminator is D for:.

[0031]

[0032] in, Where E represents the mathematical expectation, x represents the background sample in the background sample training set, and p(X BKG) represents the data distribution of the background sample training set, E(x) represents the output of the encoder, G(E(x)) represents the reconstructed image output by the decoder, Diff(G(E(x)) represents the enhanced reconstructed image after differentiable data enhancement processing is performed on the reconstructed image, D(Diff(G(E(x)))) represents the prediction result of the discriminator output on the true probability of the input sample Diff(G(E(x))), Diff(x) represents the enhanced background sample after differentiable data enhancement processing is performed on the background sample, and D(Diff(x)) represents the prediction result of the discriminator output on the true probability of the input sample Diff(x).

[0033] In one embodiment of the present invention, the expression for the abnormality probability of each pixel in the reconstructed image is:

[0034] Pic (i,j) =MSE(X (i,j) ,G(E(X (i,j) ))),(0≤i≤m,0≤j≤n)

[0035] Among them, MSE represents the mean square error loss function, m represents the number of rows of the hyperspectral image data to be tested, n represents the number of columns of the hyperspectral image data to be tested, and X (i,j) Represents the pixel point in the hyperspectral image to be measured, E(X (i,j) ) represents the encoder's response to input X (i,j) The output of G(E(X (i,j) )) represents the pixel point in the reconstructed image corresponding to the hyperspectral image to be measured.

[0036] The present invention provides a hyperspectral anomaly detection system based on a capsule differential adversarial network, comprising:

[0037] Image input module, used to input the hyperspectral image to be measured;

[0038] A detection module is configured to detect the input hyperspectral image to be tested based on a pre-trained generative adversarial network stored therein, and obtain a reconstructed image corresponding to the hyperspectral image to be tested and an abnormality probability of each pixel in the reconstructed image;

[0039] Among them, the generative adversarial network is obtained by training based on a background sample training set. The generative adversarial network includes a cascaded generator and a discriminator. The generator is used to generate reconstructed image data with a distribution similar to its input image data, and the discriminator is used to judge the authenticity of its input data; both the generator and the discriminator are one-dimensional capsule network structures.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The hyperspectral anomaly detection method based on capsule differential adversarial network of the present invention uses a pre-trained generative adversarial network to detect anomalies in the hyperspectral image to be tested. Compared with traditional convolutional neural networks, generative adversarial networks can effectively alleviate the problem of overfitting. In anomaly detection data, there are usually only a small number of outliers. Generative adversarial networks are better suited for scenarios with unbalanced data.

[0042] 2. The hyperspectral anomaly detection method based on capsule differential adversarial network of the present invention generates an adversarial network with a capsule network structure, which can preserve the posture parameters of the sample and improve the generalization performance of the model.

[0043] 3. The hyperspectral anomaly detection method based on capsule differential adversarial network of the present invention can improve the detection accuracy of the generative adversarial network by processing the spectral information and spatial information of the hyperspectral image during the training process of the generative adversarial network, effectively integrating and making full use of the spatial spectrum information.

[0044] 4. The hyperspectral anomaly detection method based on capsule differential adversarial network of the present invention uses differentiable data augmentation to process samples during the training process of the generative adversarial network, which can further improve the detection performance of the network.

[0045] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of a hyperspectral anomaly detection method based on capsule differential adversarial network provided by an embodiment of the present invention;

[0047] Figure 2 1 is a schematic diagram of the structure of a generative adversarial network provided by an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the training process of a generative adversarial network provided by an embodiment of the present invention;

[0049] Figure 4 This is a diagram of anomaly detection results in an urban scenario provided by an embodiment of the present invention;

[0050] Figure 5 This is a diagram showing anomaly detection results in a beach scenario provided by an embodiment of the present invention;

[0051] Figure 6 This is a diagram of anomaly detection results in an airport scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a hyperspectral anomaly detection method and system based on a capsule differential adversarial network proposed in accordance with the present invention, in conjunction with the accompanying drawings and specific embodiments.

[0053] The aforementioned and other technical contents, features, and effects of the present invention are clearly presented in the following detailed description of the specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the intended purpose can be obtained. However, the accompanying drawings are provided for reference and illustration purposes only and are not intended to limit the technical solutions of the present invention.

[0054] Example 1

[0055] See Figure 1 , Figure 1 This is a flow chart of a hyperspectral anomaly detection method based on a capsule differential adversarial network provided by an embodiment of the present invention. As shown in the figure, the hyperspectral anomaly detection method based on a capsule differential adversarial network of this embodiment includes:

[0056] Step 1: Obtain the hyperspectral image to be measured;

[0057] Step 2: Input the hyperspectral image to be tested into the pre-trained generative adversarial network to obtain the reconstructed image corresponding to the hyperspectral image to be tested and the abnormal probability of each pixel in the reconstructed image.

[0058] In this embodiment, the generative adversarial network is obtained by training based on a background sample training set.

[0059] Specifically, a generative adversarial network (GAN) is a generative model consisting of a cascade of generators and discriminators. The generator is used to generate reconstructed image data with a distribution similar to its input image data, while the discriminator is used to determine the authenticity of its input data, that is, to distinguish whether the input data is real training data or reconstructed data generated by the generator. The discriminator outputs a prediction value for the authenticity of its input sample. A larger value indicates a more authentic sample, while a smaller value indicates a more erroneous sample.

[0060] In a generative adversarial network, the generator and the discriminator compete with each other in an adversarial manner. Therefore, the generative adversarial network can generate realistic reconstructed data (pseudo data) that can be used for background pixel reconstruction in anomaly detection.

[0061] In this embodiment, both the generator and the discriminator are one-dimensional capsule network structures. Capsule networks are a new type of network structure that addresses the shortcomings of convolutional neural networks and enhances recognition capabilities. Capsule networks encode data relationships into vectors (rather than scalars), using vector neurons instead of scalar neurons. The vector's modulus represents the probability of a feature's presence, and its direction represents the feature's posture information (position, color, orientation, etc.).

[0062] The hyperspectral anomaly detection method based on capsule differential adversarial network in this embodiment uses a pre-trained generative adversarial network to detect anomalies in the hyperspectral image to be tested. Compared with traditional convolutional neural networks, generative adversarial networks can effectively alleviate the problem of overfitting. In anomaly detection data, there are usually only a small number of outliers. Generative adversarial networks are better suited for scenarios with unbalanced data.

[0063] Further, see Figure 2 , Figure 2 This is a structural diagram of a generative adversarial network provided by an embodiment of the present invention. As shown in the figure, in the generative adversarial network of this embodiment, the generator includes a cascaded encoder and decoder. The encoder is used to map the input original data to a latent space, and the decoder is used to map the data in the latent space to a spectral vector of the same size as the original data.

[0064] To improve the network's modeling capabilities, the encoder in this implementation is designed as a one-dimensional capsule network structure. Specifically, the encoder includes several cascaded first fully connected layers, first primary capsule layers, and first dense capsule layers. The first primary capsule layers and first dense capsule layers are connected via a dynamic routing strategy. The decoder includes several cascaded second fully connected layers.

[0065] Similar to the encoder of the generator, due to the excellent feature modeling capabilities of the capsule network, the discriminator is also designed as a one-dimensional capsule network structure in this implementation. Unlike the encoder of the generator which contains a fully connected layer, the discriminator includes several one-dimensional convolutional layers, a second primary capsule layer, and a second dense capsule layer cascaded in sequence. The second primary capsule layer and the second dense capsule layer are connected by a dynamic routing strategy.

[0066] Furthermore, the constructed Figure 2 For a detailed description of the training method of the generative adversarial network shown in Figure 3 , Figure 3 Schematic diagram of the training process of a generative adversarial network provided by an embodiment of the present invention. As shown in the figure, the training method of the generative adversarial network of this embodiment includes:

[0067] Step a: Obtain background sample training set;

[0068] Since the goal of anomaly detection is to identify objects that are significantly different from their surrounding background in space or spectrum without any prior knowledge of the background and target, in order for the network to learn the distribution of background pixels, it is first necessary to perform a preliminary background pixel screening based on spectral information.

[0069] Specifically, step a includes:

[0070] Step a1: Obtain a hyperspectral anomaly detection dataset, which includes several hyperspectral images;

[0071] In this embodiment, the data of the hyperspectral image is a data cube of size (m, n, channels), where m represents the number of rows of the hyperspectral image data, n represents the number of columns of the hyperspectral image data, and channels represents the number of spectral channels of the hyperspectral image data. Therefore, each pixel point is a spectral curve with a length of channels.

[0072] For example, the data of the hyperspectral image abu-airport-1 is a data cube with a size of (100, 100, 205), where the two 100s represent the number of rows and columns of the abu-airport-1 data, and 205 represents the number of channels of the abu-airport-1 data. Therefore, each pixel point is a spectral curve with a length of 205.

[0073] Alternatively, the hyperspectral anomaly detection dataset can be obtained from existing public datasets.

[0074] Step a2: Calculate the cosine similarity between each pixel and its adjacent pixels in the hyperspectral image, and record the coordinate set of pixels with cosine similarity greater than or equal to 0.99 as the background sample coordinate set X index ;

[0075] It should be noted that in anomaly detection tasks, the spectral curves of anomaly pixels and background pixels are significantly different, while the spectral curves of adjacent background pixels are less different. Therefore, by calculating the cosine similarity between each pixel in the hyperspectral image and its adjacent pixels, the type of pixel (i.e., background pixel or anomaly pixel) can be determined based on the size of the cosine similarity.

[0076] Specifically, in this embodiment, if the cosine similarity is less than 0.99, it is considered that the pixel is significantly different from its adjacent pixels and is considered an abnormal pixel. If the cosine similarity is greater than or equal to 0.99, the coordinates of the pixel are added to the background sample coordinate set X. index .

[0077] Assume X index There are N elements in , then For example, for the hyperspectral image abu-airport-1, its background sample coordinate set

[0078] Step a3: Perform spatial-spectral mean fusion processing on each pixel in the hyperspectral image to obtain the fused dataset dataSpeSpa;

[0079] Specifically, in this example, a spatial block of size (a, a, channels) is defined, centered around each pixel in the hyperspectral image. Global average pooling is then performed to obtain one-dimensional data of size (1, 1, channels) that incorporates spatial information. This one-dimensional data is then concatenated with the original spectrum to obtain new data of size (1, 1, 2 channels). This operation is performed on each pixel in the hyperspectral image, resulting in a fused dataset (dataSpeSpa) of size (m, n, 2 channels) that combines its own spectral information with the spatial information of neighboring pixels.

[0080] For example, taking the data (100,100,205) of the hyperspectral image abu-airport-1 as an example, we define a spatial block of size (3,3,205) with each pixel as the center, and perform global average pooling to obtain a one-dimensional data of size (1,1,205) that integrates spatial information. This one-dimensional data is then concatenated with the original spectrum to obtain a spectral vector of size (1,1,410). This operation is performed on each pixel in the hyperspectral image, and the fused dataset dataSpeSpa that integrates its own spectral information and the spatial information of adjacent pixels is obtained. Its size is (100,100,410).

[0081] During the training process of the generative adversarial network, the detection accuracy of the generative adversarial network can be improved by processing the spectral information and spatial information of the hyperspectral images of the training samples, effectively fusing and making full use of the spatial spectrum information.

[0082] Step a4: Extract the coordinates in the background sample coordinate set X from the fusion dataset dataSpeSpa index The pixels in the background sample training set X BKG .

[0083] In this embodiment,

[0084] Step b: Input the background sample training set into the generative adversarial network for training and calculate the loss function;

[0085] Specifically, it includes:

[0086] Step b1: Input the background sample training set into the generator of the generative adversarial network for training and calculate the generator's loss function L G, the generator outputs the reconstructed image set corresponding to its input, and at the same time, performs data enhancement processing on the reconstructed image set to obtain an enhanced reconstructed image set;

[0087] Specifically, first, the background sample training set X BKG The background sample x in is input into the encoder of the generator. The encoder based on the one-dimensional capsule network structure maps the background spectrum vector to a 200-dimensional potential vector, which is recorded as E(x).

[0088] Secondly, after mapping by the encoder, the decoder reconstructs the latent vector into a 2-channels-dimensional reconstructed image G(E(x)).

[0089] It should be noted that the reconstructed image G(E(x)) obtained is input into the discriminator after differentiable data enhancement processing, and the enhanced reconstructed image after differentiable data enhancement processing is recorded as Diff(G(E(x))), and the discriminator outputs the predicted result of the true probability of the input sample Diff(G(E(x))) as D(Diff(G(E(x)))).

[0090] Finally, the network loss is calculated. In this embodiment, the network loss includes two parts: reconstruction loss and prediction loss.

[0091] Specifically, the mean square error loss function is used to calculate the loss value between the input background sample x and the reconstructed image G(E(x)) generated by the generator.

[0092] Since the generator and discriminator of the generative adversarial network are trained in a game-playing manner, when training the generator, the generator deceives the discriminator into thinking that the reconstructed image is a real sample, so the gradient descent algorithm is used to minimize the loss (-D(Diff(G(E(x))))).

[0093] The above two losses are used to perform backpropagation training on the generative adversarial network.

[0094] In this embodiment, the generator's loss function L G Expressed as:

[0095]

[0096] in, Where E represents the mathematical expectation, x represents the background sample in the background sample training set, and p(X BKG) represents the data distribution of the background sample training set, E(x) represents the output of the encoder, G(E(x)) represents the reconstructed image output by the decoder, Diff(G(E(x)) represents the enhanced reconstructed image after differentiable data enhancement processing is performed on the reconstructed image, D(Diff(G(E(x)))) represents the prediction result of the true probability of the input sample Diff(G(E(x))) output by the discriminator, and MSE represents the mean square error loss function.

[0097] Step b2: Perform differentiable data enhancement processing on the background sample training set to obtain an enhanced background sample set;

[0098] In this embodiment, in order to prevent the discriminator of the generative adversarial network from overfitting and improve the performance of the generative adversarial network, differentiable data augmentation is used on the training set. This differentiable data augmentation method is not only used for the background sample training set, but also for the reconstructed image set generated by the generator to avoid destroying the dynamic balance between the generator and the discriminator.

[0099] Specifically, the differentiable data enhancement method is recorded as Diff, then, for the background sample training set X BKG The enhanced background sample after differentiable data enhancement processing of the background sample x in is recorded as Diff(x).

[0100] It should be noted that the differentiable data enhancement method Diff includes data enhancement for color (eg, changes in image brightness, saturation, and / or contrast) and data enhancement for shift (eg, movement along the X and / or Y directions).

[0101] In the process of training the generative adversarial network, using differentiable data to enhance training samples can further improve the detection performance of the network.

[0102] Step b3: Input the enhanced background sample set and the enhanced reconstructed image set into the discriminator of the generative adversarial network for training, and calculate the loss function L of the discriminator D .

[0103] Specifically, first, the enhanced background sample Diff(x) is input into the discriminator for nonlinear mapping, and the prediction result D(Diff(x)) of the true probability of Diff(x) is output.

[0104] Secondly, the enhanced reconstructed image Diff(G(E(x))) is input into the discriminator for nonlinear mapping, and the prediction result D(Diff(G(E(x)))) of the true probability of Diff(G(E(x))) is output.

[0105] Finally, the network loss is calculated. In this embodiment, the loss here includes two parts: the prediction loss of the enhanced background sample and the prediction loss of the enhanced reconstructed image.

[0106] When training the discriminator, in order to enhance the discriminative ability of the discriminator, the gradient descent algorithm is used to minimize the loss (-D(Diff(x))) and D(Diff(G(E(x)))), and the above two losses are used to perform backpropagation training on the generative adversarial network.

[0107] In this embodiment, the loss function L of the discriminator is D Expressed as:.

[0108]

[0109] in, Where E represents the mathematical expectation, x represents the background sample in the background sample training set, and p(X BKG ) represents the data distribution of the background sample training set, E(x) represents the output of the encoder, G(E(x)) represents the reconstructed image output by the decoder, Diff(G(E(x)) represents the enhanced reconstructed image after differentiable data enhancement processing is performed on the reconstructed image, D(Diff(G(E(x)))) represents the prediction result of the discriminator output on the true probability of the input sample Diff(G(E(x))), Diff(x) represents the enhanced background sample after differentiable data enhancement processing is performed on the background sample, and D(Diff(x)) represents the prediction result of the discriminator output on the true probability of the input sample Diff(x).

[0110] Step c: Optimize and update the generative adversarial network according to the loss function to obtain the trained generative adversarial network.

[0111] In this embodiment, the training is completed by iterative training 5000 times to obtain a trained generative adversarial network.

[0112] Furthermore, the trained generative adversarial network is used to perform anomaly detection on the hyperspectral images to be tested.

[0113] Because the GAN has learned the distribution of background pixels, it can output similar pseudo-data (reconstructed images) for input background pixels. However, for outlier pixels, the GAN output will differ significantly from the original data. Therefore, the reconstruction error for outlier pixels will be higher than for background pixels. By calculating the GAN's reconstruction error for a particular pixel, we can determine the probability that the pixel is an outlier.

[0114] In this embodiment, the generator in the generative adversarial network can reconstruct the spectral vector of each pixel in the input hyperspectral image to be tested. Since the trained generative adversarial network has mastered the data distribution of background pixels but has no knowledge of the data distribution of abnormal pixels, all pixels of the hyperspectral image to be tested are input into the generative adversarial network, and the x value of each pixel in the hyperspectral image to be tested is calculated. (i,j) , (0≤i≤m,0≤j≤n) and the pixel point G(E(x (i,j) )), it is clear that the loss value of background pixels is smaller, while the loss value of abnormal pixels is larger. Using this loss value as the probability that a pixel in the hyperspectral image to be tested is an abnormal pixel, anomaly detection is completed and the detection result is output. This detection result is the reconstructed image corresponding to the hyperspectral image to be tested and the abnormality probability of each pixel in the reconstructed image.

[0115] Specifically, the expression of the abnormal probability of each pixel in the reconstructed image is:

[0116] Pic (i,j) =MSE(X (i,j) ,G(E(X (i,j) ))),(0≤i≤m,0≤j≤n)

[0117] Among them, MSE represents the mean square error loss function, m represents the number of rows of the hyperspectral image data to be tested, n represents the number of columns of the hyperspectral image data to be tested, and X (i,j) Represents the pixel point in the hyperspectral image to be measured, E(X (i,j) ) represents the encoder's response to input X (i,j) The output of G(E(X (i,j) )) represents the pixel point in the reconstructed image corresponding to the hyperspectral image to be measured.

[0118] The hyperspectral anomaly detection method based on capsule differential adversarial network in this embodiment uses a pre-trained generative adversarial network to detect anomalies in the hyperspectral image to be tested. The generative adversarial network uses spatial-spectral mean fusion to fully utilize and extract the spatial-spectral information of the hyperspectral image, and adopts a differentiable data enhancement method during the network training process. Compared with traditional convolutional neural networks, it can effectively alleviate the problem of overfitting and improve the network detection accuracy. In addition, in the anomaly detection data, there are usually only a small number of abnormal points. The generative adversarial network is better suited for scenarios with imbalanced data.

[0119] Example 2

[0120] This embodiment verifies and illustrates the effect of the hyperspectral anomaly detection method based on capsule differential adversarial network in embodiment 1 through simulation experiments.

[0121] 1. Experimental Conditions

[0122] In this embodiment, the ABU (Airport-Beach-Urban) dataset used for experimental verification is divided into three scenes: urban, beach, and airport. The urban scene has 5 hyperspectral images, the beach scene has 4 hyperspectral images, and the airport scene has 4 hyperspectral images. The specific hyperspectral image data can be found in Table 1.

[0123] Table 1 Hyperspectral image data

[0124] Dataset Number of lines Number of columns Number of spectral bands abu-urban-1 100 100 204 abu-urban-2 100 100 207 abu-urban-3 100 100 191 abu-urban-4 100 100 205 abu-urban-5 100 100 205 abu-beach-1 150 150 188 abu-beach-2 100 100 193 abu-beach-3 100 100 188 abu-beach-4 150 150 102 abu-airport-1 100 100 205 abu-airport-2 100 100 205 abu-airport-3 100 100 205 abu-airport-4 100 100 191

[0125] The ABU dataset was tested under the same experimental conditions using the capsule differential adversarial network-based hyperspectral anomaly detection method proposed in implementation one and four existing hyperspectral anomaly detection methods: Reed–Xiaoli (RX), kernel-RX (KRX), local RX (LRX) and Collaborative Representation-based Detector (CRD). The AUC values of their detection results were analyzed and evaluated.

[0126] The AUC (Area Under Curve) value is defined as the area under the ROC curve and the coordinate axes. Since the ROC curve is generally above the line y = x, the AUC value range is between 0.5 and 1. The closer the AUC is to 1, the higher the authenticity of the detection method; when it is equal to 0.5, it is the lowest authenticity and has no application value.

[0127] 2. Analysis of experimental results

[0128] Please refer to Table 2, which shows the AUC scores of each detection method for each hyperspectral image detection result.

[0129] Table 2 AUC scores of each detection method for each hyperspectral image detection result

[0130] Dataset RX KRX LRX CRD This method abu-urban-1 0.9906 0.9906 0.98181 0.98013 0.9917 abu-urban-2 0.9945 0.9945 0.98016 0.94706 0.9993 abu-urban-3 0.9524 0.9524 0.93676 0.94726 0.9811 abu-urban-4 0.9896 0.9896 0.96898 0.95938 0.9967 abu-urban-5 0.9694 0.9694 0.96939 0.93383 0.9754 abu-beach-1 0.9731 0.9731 0.98256 0.97268 0.9912 abu-beach-2 0.9103 0.9103 0.91683 0.89812 0.9015 abu-beach-3 0.9997 0.9997 0.99608 0.98973 0.9999 abu-beach-4 0.9893 0.9893 0.9787 0.97113 0.9988 abu-airport-1 0.8213 0.8213 0.94432 0.88994 0.9331 abu-airport-2 0.8413 0.8413 0.9754 0.90564 0.9292 abu-airport-3 0.9283 0.9283 0.93761 0.83916 0.9673 abu-airport-4 0.9508 0.9508 0.98959 0.90017 0.9923

[0131] Please refer to Figure 4-Figure 6 , Figure 4 Figures 1 to 2 show anomaly detection results for an urban scene provided by an embodiment of the present invention. Figures (a) to (e) are the anomaly detection results for the hyperspectral images abu-urban-1 to abu-urban-5, respectively. From left to right, they are the pseudo-color image of the hyperspectral image, the groundtruth image, the detection result image of the method of the present invention, the RX detection result image, the KRX detection result image, the LRX detection result image, and the detection result image of the CRD method. Figure 5 These are the anomaly detection result diagrams for the beach scene provided by an embodiment of the present invention, where (a) to (d) are the anomaly detection result diagrams for the hyperspectral image abu-beach-1 to the hyperspectral image abu-beach-4, respectively. From left to right, they are the pseudo-color image of the hyperspectral image, the groundtruth image, the detection result diagram of the method of the present invention, the RX detection result diagram, the KRX detection result diagram, the LRX detection result diagram, and the detection result diagram of the CRD method. Figure 6 These are the anomaly detection result diagrams for the airport scene provided by an embodiment of the present invention, where (a) to (d) are the anomaly detection result diagrams for the hyperspectral image abu-airport-1 to the hyperspectral image abu-airport-4, respectively. From left to right, they are the pseudo-color image of the hyperspectral image, the groundtruth image, the detection result diagram of the method of the present invention, the RX detection result diagram, the KRX detection result diagram, the LRX detection result diagram, and the detection result diagram of the CRD method.

[0132] From Table 2 and Figure 4-Figure 6 It can be seen that on most data sets, the hyperspectral anomaly detection method based on capsule differential adversarial network of the present invention is superior to other traditional detection methods and has stable and excellent performance. Since the spectra of some abnormal pixels in the detection data are close to the spectra of background pixels, the algorithm may judge these abnormal pixels as background, or judge these background pixels as abnormal. In this case, the RX, LRX and CRD algorithms have a large number of abnormal missed detections and the abnormal detection rate is low. Although the KRX algorithm detects most abnormal pixels, many background pixels are detected as abnormal at the same time, that is, KRX has a high false alarm rate. The method of the present invention achieves a balance between detection rate and false alarm rate, reducing the false detection rate while detecting abnormal pixels as comprehensively as possible.

[0133] The hyperspectral anomaly detection method based on capsule differential adversarial network in this embodiment replaces the widely used convolutional neural network with capsule network and combines it with generative adversarial network to effectively utilize and integrate the discriminative information in hyperspectral data. At the same time, it uses differentiable data enhancement method to alleviate the overfitting of the discriminator and improve the performance of the generative adversarial network.

[0134] Example 3

[0135] Based on the same inventive concept, an embodiment of the present invention provides a hyperspectral anomaly detection system based on a capsule differential adversarial network, comprising:

[0136] Image input module, used to input the hyperspectral image to be measured;

[0137] The detection module is used to detect the input hyperspectral image to be tested based on the pre-trained generative adversarial network stored in it, and obtain the reconstructed image corresponding to the hyperspectral image to be tested and the abnormality probability of each pixel in the reconstructed image;

[0138] Among them, the generative adversarial network is obtained by training based on the background sample training set. The generative adversarial network includes a cascaded generator and a discriminator. The generator is used to generate reconstructed image data with a distribution similar to its input image data, and the discriminator is used to judge the authenticity of its input data. Both the generator and the discriminator are one-dimensional capsule network structures.

[0139] The hyperspectral anomaly detection system based on capsule differential adversarial network provided in the embodiment of the present invention can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0140] The present invention also provides a terminal device, which includes a processor and a memory, the memory being used to store a computer program, the computer program including program instructions, and the processor being used to execute the program instructions stored in a computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor of the embodiment of the present invention can implement any of the method steps described in the hyperspectral anomaly detection method based on the capsule differential adversarial network, or implement the functions implemented by any of the above-mentioned generative adversarial networks.

[0141] The present invention also provides a storage medium, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0142] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the method steps described in any of the above-mentioned hyperspectral anomaly detection methods based on capsule differential adversarial networks, or to implement the functions implemented by any of the above-mentioned generative adversarial networks.

[0143] As for the electronic device / storage medium / computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0144] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations are intended to cover non-exclusive inclusion, so that an article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of additional identical elements in the article or device comprising the element. Similar words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0145] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A hyperspectral anomaly detection method based on capsule differential adversarial network, characterized in that: include: Acquire the hyperspectral image to be measured; Inputting the hyperspectral image to be tested into a pre-trained generative adversarial network to obtain a reconstructed image corresponding to the hyperspectral image to be tested and the abnormality probability of each pixel in the reconstructed image; The generative adversarial network is obtained by training based on a background sample training set. The generative adversarial network includes a cascaded generator and a discriminator. The generator is used to generate reconstructed image data with a distribution similar to that of its input image data, and the discriminator is used to judge the authenticity of its input data. Both the generator and the discriminator are one-dimensional capsule network structures. The generator includes a cascaded encoder and decoder, the encoder is used to map the input raw data into a latent space, and the decoder is used to map the data in the latent space into a spectral vector of the same size as the raw data; The encoder includes a plurality of first fully connected layers, a first primary capsule layer, and a first dense capsule layer cascaded in sequence, wherein the first primary capsule layer and the first dense capsule layer are connected via a dynamic routing strategy; The decoder includes a plurality of cascaded second fully connected layers; The process of generating a background sample training set during the generative adversarial network training process includes: Acquire a hyperspectral anomaly detection dataset, where the hyperspectral anomaly detection dataset includes a plurality of hyperspectral images; Calculate the cosine similarity between each pixel and its adjacent pixels in the hyperspectral image, and record the coordinate set of pixels with a cosine similarity greater than or equal to 0.99 as the background sample coordinate set. ; Perform spatial spectrum mean fusion processing on each pixel in the hyperspectral image to obtain a fused data set dataSpeSpa ; In the fusion dataset dataSpeSpa Extract coordinates from the background sample coordinate set The pixels in constitute the background sample training set .

2. The hyperspectral anomaly detection method based on capsule differential adversarial network according to claim 1 is characterized in that: The discriminator includes a plurality of one-dimensional convolutional layers, a second primary capsule layer and a second dense capsule layer that are cascaded in sequence, and the second primary capsule layer and the second dense capsule layer are connected through a dynamic routing strategy.

3. The hyperspectral anomaly detection method based on capsule differential adversarial network according to claim 1 is characterized in that: The training method of the generative adversarial network includes: Obtain background sample training set; Inputting the background sample training set into the generative adversarial network for training and calculating the loss function; The generative adversarial network is optimized and updated according to the loss function to obtain a trained generative adversarial network.

4. The hyperspectral anomaly detection method based on capsule differential adversarial network according to claim 3 is characterized in that: Inputting the background sample training set into the generative adversarial network for training and calculating the loss function includes: Input the background sample training set into the generator of the generative adversarial network for training, and calculate the loss function of the generator , the generator outputs a reconstructed image set corresponding to its input, and at the same time, performs differentiable data enhancement processing on the reconstructed image set to obtain an enhanced reconstructed image set; Performing differentiable data enhancement processing on the background sample training set to obtain an enhanced background sample set; The enhanced background sample set and the enhanced reconstructed image set are input into the discriminator of the generative adversarial network for training, and the loss function of the discriminator is calculated. .

5. The hyperspectral anomaly detection method based on capsule differential adversarial network according to claim 4 is characterized in that: The loss function of the generator for: ， in, E stands for mathematical expectation. x represents the background samples in the background sample training set, represents the data distribution of the background sample training set, represents the output of the encoder, represents the reconstructed image output by the decoder, represents the enhanced reconstructed image after differentiable data enhancement processing is performed on the reconstructed image, Represents the discriminator output for the input sample The predicted results of the true probability, represents the mean squared error loss function.

6. The hyperspectral anomaly detection method based on capsule differential adversarial network according to claim 4 is characterized in that: The loss function of the discriminator for: ， in, E stands for mathematical expectation. x represents the background samples in the background sample training set, represents the data distribution of the background sample training set, represents the output of the encoder, represents the reconstructed image output by the decoder, represents the enhanced reconstructed image after differentiable data enhancement processing is performed on the reconstructed image, Represents the discriminator output for the input sample The predicted results of the true probability, represents the enhanced background sample after differentiable data enhancement processing is performed on the background sample, Represents the discriminator output for the input sample The predicted results of the true probability.

7. The hyperspectral anomaly detection method based on capsule differential adversarial network according to claim 1 is characterized in that: The expression of the abnormal probability of each pixel in the reconstructed image is: in, represents the mean square error loss function, m Indicates the number of rows of the hyperspectral image data to be measured, n Indicates the number of columns of the hyperspectral image data to be measured, represents the pixel point in the hyperspectral image to be measured, Represents the encoder about the input The output, Represents the pixel point in the reconstructed image corresponding to the hyperspectral image to be measured.

8. A hyperspectral anomaly detection system based on capsule differential adversarial network, characterized by: include: Image input module, used to input the hyperspectral image to be measured; A detection module is configured to detect the input hyperspectral image to be tested based on a pre-trained generative adversarial network stored therein, and obtain a reconstructed image corresponding to the hyperspectral image to be tested and an abnormality probability of each pixel in the reconstructed image; The generative adversarial network is obtained by training based on a background sample training set. The generative adversarial network includes a cascaded generator and a discriminator. The generator is used to generate reconstructed image data with a distribution similar to that of its input image data, and the discriminator is used to judge the authenticity of its input data. Both the generator and the discriminator are one-dimensional capsule network structures. The generator includes a cascaded encoder and decoder, the encoder is used to map the input raw data into a latent space, and the decoder is used to map the data in the latent space into a spectral vector of the same size as the raw data; The encoder includes a plurality of first fully connected layers, a first primary capsule layer, and a first dense capsule layer cascaded in sequence, wherein the first primary capsule layer and the first dense capsule layer are connected via a dynamic routing strategy; The decoder includes a plurality of cascaded second fully connected layers; The process of generating a background sample training set during the generative adversarial network training process includes: Acquire a hyperspectral anomaly detection dataset, where the hyperspectral anomaly detection dataset includes a plurality of hyperspectral images; Calculate the cosine similarity between each pixel and its adjacent pixels in the hyperspectral image, and record the coordinate set of pixels with a cosine similarity greater than or equal to 0.99 as the background sample coordinate set. ; Perform spatial spectrum mean fusion processing on each pixel in the hyperspectral image to obtain a fused data set dataSpeSpa ; In the fusion dataset dataSpeSpa Extract coordinates from the background sample coordinate set The pixels in constitute the background sample training set .

Images