Hyperspectral remote sensing image anomaly detection method based on deep learning

Abstract

The present invention relates to the technical field of image processing, and in particular to a hyperspectral remote sensing image anomaly detection method based on deep learning. The method comprises: performing normalization preprocessing on a hyperspectral remote sensing image, so as to obtain an original hyperspectral image; inputting the original hyperspectral image into a low-rank background feature extraction network, so as to generate a low-rank background feature; inputting the low-rank background feature into a backbone feature extraction network, so as to obtain a backbone feature; inputting the backbone feature into a feature convolutional network, so as to obtain a convolution feature; inputting the convolution feature into an up-sampling network, so as to obtain an up-sampling feature; inputting the up-sampling feature into a feature convolutional sub-network, so as to obtain a reconstructed background feature map; and subtracting the reconstructed background feature map from the original hyperspectral image, so as to obtain a reconstructed error map, and using an adaptive weighted loss function to suppress an abnormal reconstruction weight. The present invention aims to solve the problem whereby low-rank characteristics of a background are not fully utilized in existing hyperspectral anomaly detection, and the problem whereby existing algorithms still face challenges in separating the background and an anomaly.

Description

Anomaly detection method for hyperspectral remote sensing images based on deep learning Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for detecting anomalies in hyperspectral remote sensing images based on deep learning. Background Art

[0002] Hyperspectral images are three-dimensional images that utilize both imaging and spectral techniques. They contain not only two-dimensional spatial information but also one-dimensional spectral information. Thanks to their rich spectral and spatial information, hyperspectral images have a wide range of applications in environmental monitoring, agriculture, geological exploration, urban planning, and military affairs. Hyperspectral anomaly detection has attracted widespread attention in recent years and has become a key research area in hyperspectral information processing.

[0003] Nowadays, the application of deep learning has achieved remarkable results in this field. The low-rank characteristic of hyperspectral data refers to the strong correlation between the spectral bands in the hyperspectral data. The rank of the image matrix is ​​very small, indicating that most of the information in the hyperspectral data can be represented by a small number of basic features. In hyperspectral anomaly detection, the background of the hyperspectral image has a low-rank characteristic. Except for the anomalies, the matrix of the entire hyperspectral image is a low-rank matrix. However, most of the anomaly detection algorithms based on deep learning do not make full use of the low-rank characteristics of the background, resulting in the failure to effectively extract the common information and background features in the hyperspectral data, making it difficult for the anomaly information to be clearly highlighted in the data and reducing the difference between the anomaly and the background. On the other hand, although the existing algorithms have achieved good detection results, the separation of background and anomalies is still challenging.

[0004] Summary of the Invention

[0005] Aiming at the shortcomings of existing methods, the present invention addresses the problem that existing hyperspectral anomaly detection does not fully utilize the low-rank characteristics of the background and that the separation of background and anomalies by existing algorithms is still challenging.

[0006] The technical solution adopted by the present invention is: a method for detecting anomalies in hyperspectral remote sensing images based on deep learning includes the following steps:

[0007] Step 1: normalize and preprocess the hyperspectral remote sensing image to obtain the original hyperspectral image H;

[0008] Step 2: Build a low-rank background feature extraction network and input the original hyperspectral image into the low-rank background feature extraction network to generate low-rank background features;

[0009] Furthermore, the low-rank background feature extraction network includes: three 1x1 convolutions and three branches, the first and third branches include LRB modules; the features of the first 1x1 convolution are input into the three branches respectively, the first branch is multiplied with the features of the second branch through the LRB1 module, and multiplied with the features of the third branch through the LRB2 module, and then the low-rank background features are output after the second and third 1x1 convolutions.

[0010] Furthermore, the structure of the LRB module includes: two 3x3 convolutions, 1x1 convolution, BN layer and scSE attention mechanism, the 3x3 convolution is followed by LeakyReLU activation function, the 1x1 convolution is followed by BN layer, and connected in a dense skip manner.

[0011] Step 3: Construct a backbone feature extraction network, input low-rank background features into the backbone feature extraction network, and obtain backbone features;

[0012] Furthermore, the backbone feature extraction network includes a first and a second backbone feature extraction sub-network composed of a FEB module and a DRB module; the first backbone feature extraction sub-network includes the FEB1 module and the FEB2 module to form a main branch, and adds the output of the branch formed by the DRB1 module; the second backbone feature extraction sub-network includes the FEB3 module and the FEB4 module to form a main branch, and adds the output of the branch formed by the DRB2 module to output the backbone features.

[0013] Furthermore, the FEB module includes: an inversion module and a 1x1 convolution, wherein the inversion module includes 1x1 dimensionality increase convolution, 3x3 convolution, and 1x1 dimensionality reduction convolution.

[0014] Furthermore, the DRB module includes two 3x3 convolutions and scSE attention mechanism; among them, the 3x3 convolution is followed by a BN layer and a LeakyReLU activation function.

[0015] Step 4: Construct a feature convolutional network, input the backbone features into the feature convolutional network, and obtain convolution features;

[0016] Furthermore, the feature convolution network includes: 3x3 convolution, 1x1 convolution and the first upsampling by 2 times; and adding it to the output of the second upsampling by 2 times of the branch to output the convolution feature; 3x3 convolution and 1x1 convolution are followed by BN layer and LeakyReLU activation function.

[0017] Step 5: Build an upsampling network, input the convolutional features into the upsampling network, and obtain upsampling features;

[0018] Furthermore, the upsampling network includes three upsampling sub-networks consisting of a URB module and a 2-fold upsampling module; the common output of the two FEB modules of the second backbone feature extraction sub-network is upsampled by 2 times, added to the output of the first upsampling sub-network, and used as the input of the second upsampling sub-network;

[0019] The common output of the first and second backbone feature extraction subnetworks is upsampled by a factor of 2, added to the output of the second upsampling subnetwork, and used as the input of the third upsampling subnetwork.

[0020] The common output of the two FEB modules of the second backbone feature extraction sub-network is upsampled by a factor of 2, added to the output of the third upsampling sub-network, and used as the input of the URB4 module, which outputs the upsampled features.

[0021] Furthermore, the URB module includes: three 1x1 convolutions, one 3x3 convolution and scSE attention mechanism, where two 1x1 convolutions and 3x3 convolutions are followed by BN layer and LeakyReLU activation function.

[0022] Step 6: Construct a feature convolution sub-network, input the upsampled features into the feature convolution sub-network, and obtain the reconstructed background feature map;

[0023] Furthermore, the feature convolution subnetwork includes: 1x1 convolution and 1x1 dimensionality-increasing convolution; 1x1 convolution is followed by a BN layer and a LeakyReLU activation function; 1x1 dimensionality-increasing convolution is followed by a BN layer.

[0024] Step 7: Subtract the original hyperspectral image from the reconstructed background feature map to obtain the reconstruction error map, and use the adaptive weighted loss function to suppress the abnormal reconstruction weights to effectively separate the background and anomalies.

[0025] Beneficial effects of the present invention:

[0026] 1. The low-rank background feature extraction network fully utilizes the low-rank characteristics of the background, effectively extracts common information and background features in hyperspectral data, suppresses the influence of noise in the data, obtains pure background features, and improves the accuracy of hyperspectral anomaly detection;

[0027] 2. The backbone feature extraction module and downsampling residual block effectively reduce the loss of spectral-spatial information, while the skip connection provides additional, detailed spectral-spatial information, which can capture local details in low-level features, strengthen the impact on the reconstructed background results, and effectively achieve the reconstruction of hyperspectral background images;

[0028] 3. No complicated post-processing operations are required. The background can be directly reconstructed through the end-to-end network, and the anomaly and background can be effectively separated automatically to obtain the anomaly detection results of the hyperspectral remote sensing image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a flow chart of a method for detecting anomalies in hyperspectral remote sensing images based on deep learning according to the present invention;

[0030] FIG2 is a structural diagram of a low-rank background feature extraction module (LRB) of the present invention;

[0031] FIG3 is a structural diagram of the scSE attention mechanism of the present invention;

[0032] FIG4 is a structural diagram of a feature extraction module (FEB) of the present invention;

[0033] FIG5 is a structural diagram of a down-sampling residual block (DRB) according to the present invention;

[0034] FIG6 is a structural diagram of an upsampled residual block (URB) according to the present invention;

[0035] FIG7 is a pseudo-color image, a true value image, and a comparison result image of data 1;

[0036] Figure 8 is the pseudo-color image, true value image and comparison result image of Data II;

[0037] Figure 9 shows the pseudo-color image, true value image and comparison result of data III. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.

[0039] As shown in Figure 1, the hyperspectral remote sensing image anomaly detection method based on deep learning includes the following steps:

[0040] Step 1: normalize and preprocess the hyperspectral remote sensing image to obtain the original hyperspectral image H;

[0041] Step 2: Build a low-rank background feature extraction network and input H into the low-rank background feature extraction network to generate low-rank background features F;

[0042] As shown in Figure 1, a hyperspectral remote sensing image anomaly detection network model is built; it includes a low-rank background feature extraction network as the extraction of low-rank background features, the scSE attention mechanism integrates channel and spatial feature information, strengthens important features, the LBR module and DRB module effectively extract spectral-spatial information, and the skip connection and URB module effectively realize the reconstruction of the background image.

[0043] The low-rank background feature extraction network consists of three 1x1 convolutions and three branches. The first and third branches include the LRB module. The LRB module consists of two 3x3 convolutions, a 1x1 convolution, a batch normalization layer, and the scSE attention mechanism. The 3x3 convolution is followed by a LeakyReLU activation function, and the LRB module is connected in a dense skip connection. This fully extracts both local and global features, retaining the features of the previous layer of the hyperspectral image while extracting local features. Skip residual connections enable a better fusion of the two. The scSE attention mechanism further focuses the network on background features. The LRB module better combines local and global feature information, more effectively extracting low-rank background features. Rather than simply extracting local features, it also takes global features into account.

[0044] As shown in Figures 1 and 2, the number of channels is first reduced to 128 by 1x1 convolution; U is obtained by extracting the low-rank background feature through three branches of the network. wh*k 、H' wh*128 、U' k*wh , H' wh*128 with U' k*wh Multiply to get V k*128 , and then the obtained V k*128 with U wh*k Multiply to get F 128*h*w ; The low-rank background feature extraction network outputs low-rank background features, the formula is as follows: F b*h*w =U wh*k ·V k*b (1)

[0045] Among them, b is the number of bands, h is the height of the hyperspectral image, w is the width of the hyperspectral image, k is the low rank, F b*h*w is a low-rank background feature.

[0046] After 1x1 convolution and 1x1 dimensionality increase convolution, the low-rank background feature F is finally generated; among them, the first and third branches pass through the scSE attention mechanism, which includes two parallel spatial and channel branches. The spatial branch processes data through 1x1 convolution and sigmoid function and multiplies it with the original data; the channel branch passes through an adaptive pooling layer followed by two 1x1 convolutions, and then obtains the weight through the sigmoid function, multiplies it with the original data, and finally fuses the outputs of the two branches together to adaptively enhance the important features in the network while capturing channel and spatial pixel information.

[0047] Step 3: Construct a backbone feature extraction network, input the low-rank background features into the backbone feature extraction network, and obtain the backbone feature F5;

[0048] Input the low-rank background feature F into the convolution to generate feature F1, and input feature F1 into the backbone feature extraction network to generate multi-level features F2, F3, F4, and F5;

[0049] The FEB module structure is shown in Figure 4, which includes an inversion module and a 1x1 convolution. The inversion module includes 1x1 dimensionality increase convolution, 3x3 convolution, and 1x1 dimensionality reduction convolution. The input features are first dimensionality increased and then dimensionality reduced. In the dimensionality increase stage, more spectral-spatial semantic information is captured, reducing the loss of spectral-spatial information. The hyperspectral image undergoes the FEB1 module and the FEB2 module to obtain multi-level features F2128*h / 2*w / 2 and F3128*h / 4*w / 4.

[0050] The DRB module structure is shown in Figure 5, which includes two 3x3 convolutions and scSE attention mechanism, where the 3x3 convolution is followed by a BN layer and a LeakyReLU activation function.

[0051] The feature F1 128*h*w After the DRB1 module and DRB2 module output size is 128*h / 4*w / 4, and after adding it with F3, the output feature F 31 128*h / 4*w / 4, F 31 128*h / 4*w / 4 input FEB3 module, FEB4 module, generate multi-level features F4128*h / 8*w / 8, F5128*h / 16*w / 16, feature F 31 The output size of 128*h / 4*w / 4 after the DRB2 module is 128*h / 16*w / 16, which is added to F5128*h / 16*w / 16 to output the backbone feature F 51 128*h / 16*w / 16.

[0052] Step 4: Construct a feature convolutional network, input the backbone features into the feature convolutional network, and obtain the convolution feature F 61 ;

[0053] F 51 Input the feature convolution network to obtain feature F6; the feature convolution network includes 3x3 convolution and 1x1 convolution, each of which is followed by a BN layer and a LeakyReLU activation function; F 51 128xh / 16xw / 16 is passed through 3x3 convolution and 1x1 convolution, where each convolution is followed by BN layer and LeakyReLU activation function to obtain the potential feature F6, F6 is upsampled by 2 times and F 51 The features obtained by upsampling by 2 times are added and compared with F4 to obtain the convolution feature F 61 The output size is 128*h / 8*w / 8.

[0054] Step 5: Build an upsampling network, input the convolutional features into the upsampling network, and obtain the upsampling features F 10 ;

[0055] F2、F 31 , F4, F 51 Jump connection with the upsampling network to obtain the upsampling feature F 10 ;

[0056] As shown in Figure 6, the URB module consists of three 1x1 convolutions, one 3x3 convolution and the scSE attention mechanism, where two 1x1 convolutions and 3x3 convolutions are followed by a BN layer and a LeakyReLU activation function.

[0057] The feature F 61 After the URB1 module, F7128xh / 8xw / 8 is obtained; the features obtained by upsampling F7 by 2 times and F4 by 2 times are added and added together with F 31 Compare the size to get feature F 71 , the output size is 128*h / 4*w / 4, and the feature F 71 After the URB2 module, we get F8128*h / 4*w / 4; we upsample F8 by 2 times and 31 Add the features obtained by upsampling by 2 times and compare them with F2 to get feature F 81 , the output size is 128*h / 2*w / 2, and after the URB3 module, F9128*h / 2*w / 2 is obtained; the features obtained by upsampling F9 by 2 times and upsampling F2 by 2 times are compared with the size of F1 to obtain the feature F 91 , the output size is 128*h*w, and the original image size upsampling feature F is obtained through the URB4 module 10 128*h*w The 128-dimensionality is kept unchanged in order to preserve the spectral features. The downsampled features are upsampled twice and jump-connected with the upsampled network. This interactive mode of information features can capture the potential features in the low-level features, so that they can be better integrated with the extracted deep semantic information, retaining important spectral-spatial information and reconstructing the background more accurately.

[0058] Step 6: Construct a feature convolution sub-network, input the up-sampled features into the feature convolution sub-network, and obtain the reconstructed background feature map H';

[0059] By F 10 Get the reconstructed background image H', and transform F 10 128xhxwAfter a 1x1 convolution and a 1x1 upscaling convolution, followed by a batch normalization layer and a LeakyReLU activation function, the 1x1 upscaling convolution upscales the 128-dimensional image to the number of bands of the original hyperspectral image, obtaining the original resolution and reconstructing the background image H'.

[0060] Step 7: Subtract the original hyperspectral image from the reconstructed background feature map to obtain a reconstruction error map, and use the adaptive weighted loss function to suppress the abnormal reconstruction weights, so that the background and anomalies are effectively separated, and the hyperspectral remote sensing image anomaly detection results are obtained;

[0061] The original hyperspectral image H is subtracted from the reconstructed background image H' to obtain the reconstruction error map R, which is the anomaly detection result of the hyperspectral remote sensing image. The formula of the reconstruction error map is: R = ||H-H'||2.

[0062] During the iterative training process, the adaptive weighted loss function suppresses the reconstruction weight of the anomaly, so that the background and anomaly are effectively separated. Its calculation formula is as follows:

[0063] Among them, L represents the adaptive weighted loss function, H i,j ∈R b*1 is the pixel vector of the original hyperspectral image at position (i, j), H' i,j ∈R b*1 is the pixel vector of the reconstructed background image at position (i, j), E is the reconstruction error of all pixels, d i,j is the reconstruction error at position (i, j), h, w, and b are the height, width, and number of bands of the hyperspectral image, respectively.

[0064] The following is a comparison between three sets of hyperspectral data and existing anomaly detection algorithms (such as RX, PCA-TLRSR, KIFD, GTVLRR, 2S-GLRT, CRD, and AUTO-AD) and the Ours method of the present invention; the three sets of hyperspectral data are all public data sets, and Data I to Data III are captured by the airborne visible / infrared imaging spectrometer sensor (AVIRIS); among them, Data I is a beach scene with an image space size of 100x100 and a total of 188 spectral channels. Sea surface objects in the data are considered anomalies; Data II and Data III are urban scenes with an image space size of 100x100 and 204 and 207 spectral channels respectively. Houses of different shapes are considered anomalies; the three sets of hyperspectral data used are listed in detail in Table 1 and the experimental configuration is listed in detail in Table 2.

[0065] Table 1 Detailed information of the hyperspectral image data used

[0066] Table 2 Experimental configuration used

[0067] In this specific embodiment, the area under the curve (AUC) of the 2D receiver operating characteristic (ROC) curve is used as an evaluation metric to measure the effectiveness of anomaly detection through quantitative evaluation and qualitative observation. The better the anomaly detection method, the higher the AUC (PD, PF), that is, the closer the AUC value is to 1. The AUC results for different data types are shown in Table 3.

[0068] Table 3 Test results

[0069] After analysis of specific embodiments, the method of the present invention can obtain AUC values ​​of 0.9991, 0.9944, and 0.9993 on three data sets, respectively. The pseudo-color images, true value images, and detection results of Data Set I, Data Set II, and Data Set III are shown in Figures 7, 8, and 9; the results of the existing anomaly detection algorithms, such as RX, PCA-TLRSR, KIFD, GTVLRR, 2S-GLRT, CRD, AUTO-AD, and the method of the present invention Ours on data I are shown in Figures 7(b) and 7(c), and Figure 7(d) is a comparison of the ROC curves of the existing anomaly detection algorithms and the method of the present invention Ours on data I. It can be seen from the figure and the curve that the method of the present invention has a significant anomaly detection effect on sea surface objects in data I. Although the AUC of the RX algorithm is 0.9998, the background and anomalies are not well separated; the existing anomaly detection algorithms are relatively poor in detection accuracy. 8(b) and 8(c) show the results of the detection of data II by the normal detection algorithm and the method of the present invention Ours, and FIG8(d) shows a comparison of the ROC curves of the detection of data II by the existing anomaly detection algorithm and the method of the present invention Ours. As can be seen from the figure and the curve, the method of the present invention has a significant anomaly detection effect on the buildings in data II; the results of the detection of data III by the existing anomaly detection algorithm and the method of the present invention Ours are shown in FIG9(b) and 9(c), and FIG9(d) shows a comparison of the ROC curves of the detection of data III by the existing anomaly detection algorithm and the method of the present invention Ours. As can be seen from the figure and the curve, the method of the present invention has a significant anomaly detection effect on the buildings in data III.

[0070] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A hyperspectral remote sensing image anomaly detection method based on deep learning, characterized in that: The following steps are involved: Step 1: normalize and preprocess the hyperspectral remote sensing image to obtain the original hyperspectral image; Step 2: Build a low-rank background feature extraction network, and input the original hyperspectral image into the low-rank background feature extraction network to generate low-rank background features; Step 3: Construct a backbone feature extraction network, input low-rank background features into the backbone feature extraction network, and obtain backbone features; Step 4: Construct a feature convolution network, input the backbone features into the feature convolution network, and obtain convolution features; Step 5: Build an upsampling network, input the convolutional features into the upsampling network, and obtain upsampling features; Step 6: Construct a feature convolution sub-network, input the up-sampled features into the feature convolution sub-network, and obtain a reconstructed background feature map; Step 7: Subtract the original hyperspectral image from the reconstructed background feature map to obtain the reconstruction error map, and use the adaptive weighted loss function to suppress abnormal reconstruction weights.

2. The method for detecting anomalies in hyperspectral remote sensing images based on deep learning according to claim 1, characterized in that: The low-rank background feature extraction network includes: three 1x1 convolutions and three branches, the first and third branches include LRB modules; the features of the first 1x1 convolution are input into the three branches respectively, the first branch is multiplied with the features of the second branch through the LRB1 module, and multiplied with the features of the third branch through the LRB2 module, and then the low-rank background features are output after the second and third 1x1 convolutions.

3. The method for detecting anomalies in hyperspectral remote sensing images based on deep learning according to claim 2, characterized in that: The structure of the LRB module includes: two 3x3 convolutions, 1x1 convolution, BN layer and scSE attention mechanism. The 3x3 convolution is followed by LeakyReLU activation function, and the 1x1 convolution is followed by BN layer, which are connected in a dense skip manner.

4. The method for detecting anomalies in hyperspectral remote sensing images based on deep learning according to claim 1, characterized in that: The backbone feature extraction network includes the first and second backbone feature extraction sub-networks composed of FEB modules and DRB modules; the first backbone feature extraction sub-network includes FEB1 modules and FEB2 modules to form a main branch, and the output of the branch road formed by the DRB1 module is added; the second backbone feature extraction sub-network includes FEB3 modules and FEB4 modules to form a main branch, and the output of the branch road formed by the DRB2 module is added to output the backbone features.

5. The method for detecting anomalies in hyperspectral remote sensing images based on deep learning according to claim 4, characterized in that: The FEB module includes: an inversion module and a 1x1 convolution. The inversion module includes 1x1 dimensionality increase convolution, 3x3 convolution, and 1x1 dimensionality reduction convolution.

6. The method for detecting anomalies in hyperspectral remote sensing images based on deep learning according to claim 4, characterized in that: The DRB module consists of two 3x3 convolutions and the scSE attention mechanism; the 3x3 convolution is followed by a BN layer and a LeakyReLU activation function.

7. The method for detecting anomalies in hyperspectral remote sensing images based on deep learning according to claim 1, characterized in that: The feature convolution network includes: 3x3 convolution, 1x1 convolution and the first upsampling by 2 times; and adding it to the output of the second upsampling by 2 times of the branch to output the convolution feature; 3x3 convolution and 1x1 convolution are followed by BN layer and LeakyReLU activation function.

8. The method for detecting anomalies in hyperspectral remote sensing images based on deep learning according to claim 1, characterized in that: The upsampling network includes the URB module and three upsampling sub-networks consisting of 2 times upsampling; The common output of the two FEB modules of the second backbone feature extraction subnetwork is upsampled by 2 times, added to the output of the first upsampling subnetwork, and used as the input of the second upsampling subnetwork; The common output of the first backbone feature extraction subnetwork and the second backbone feature extraction subnetwork is upsampled by 2 times, added to the output of the second upsampling subnetwork, and used as the input of the third upsampling subnetwork. enter; The common output of the two FEB modules of the first backbone feature extraction subnetwork is upsampled by 2 times, added to the output of the third upsampling subnetwork, and used as the input of the URB4 module to output the upsampled features.

9. The method for detecting anomalies in hyperspectral remote sensing images based on deep learning according to claim 8, characterized in that: The URB module includes: three 1x1 convolutions, one 3x3 convolution and scSE attention mechanism, two 1x1 convolutions and 3x3 convolutions followed by BN layer and LeakyReLU activation function.

10. The method for anomaly detection of hyperspectral remote sensing images based on deep learning according to claim 1, characterized in that: The feature convolution subnetwork includes: 1x1 convolution and 1x1 dimension-increasing convolution; 1x1 convolution followed by BN layer and LeakyReLU activation function; 1x1 dimension-increasing convolution followed by BN layer.

Images