Hyperspectral remote sensing image anomaly detection method based on deep learning

Abstract

The application relates to the technical field of image processing, in particular to a hyperspectral remote sensing image anomaly detection method based on deep learning, which comprises the following steps: carrying out normalization preprocessing on a hyperspectral remote sensing image to obtain an original hyperspectral image; inputting the original hyperspectral image into a low-rank background feature extraction network to generate a low-rank background feature; inputting the low-rank background feature into a backbone feature extraction network to obtain a backbone feature; inputting the backbone feature into a feature convolution network to obtain a convolution feature; inputting the convolution feature into an up-sampling network to obtain an up-sampling feature; inputting the up-sampling feature into a feature convolution sub-network to obtain a reconstructed background feature map; subtracting the reconstructed background feature map from the original hyperspectral image to obtain a reconstruction error map, and utilizing an adaptive weighted loss function to suppress the reconstruction weight of the anomaly. The application aims at the problem that the low-rank characteristics of the background are not fully utilized in the existing hyperspectral anomaly detection, and the separation of the background and the anomaly is still challenging in the existing algorithm.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for anomaly detection in hyperspectral remote sensing images based on deep learning. Background Technology

[0002] Hyperspectral images are three-dimensional images created using imaging and spectral techniques. They contain not only two-dimensional spatial information but also one-dimensional spectral information. Thanks to the rich spectral and spatial information in hyperspectral images, they have wide applications in environmental monitoring, agriculture, geological exploration, urban planning, and the military. Hyperspectral anomaly detection has attracted considerable attention in recent years and has become one of the important research areas in hyperspectral information processing.

[0003] Today, deep learning has achieved remarkable results in this field. The low-rank characteristic of hyperspectral data refers to the strong correlation between spectral bands and the small rank of the image matrix, indicating that most information in hyperspectral data can be represented by a small number of basic features. In hyperspectral anomaly detection, the background of a hyperspectral image exhibits low-rank characteristics; excluding the anomaly, the entire hyperspectral image matrix is ​​a low-rank matrix. However, most deep learning-based anomaly detection algorithms do not fully utilize the low-rank characteristics of the background, resulting in ineffective extraction of common information and background features from the hyperspectral data. This makes it difficult to clearly highlight anomalies and reduce the difference between anomalies and background. On the other hand, while existing algorithms achieve good detection results, separating the background from anomalies remains challenging. Summary of the Invention

[0004] To address the shortcomings of existing methods, this invention addresses the problem that existing hyperspectral anomaly detection methods do not fully utilize the low-rank characteristics of the background, and that existing algorithms still face challenges in separating the background from the anomaly.

[0005] The technical solution adopted in this invention is: a method for anomaly detection in hyperspectral remote sensing images based on deep learning, comprising the following steps:

[0006] Step 1: Normalize and preprocess the hyperspectral remote sensing image to obtain the original hyperspectral image H;

[0007] Step 2: Build a low-rank background feature extraction network. Input the original hyperspectral image into the low-rank background feature extraction network to generate low-rank background features.

[0008] Furthermore, the low-rank background feature extraction network includes three 1x1 convolutions and three branches. The first and third branches include LRB modules. The features of the first 1x1 convolution are input into the three branches respectively. The features of the first branch are multiplied by the features of the second branch through the LRB1 module and multiplied by the features of the third branch through the LRB2 module. After passing through the second and third 1x1 convolutions, the low-rank background features are output.

[0009] Furthermore, the structure of the LRB module includes: two 3x3 convolutions, a 1x1 convolution, a BN layer, and an scSE attention mechanism. The 3x3 convolutions are followed by a LeakyReLU activation function, and the 1x1 convolutions are followed by a BN layer, which are connected in a dense skip manner.

[0010] Step 3: Construct the backbone feature extraction network. Input the low-rank background features into the backbone feature extraction network to obtain the backbone features.

[0011] Furthermore, the backbone feature extraction network includes first and second backbone feature extraction sub-networks composed of FEB modules and DRB modules; the first backbone feature extraction sub-network includes FEB1 and FEB2 modules forming the main branch, and the outputs of the branch path formed by the DRB1 module are added together; the second backbone feature extraction sub-network includes FEB3 and FEB4 modules forming the main branch, and the outputs of the branch path formed by the DRB2 module are added together to output the backbone features.

[0012] Furthermore, the FEB module includes an inverted module and a 1x1 convolution, wherein the inverted module includes a 1x1 up-dimensional convolution, a 3x3 convolution, and a 1x1 down-dimensional convolution.

[0013] Furthermore, the DRB module includes two 3x3 convolutions and an scSE attention mechanism; the 3x3 convolutions are followed by a BN layer and a LeakyReLU activation function.

[0014] Step 4: Construct a feature convolutional network by inputting the backbone features into the feature convolutional network to obtain convolutional features;

[0015] Furthermore, the feature convolutional network includes: 3x3 convolution, 1x1 convolution and a first upsampling of 2x; and is added to the output of the second upsampling of 2x in the branch to output convolutional features; the 3x3 convolution and 1x1 convolution are followed by a BN layer and a LeakyReLU activation function.

[0016] Step 5: Construct an upsampling network by inputting the convolutional features into the upsampling network to obtain upsampling features;

[0017] Furthermore, the upsampling network includes a URB module and three upsampling sub-networks consisting of upsampling by 2; the common output of the two FEB modules of the second backbone feature extraction sub-network is upsampled by 2, added to the output of the first upsampling sub-network, and used as the input of the second upsampling sub-network;

[0018] The common output of the first and second backbone feature extraction subnetworks is upsampled by a factor of 2, added to the output of the second upsampled subnetwork, and used as the input of the third upsampled subnetwork.

[0019] The common output of the two FEB modules of the second backbone feature extraction subnetwork is upsampled by a factor of 2, added to the output of the third upsampling subnetwork, and used as the input of the URB4 module to output the upsampled features.

[0020] Furthermore, the URB module includes: three 1x1 convolutions, one 3x3 convolution, and an scSE attention mechanism, wherein the two 1x1 convolutions and the 3x3 convolution are followed by a BN layer and a LeakyReLU activation function.

[0021] Step 6: Construct a feature convolutional sub-network. Input the upsampled features into the feature convolutional sub-network to obtain the reconstructed background feature map;

[0022] Furthermore, the feature convolutional subnetwork includes: 1x1 convolution and 1x1 up-dimensional convolution; 1x1 convolution followed by BN layer and LeakyReLU activation function; 1x1 up-dimensional convolution followed by BN layer.

[0023] Step 7: Subtract the original hyperspectral image from the reconstructed background feature image to obtain the reconstruction error image, and use an adaptive weighted loss function to suppress the reconstruction weight of anomalies, so that the background and anomalies are effectively separated.

[0024] The beneficial effects of this invention are:

[0025] 1. The low-rank background feature extraction network makes full use of the low-rank characteristics of the background, effectively extracts common information and background features in hyperspectral data, suppresses the influence of noise in the data, obtains pure background features, and improves the accuracy of hyperspectral anomaly detection.

[0026] 2. The backbone feature extraction module and downsampling residual block effectively reduce the loss of spectral-spatial information, while skip connections provide additional, detailed spectral-spatial information, which can capture local details in low-level features and enhance the impact on the reconstructed background results, effectively realizing the reconstruction of hyperspectral background images;

[0027] 3. No complex post-processing is required. Background reconstruction is achieved directly through an end-to-end network, and anomalies are automatically and effectively separated from the background to obtain anomaly detection results in hyperspectral remote sensing images. Attached Figure Description

[0028] Figure 1 This is a flowchart of the hyperspectral remote sensing image anomaly detection method based on deep learning according to the present invention;

[0029] Figure 2 This is a structural diagram of the low-rank background feature extraction module (LRB) of the present invention;

[0030] Figure 3 This is a structural diagram of the scSE attention mechanism of the present invention;

[0031] Figure 4 This is a structural diagram of the main feature extraction module (FEB) of the present invention;

[0032] Figure 5 This is a structural diagram of the downsampling residual module (DRB) of the present invention;

[0033] Figure 6 This is a structural diagram of the upsampling residual block (URB) of the present invention;

[0034] Figure 7 This includes a pseudo-color image, a truth image, and a comparison result image for data I.

[0035] Figure 8 The pseudo-color plot, truth plot, and comparison results plot for Data II;

[0036] Figure 9 This is a pseudo-color plot, a truth plot, and a comparison result plot for Data III. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0038] like Figure 1 As shown, the anomaly detection method for hyperspectral remote sensing images based on deep learning includes the following steps:

[0039] Step 1: Normalize the hyperspectral remote sensing image to obtain the original hyperspectral image H;

[0040] Step 2: Construct a low-rank background feature extraction network. Input H into the low-rank background feature extraction network to generate low-rank background features F.

[0041] like Figure 1As shown, a hyperspectral remote sensing image anomaly detection network model is constructed. This model includes a low-rank background feature extraction network for extracting low-rank background features, an scSE attention mechanism that integrates channel and spatial feature information to enhance important features, an LBR module and a DRB module for effectively extracting spectral-spatial information, and a skip connection and URB module for effectively reconstructing the background image.

[0042] The low-rank background feature extraction network consists of three 1x1 convolutions and three branches, with the first and third branches including LRB modules. The LRB modules comprise two 3x3 convolutions, a 1x1 convolution, a batch normalization (BN) layer, and a scSE attention mechanism. The 3x3 convolutions are followed by a Leaky ReLU activation function, and the LRB modules are connected in a dense, skip-connection manner. This effectively extracts both local and global features, preserving features from the previous layer of the hyperspectral image while also extracting local features. The skip-connection method allows for better fusion of these features. The scSE attention mechanism further emphasizes background features. The LRB modules better combine local and global feature information, more effectively extracting low-rank background features, rather than solely focusing on local features, but also considering global features.

[0043] like Figure 1 and 2 As shown, the number of channels is first reduced to 128 using 1x1 convolution; then, U is obtained through the three branches of the low-rank background feature extraction network. wh*k H' wh*128 、U' k*wh , will H' wh*128 with U' k*wh Multiply to get V k*128 Then the obtained V k*128 with U wh*k Multiply to get F 128*h*w The low-rank background feature extraction network outputs low-rank background features, as shown in the following formula:

[0044] F b*h*w =U wh*k ·V k*b (1)

[0045] Where b is the number of bands, h is the height of the hyperspectral image, w is the width of the hyperspectral image, k is the low rank, and F b*h*w This is a low-rank background feature.

[0046] After 1x1 convolution and 1x1 up-dimensional convolution, the low-rank background feature F is finally generated. The first and third branches are processed through the scSE attention mechanism, which includes two parallel spatial and channel branches. The spatial branch processes the data through 1x1 convolution and the sigmoid function, and multiplies it with the original data. The channel branch passes through an adaptive pooling layer followed by two 1x1 convolutions, then obtains weights through the sigmoid function, multiplies them with the original data, and finally merges the outputs of the two branches together to adaptively enhance the important features in the network while capturing channel and spatial pixel information.

[0047] Step 3: Construct the backbone feature extraction network. Input the low-rank background features into the backbone feature extraction network to obtain the backbone feature F5.

[0048] The low-rank background feature F is input into the convolution to generate feature F1. Feature F1 is then input into the backbone feature extraction network to generate multi-level features F2, F3, F4, and F5.

[0049] The FEB module structure is as follows: Figure 4 The diagram shows an inverted module and a 1x1 convolution. The inverted module includes a 1x1 upscaling convolution, a 3x3 convolution, and a 1x1 downscaling convolution. It first upscales the input features and then downscales them, capturing more spectral-spatial semantic information and reducing the loss of spectral-spatial information during the upscaling stage. The hyperspectral image undergoes the FEB1 and FEB2 modules to obtain multi-level features F2. 128*h / 2*w / 2 F3 128*h / 4*w / 4 .

[0050] The DRB module structure is as follows: Figure 5 As shown, it includes two 3x3 convolutions and an scSE attention mechanism, where the 3x3 convolutions are followed by a BN layer and a LeakyReLU activation function.

[0051] Feature F1 128*h*w The output size after passing through the DRB1 and DRB2 modules is 128*h / 4*w / 4. After adding it to F3, the output feature F is obtained. 31 128*h / 4*w / 4 , will F 31 128*h / 4*w / 4 Input FEB3 and FEB4 modules to generate multi-level feature F4. 128 *h / 8*w / 8 F5 128*h / 16*w / 16 Feature F 31 128*h / 4*w / 4 The output size after passing through the DRB2 module is 128*h / 16*w / 16, and it is compatible with F5. 128 *h / 16*w / 16 The main characteristic F is output after the addition operation. 51 128*h / 16*w / 16 .

[0052] Step 4: Construct a feature convolutional network. Input the backbone features into the feature convolutional network to obtain the convolutional features F. 61 ;

[0053] F 51 The input feature convolutional network yields feature F6; the feature convolutional network includes 3x3 convolutions and 1x1 convolutions, where each convolution is followed by a BN layer and a LeakyReLU activation function; F... 51 128xh / 16xw / 16 The latent feature F6 is obtained by performing 3x3 convolutions and 1x1 convolutions, with each convolution followed by a BN layer and a LeakyReLU activation function. F6 is then upsampled by a factor of 2 and... 51 The features obtained after upsampling by a factor of 2 are summed and compared with F4 to obtain the convolutional feature F. 61 The output size is 128*h / 8*w / 8.

[0054] Step 5: Construct an upsampling network by inputting the convolutional features into the upsampling network to obtain the upsampling features F. 10 ;

[0055] F2, F 31 F4, F 51 The upsampled features F are obtained by skip connections to the upsampled network. 10 ;

[0056] like Figure 6 As shown, the URB module includes three 1x1 convolutions, one 3x3 convolution, and an scSE attention mechanism. The two 1x1 convolutions and the 3x3 convolution are followed by a BN layer and a LeakyReLU activation function.

[0057] Feature F 61 F7 was obtained through the URB1 module. 128xh / 8xw / 8 Add the features obtained by upsampling F7 by a factor of 2 and upsampling F4 by a factor of 2, and then add them to F. 31 The feature F is obtained by comparing the sizes. 71 The output size is 128*h / 4*w / 4, and the feature F 71 F8 is obtained through the URB2 module. 128*h / 4*w / 4 ; Upsample F8 by 2 times and F 31 The features obtained by upsampling by 2 are summed and compared with F2 to obtain the feature F. 81 The output size is 128*h / 2*w / 2, which is then processed by the URB3 module to obtain F9. 128 *h / 2*w / 2 The feature pairs obtained by upsampling F9 by a factor of 2 and upsampling F2 by a factor of 2 are compared with F1 to obtain the feature F.91 The output size is 128*h*w, and the URB4 module is used to obtain the upsampled feature F of the original image size. 10 128*h*w Maintaining 128 dimensions is to preserve spectral features; downsampled features are upsampled twice and skipped connections are made with the upsampled network. This interaction of information features can capture potential features in low-level features, allowing for better integration with the extracted deep semantic information, preserving important spectral-spatial information, and more accurately reconstructing the background.

[0058] Step 6: Construct a feature convolutional sub-network. Input the upsampled features into the feature convolutional sub-network to obtain the reconstructed background feature map H'.

[0059] Through F 10 Obtain the reconstructed background image H', and then... 10 128xhxw After a 1x1 convolution and a 1x1 upscaling convolution, where the 1x1 convolution is followed by a BN layer and a LeakyReLU activation function, the 1x1 upscaling convolution increases the dimensionality from 128 to the number of bands in the original hyperspectral image, obtaining the original resolution size and reconstructing the background image H'.

[0060] Step 7: Subtract the original hyperspectral image from the reconstructed background feature image to obtain the reconstruction error image, and use an adaptive weighted loss function to suppress the reconstruction weight of anomalies, so that the background and anomalies are effectively separated, and the anomaly detection results of the hyperspectral remote sensing image are obtained.

[0061] Subtracting the original hyperspectral image H from the reconstructed background image H' yields the reconstruction error map R, which is the anomaly detection result of the hyperspectral remote sensing image. The formula for the reconstruction error map is: R = ||H - H'||2.

[0062] During iterative training, the adaptive weighted loss function suppresses the reconstruction weights of anomalies, effectively separating the background from the anomalies. Its calculation formula is as follows:

[0063] E = [d 1,1 ,...,d 1,w ;...;d h,1 ,...,d h,w ]

[0064]

[0065] Where L represents the adaptive weighted loss function, H i,j ∈R b*1 H' is the pixel vector at position (i,j) of the original hyperspectral image. i,j ∈R b*1 To reconstruct the pixel vector at position (i,j) of the background image, E represents the reconstruction error of all pixels, and di,j Let be the reconstruction error at position (i,j), and h, w, and b be the height, width, and number of bands of the hyperspectral image, respectively.

[0066] The following section compares the proposed method Ours with three sets of hyperspectral data and existing anomaly detection algorithms (e.g., RX, PCA-TLRSR, KIFD, GTVLRR, 2S-GLRT, CRD, AUTO-AD). All three sets of hyperspectral data are publicly available datasets. Data I-III were captured by an airborne visible / infrared imaging spectrometer sensor (AVIRIS). Data I depicts a beach scene with an image size of 100x100 and 188 spectral channels; objects on the sea surface in this data are considered anomalies. Data II-III depict urban scenes with image sizes of 100x100 and 204 and 207 spectral channels respectively; houses of different shapes are considered anomalies. Details of the three sets of hyperspectral data used are listed in Table 1, and the experimental configuration is detailed in Table 2.

[0067] Detailed information on the hyperspectral image data used in Table 1

[0068] Dataset sensor Capture location Spatial resolution size I AVIRIS Bay Champagne 4.4m 100x100x188 II AVIRIS Texas Coast 17.2m 100x100x204 III AVIRIS Texas Coast 17.2m 100x100x207

[0069] Table 2 shows the experimental setup used.

[0070]

[0071] In a specific embodiment, the area under the curve (AUC) of the 2D receiver operating characteristic (ROC) is used as an evaluation metric to measure the effectiveness of anomaly detection through both quantitative evaluation and qualitative observation. The better the performance of the anomaly detection method, the higher the AUC (PD, PF), i.e., the closer the AUC value is to 1; the AUC results for different data are shown in Table 3.

[0072] Table 3 Detection Results

[0073]

[0074] Through analysis of specific embodiments, the method of the present invention can obtain AUC values ​​of 0.9991, 0.9944, and 0.9993 on three datasets, respectively. The pseudo-color images, ground truth images, and detection results for datasets I, II, and III are shown below. Figure 7 , Figure 8 , Figure 9 As shown; the results of existing anomaly detection algorithms, such as RX, PCA-TLRSR, KIFD, GTVLRR, 2S-GLRT, CRD, AUTO-AD, and the method of this invention, on detecting data I are illustrated in the figure. Figure 7 (b) Figure 7 As shown in (c), and as Figure 7 (d) shows a comparison of the ROC curves of existing anomaly detection algorithms and the method of our invention for detecting anomalies in Data I. As can be seen from the figure and curves, the method of our invention has a significant anomaly detection effect on sea surface objects in Data I. Although the AUC of the RX algorithm is 0.9998, the background and anomalies are not well separated. The results of existing anomaly detection algorithms and the method of our invention for detecting anomalies in Data II are shown in the figure. Figure 8 (b) Figure 8 As shown in (c), and as Figure 8 (d) shows a comparison of the ROC curves of existing anomaly detection algorithms and the method of our invention for detecting anomalies in Data II. As can be seen from the figure and curves, the method of our invention has a significant anomaly detection effect on the buildings in Data II. The results of existing anomaly detection algorithms and the method of our invention for detecting anomalies in Data III are shown in the figure. Figure 9 (b) Figure 9 As shown in (c), and as Figure 9 (d) shows a comparison of the ROC curves of existing anomaly detection algorithms and the method of the present invention for detecting anomalies in Data III. As can be seen from the figure and curve, the method of the present invention has a significant anomaly detection effect on buildings in Data III.

[0075] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for anomaly detection in hyperspectral remote sensing images based on deep learning, characterized in that, Includes the following steps: Step 1: Normalize the hyperspectral remote sensing image to obtain the original hyperspectral image; Step 2: Build a low-rank background feature extraction network. Input the original hyperspectral image into the low-rank background feature extraction network to generate low-rank background features. The low-rank background feature extraction network consists of three 1x1 convolutions and three branches. The first and third branches include LRB modules. The features of the first 1x1 convolution are input into the three branches respectively. The features of the first branch are multiplied by the features of the second branch through the LRB1 module and multiplied by the features of the third branch through the LRB2 module. After passing through the second and third 1x1 convolutions, the low-rank background features are output. The structure of the LRB module includes: two 3x3 convolutions, a 1x1 convolution, a BN layer and an scSE attention mechanism. The 3x3 convolution is followed by a LeakyReLU activation function, and the 1x1 convolution is followed by a BN layer, which are connected in a dense skip manner. Step 3: Construct the backbone feature extraction network. Input the low-rank background features into the backbone feature extraction network to obtain the backbone features. The backbone feature extraction network includes first and second backbone feature extraction sub-networks composed of FEB modules and DRB modules; the first backbone feature extraction sub-network includes FEB1 and FEB2 modules forming the main branch, and the outputs of the sub-networks formed by the DRB1 module are added together; the second backbone feature extraction sub-network includes FEB3 and FEB4 modules forming the main branch, and the outputs of the sub-networks formed by the DRB2 module are added together to output the backbone features. The FEB module includes: an inverted module and a 1x1 convolution. The inverted module includes a 1x1 up-dimensional convolution, a 3x3 convolution, and a 1x1 down-dimensional convolution. The DRB module includes two 3x3 convolutions and an scSE attention mechanism; the 3x3 convolutions are followed by a BN layer and a LeakyReLU activation function. Step 4: Construct a feature convolutional network by inputting the backbone features into the feature convolutional network to obtain convolutional features; Step 5: Construct an upsampling network by inputting the convolutional features into the upsampling network to obtain upsampling features; Step 6: Construct a feature convolutional sub-network. Input the upsampled features into the feature convolutional sub-network to obtain the reconstructed background feature map; Step 7: Subtract the original hyperspectral image from the reconstructed background feature image to obtain the reconstruction error image, and use an adaptive weighted loss function to suppress abnormal reconstruction weights.

2. The method for anomaly detection in hyperspectral remote sensing images based on deep learning according to claim 1, characterized in that, The feature convolutional network consists of: 3x3 convolution, 1x1 convolution, and a first upsampling of 2x; and is added to the output of the second upsampling of 2x in the branch to output convolutional features; the 3x3 convolution and 1x1 convolution are followed by a BN layer and a LeakyReLU activation function.

3. The method for anomaly detection in hyperspectral remote sensing images based on deep learning according to claim 1, characterized in that, The upsampling network consists of a URB module and three upsampling sub-networks consisting of upsampling by 2 times; The common output of the two FEB modules of the second backbone feature extraction subnetwork is upsampled by a factor of 2, added to the output of the first upsampled subnetwork, and used as the input of the second upsampled subnetwork. The common output of the first and second backbone feature extraction subnetworks is upsampled by a factor of 2, added to the output of the second upsampled subnetwork, and used as the input of the third upsampled subnetwork. The common output of the two FEB modules of the first backbone feature extraction subnetwork is upsampled by a factor of 2, added to the output of the third upsampling subnetwork, and used as the input of the URB4 module to output the upsampled features; The URB module includes: three 1x1 convolutions, one 3x3 convolution, and an scSE attention mechanism, followed by a BN layer and a LeakyReLU activation function after the two 1x1 and 3x3 convolutions.

4. The method for anomaly detection in hyperspectral remote sensing images based on deep learning according to claim 1, characterized in that, The feature convolutional subnetwork includes: 1x1 convolution and 1x1 up-dimensional convolution; 1x1 convolution followed by BN layer and LeakyReLU activation function; 1x1 up-dimensional convolution followed by BN layer.

Images