6g satellite image landslide detection method based on multi-level transformer network

Abstract

The application provides a satellite landslide image detection method based on a multi-level Transformer network, and belongs to the field of image anomaly detection. In view of the fact that the existing landslide detection is only performed by using a convolutional neural network to extract and reconstruct image features, the application adopts a pre-training feature extraction network and an unsupervised multi-level Transformer auto-encoder to perform landslide detection. First, multi-scale features are extracted from the pre-training network, and then the image features are reconstructed by using a self-encoding Transformer network with a U shape. Through the fusion and reconstruction of the features by the multi-level Transformer encoder and decoder, coarse-grained low-level pixel features and fine-grained global semantic features can be obtained at the same time. The proposed method is applied to a satellite captured Bijie landslide data set, and experimental results show that, compared with an original CNN model, the application can improve detection accuracy and effectively distinguish landslide and non-landslide image data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unsupervised anomaly detection, and particularly relates to a satellite image anomaly detection method based on a multi-level Transformer network structure. BACKGROUND

[0002] Landslide is a common geological disaster that can cause unpredictable damage to the natural environment, public infrastructure and human life. In recent years, with the development of 6G satellite communication networks and remote sensing technology, people can apply high-resolution satellite images to disaster management, landslide detection and identification. There are various landslide features in landslide images, and how to comprehensively utilize these features for accurate and effective landslide detection is of great significance for reducing economic risks and protecting human life.

[0003] Traditional landslide detection methods usually use image texture and color statistical features to detect landslides. In addition, some machine learning methods, such as support vector machines, random forests and genetic algorithms, combine statistical methods for landslide detection. However, machine learning methods do not have good feature extraction capabilities and cannot automatically detect.

[0004] Deep learning models have strong representation learning capabilities and can automatically extract features and apply them to landslide detection. Specifically, Convolutional Neural Networks (CNN) have surpassed traditional statistical feature extraction and machine learning methods in various image-based tasks. The results of using Convolutional Neural Networks (CNN) for landslide detection show that deep learning methods based on CNN can achieve better results. However, CNN focuses on extracting local image features and has difficulty learning global context features of images. Therefore, the CNN-based method is improved by adding an attention mechanism to facilitate CNN to extract more unique landslide feature representations for further downstream detection tasks. In recent years, Vision Transformer (VIT) has proven that the Transformer framework used for natural language processing can be transferred to image recognition and detection tasks with excellent performance. Compared with CNN, ViT can learn image context features through patch-level images and global attention mechanisms.

[0005] To capture both local and global information from images, we focus on combining Transformers at multiple levels for better image detection and recognition. Therefore, we propose an unsupervised deep learning model for landslide detection. This model primarily consists of a feature extraction network and a multi-level Transformer autoencoder framework. First, we extract features from a pre-trained network; then, we reconstruct image features using a multi-level Transformer network. Through feature reconstruction using multi-level Transformers, we can extract not only fine-grained pixel-level features but also coarse-grained contextual semantic features. Finally, we fuse two types of features from satellite imagery for the landslide detection task. Summary of the Invention:

[0006] To address the problem of landslide detection in 6G satellite images and help humans identify landslide information more accurately and quickly, this invention discloses a satellite landslide image detection method based on a multi-level Transformer network.

[0007] (I) Technical Solution

[0008] The overall framework for landslide detection using 6G satellite images is an unsupervised multi-stage transformer encoder-decoder network, comprising a pre-trained CNN network and a multi-stage autoencoder with a Transformer structure. First, a pre-trained ResNet-18 network is introduced to extract latent features from satellite images; next, a multi-stage Transformer autoencoder network is used to reconstruct the image data features; finally, the calculated reconstruction error is used as an anomaly score to determine whether an image is a landslide.

[0009] It includes the following specific steps:

[0010] 1. Data Preprocessing

[0011] like Figure 1 As shown in the flowchart, the first step of the method is to input data and perform data preprocessing. Data preprocessing mainly involves partitioning, transforming, and augmenting the original data. In dataset partitioning, for unsupervised tasks, only non-slope images are used as training data, while in the testing phase, both slope and non-slope images are used. The entire dataset is divided into training and testing sets in a 7:3 ratio. Data transformation and augmentation standardize the way data is input into the network, and also help the network learn more discriminative feature information, enhancing the network's robustness and learning ability.

[0012] Step 1_1: Divide the dataset into training sets. and test set The data file.

[0013] and The definitions are as follows:

[0014] x i ∈X train

[0015] x j ∈X test ,y j ∈Y test

[0016] x i represents the i-th sample data in the training set, n train represents the number of training set data, X train represents the set of all training data; x j represents the i-th sample data in the test set, y j represents its corresponding label value, n test represents the number of test set data, X test represents the set of all test data, Y test represents the set of all test data labels. It can be seen that the training data is not labeled, because the present application is more in line with the reality of unlabeled data, so it uses unsupervised training without labels. In the test, it is judged whether it is a landslide data.

[0017] Step 1_2 data transformation and enhancement. Use the Compose method in the transformer of pytorch to transform and enhance the image data, mainly including scaling, padding, rotating, center cropping and other common transformation methods of images.

[0018] 2. Model feature extraction stage

[0019] The feature model is shown in Figure 2 ResNet-18 is used to extract features from image data. In order to more comprehensively extract the rough and fine force features of the image, we select the middle three convolutional neural networks from ResNet-18 to output respectively, and then combine them into a multi-layer structure joint representation through concatenate. The multi-scale fused feature vector is used as the basis vector for the following picture data feature reconstruction.

[0020] Step 2_1 input the training set data into ResNet-18 for feature extraction, and get the output of three different convolutional layers, output[i], i = 1, 2, 3.

[0021] Step 2_2 add the outputs of the three feature layers obtained two by two according to the channel dimension to get a combined joint feature representation z.

[0022] Step 2_3 does not need gradient update because it directly uses the pre-trained ResNet-18. Directly take z as the feature vector to be reconstructed next. For example, the i-th sample x in the training set i , get the final output vector z i ,

[0023] z i = F Resnet18 (x i )

[0024] Where F represents the channel alignment function performed on the training set data.

[0025] 3. Model formal training phase

[0026] The formal training model is shown in Fig. Figure 3 The formal training phase reconstructs the output features obtained in the feature extraction phase. The main model is a multi-level Transformer autoencoder network. Compared with traditional convolutional neural networks, Transformer has stronger image feature extraction and reconstruction ability due to its attention mechanism. In order to capture coarse and fine-grained image features, multi-level Transformer encoders and decoders are used. In addition, the invention combines the advantages of U-Net, and uses skip connections to transmit information between encoders and decoders of the same level. The encoding or decoding information of the previous layer is transmitted between the encoders and decoders of different layers in turn.

[0027] Step 3_1 inputs the pre-trained feature data of ResNet-18.

[0028] Step 3_2 encodes the feature image data through a multi-level Transformer encoder. The invention converts the feature map into patches, and adds zero padding tokens and positional encoding as inputs to the Transformer encoder. The HxW size of the landslide image feature map is divided into blocks with a block size of P H x P W , and these blocks have a total of N H x N Wone block. For example, the output of level-1 encoder includes one latent head vector and one latent feature vector. The input of level-2 encoder is the latent head vector of level-1 encoder, and the latent feature vector is used for feature reconstruction as the input of level-1 decoder. Similar to level-1 encoder, level-2 and level-3 encoders also follow this passing process. The encoder formula of level-i Transformer is as follows,

[0029]

[0030] where j = {1,...,P i × P i} is the pre-trained feature vector. j = {0,...,P i × P i} is the output of the i-th level encoder. z0 is a zero padding vector, i is the i-th level encoder, and n is the size of the patch block.

[0031] Step 3_3 reconstructs the feature image data by a multi-level Transformer decoder. The input of the decoder consists of two parts, one is the latent feature vector from the output of the same level encoder, and the other is the output of the previous level decoder, i.e. the latent head vector. In addition, the output of the decoder is the reconstructed feature of each patch, which is reconstructed into a feature map of HxW shape. The decoder formula of level-i Transformer is as follows,

[0032]

[0033] where j = {1,...,P i × P i} is the output of the same level encoder using a skip connection, is the output of the previous level transformer encoder, j = {0,...,P i × P i} is the output of the current decoder.

[0034] Step 3_4, based on the pre-trained extracted image features, can obtain the reconstructed feature map through a symmetric multi-level U-shaped Transformer The calculation formula is as follows:

[0035]

[0036] where R(·) is the multi-level transformer autoencoder network, and z is the picture feature extracted by the pre-trained network.

[0037] Step 3_5 calculates the reconstruction feature loss As Figure 3 shown, the calculation needs to use the output z of the original feature extraction network and the output of the multi-level Transformer autoencoder.

[0038]

[0039] where represents the reconstruction loss between the reconstructed features and the original features through the l2 distance.

[0040] Step 3_6 loss backpropagation. According to the reconstruction loss value in step 3_5, the gradient is backpropagated to update the weights and biases of the network model, so that the entire multi-level Transformer autoencoder network learns the data features of non-landslide data.

[0041] 4. Model testing phase

[0042] The model in the test phase is as shown in Figure 4 , and the test data includes both non-landslide data and landslide data. The test data is sequentially extracted through the pre-trained network, the multi-level Transformer autoencoder network is reconstructed, and finally the reconstruction error is obtained. Since the reconstruction error can reflect the difference between landslide and non-landslide images, the reconstruction error can be directly used as an anomaly score. When the anomaly score exceeds the predefined threshold, the image is classified as a landslide image.

[0043] Step 4_1 loads the test set data.

[0044] Step 4_2 passes the test data through the test model in Figure 4 . For example, for the i-th test sample x i , first pass through the pre-trained ResNet-18 feature extraction network to obtain the pre-trained feature z i , and then pass through the multi-level Transformer autoencoder for feature reconstruction to obtain the reconstructed image feature The formula is as follows:

[0045]

[0046] Step 4_3, the reconstructed features are compared with the real features, the reconstruction errors between the features are calculated, the reconstruction errors are taken as anomaly scores, the anomaly scores are standardized, and the AUC (Area Under the Receiver Operating Characteristic Curve), accuracy, recall rate, precision rate and F1 score of landslide detection are calculated by using the anomaly scores and label values.

[0047] (B) Beneficial effects

[0048] 1. The present application solves the problem of incomplete feature extraction of high-dimensional satellite image data. By using a convolutional neural network to automatically extract multi-layer feature representations of images, the extracted features have both coarse-grained pixel-level features and fine-grained high-level semantic features, providing more discriminative feature representations for downstream landslide detection tasks.

[0049] 2. The present application solves the error accumulation and overfitting problem of traditional autoencoders reconstructing data. By using a multi-level Transformer autoencoder network to reconstruct feature data, the feature representations of each level are jointly learned, and the data is transmitted between the same layers through a skip connection, effectively reducing error accumulation and overfitting, and improving the accuracy of landslide detection. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The method flowchart of the present application.

[0051] Figure 2 The feature extraction model diagram of the present application.

[0052] Figure 3 The formal training model diagram of the present application.

[0053] Figure 4 The test model diagram of the present application. DETAILED DESCRIPTION

[0054] To make the technical solutions of the present application clearer and more complete, the present application is further described below in conjunction with the drawings and examples.

[0055] The following uses an open remote sensing landslide dataset named Bijie landslide dataset obtained from Wuhan University for landslide detection tasks. The dataset consists of satellite optical images, landslide boundary shape files, and digital elevation models. All images in the dataset, i.e., 770 landslide images (red dots) and 2003 non-landslide images, are cropped from captured TripleSat satellite images. This patent will detect landslide images as a single classification task. The training phase only uses non-landslide images, and the test phase contains landslide and non-landslide images. 70% of the 2003 non-landslide images are used as normal training data, and the remaining non-landslide images and landslide images are used as test data. The purpose is to identify landslide images from various landslide and non-landslide images to verify the effectiveness of the method. As Figure 1 shown, the present application provides an unsupervised 6G satellite image landslide detection method based on a multi-level Transformer autoencoder network, comprising the following steps:

[0056] Step 1 as shown in the method flowchart of Figure 1 , the first step of the method is to input data and perform data preprocessing, and the specific steps of data preprocessing are as follows:

[0057] Step 1_1 divides the dataset into training set and test set data files. Load the png data file corresponding to the training set non-landslide data under the Data folder, and after loading, there are 1415 non-landslide image data.

[0058] Step 1_2 performs data transformation and enhancement on each data in the training set, including using the Resize method to scale the picture to 256*256 size, then using the constant method to fill all faces with the given PIL image, while rotating the image by 10° and center cropping, and finally standardizing the image, with mean and std as mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225].

[0059] Step 2 model feature extraction stage, feature extraction model as shown in Figure 2 , input the training set image data into ResNet-18 to extract features, and obtain a multi-layer feature representation by combining the output results of multiple layers, the specific steps are as follows:

[0060] Step 2_1 sends the 1415 data of the training set image into the ResNet-18 network for feature extraction.

[0061] Step 2_2 obtains the output of three different convolution layers, output[i], i = 1, 2, 3.

[0062] Step 2_3 The obtained three feature layers are outputted and added in pairs according to the channel dimension to obtain a combined joint feature representation z.

[0063] Step 2_4 z is directly taken as the feature vector to be reconstructed next.

[0064] Step 3 Model formal training stage, the formal training model is as shown in Figure 3 , using the joint feature representation z for feature reconstruction, and the main model is a multi-level Transformer autoencoder network. In this stage, the reconstruction error is calculated and backpropagated, and the specific steps are as follows:

[0065] Step 3_1 The joint feature representation z with an input channel number of 448.

[0066] Step 3_2 The feature image data is encoded by a multi-level Transformer encoder. The encoder formula of level-i Transformer is as follows,

[0067]

[0068] Wherein j={1,...,P i ×P i} is the pre-trained feature vector. j={0,...,P i ×P i} is the output of the i-th level encoder. z0 is a zero padding vector, i is the i-th level encoder, and n is the size of the patch block.

[0069] Step 3_3 The feature image data is reconstructed and decoded by a multi-level Transformer decoder. The decoder formula of level-i Transformer is as follows,

[0070]

[0071] Wherein j={1,...,P i ×P i} is the output of the same level encoder using a skip connection, is the output of the previous level transformer encoder, j={0,...,P i ×P i} is the output of the current decoder.

[0072] Step 3_4 Based on the pre-trained extracted image features, the reconstructed feature image can be obtained by a symmetric multi-level U-shaped Transformer, and the calculation formula is as follows:

[0073]

[0074] where R(·) is the multi-level transformer autoencoder network, z is the picture feature extracted by the pre-trained network.

[0075] Step 3_5 calculates the reconstruction feature loss As shown in Figure 3 , the calculation of requires the output z of the original feature extraction network and the output of the multi-level Transformer autoencoder The calculation formula is as follows:

[0076]

[0077] where represents the reconstruction loss between the reconstructed features and the original features through the l2 distance calculation.

[0078] Step 3_6 loss backpropagation. According to the reconstruction loss value, the gradient is backpropagated to update the weights and biases of the network model, so that the entire multi-level Transformer autoencoder network learns the data features of non-landslide data.

[0079] Step 4 model test phase, the model in the test phase is as shown in Figure 4 , the test data includes non-landslide data and landslide data, the test data is extracted through the pre-trained network, the multi-level Transformer autoencoder network is reconstructed, and finally the reconstruction error is obtained. Since the reconstruction error can reflect the difference between landslide and non-landslide images, the reconstruction error can be directly used as an anomaly score. When the anomaly score exceeds the predefined threshold, the image is classified as a landslide image.

[0080] Step 4_1 loads 588 non-landslide data and 770 landslide data.

[0081] Step 4_2 passes the test data through the test model in Figure 4 . For example, for the i-th test sample x i , first pass through the pre-trained ResNet-18 feature extraction network to obtain the pre-trained feature z i , then pass through the multi-level Transformer autoencoder for feature reconstruction to obtain the reconstructed image feature The formula is as follows:

[0082]

[0083] ​Step 4_3, the reconstructed features are compared with the real features, the reconstruction errors between the features are calculated, and the reconstruction errors are taken as abnormal scores.

[0084] Step 4_4, after the abnormal scores are standardized, the AUC, accuracy, recall rate, precision and F1 score of landslide detection are calculated by using the abnormal scores and label values, and the traditional U-Net model is compared.

[0085] The 6G satellite image landslide detection method based on the multi-level Transformer self-encoder network in the embodiment of the application can perform feature fusion and reconstruction on the extracted feature images in a new multi-level propagation and jump connection manner, simultaneously learn low-level pixel features of coarse granularity and high-level semantic features of fine granularity, and has better performance compared with original self-encoder methods.

[0086] The above is a detailed description of the embodiments of the application in combination with the drawings. The specific embodiments are only used to help understand the method of the application. According to the idea of the application, the specific embodiments and application range can be changed and modified by those skilled in the art, and therefore the application should not be understood as limiting the application.

Claims

1. A satellite landslide image detection method based on a multi-level Transformer network, characterized in that... The method includes the following steps: Step 1: Data preprocessing, including partitioning the raw data and data transformation and enhancement; Step 2: Model feature extraction stage, extracting a multi-layer joint feature representation, the specific steps are as follows: Step 2_1 Input the training set data into ResNet-18 for feature extraction to obtain the outputs of three different convolutional layers; Step 2_2: Add the outputs of the three feature layers pairwise according to the channel dimension to obtain a combined joint feature representation; Step 2_3 directly uses the joint feature representation as the feature vector to be reconstructed next; Step 3: In the formal training phase of the model, a multi-level Transformer autoencoder is used for training. The specific steps are as follows: Step 3_1 Input the feature data pre-trained by ResNet-18; Step 3_2: Encode the feature image data using a multi-level Transformer encoder. The encoder formula for level-iTransformer is as follows: in These are pre-trained feature vectors. z0 is the output of the i-th level encoder, z0 is the zero-padding vector, i is the i-th level encoder, and n is the size of the patch block; Step 3_3 reconstructs and decodes the feature image data using a multi-level Transformer decoder. The decoder formula for the level-iTransformer is as follows: in It is the output of a sibling encoder using skip connections. It is the output of the previous stage transformer encoder. This is the output of the current decoder; Step 3_4: Based on the image features extracted during pre-training, the reconstructed feature map ẑ can be obtained through a symmetrical multi-level U-shaped Transformer. The calculation formula is as follows: in It is a multi-level Transformer autoencoder network, and z is the image feature extracted through the pre-trained network; Step 3_5 Calculate the reconstructed feature loss As shown in Figure 3, The calculation requires the output of the original feature extraction network. and the output of a multi-stage Transformer autoencoder The calculation formula is as follows: in The reconstruction loss is calculated using the L2 distance between the reconstructed features and the original features. Step 3_6 Backpropagation of loss: Backpropagate gradients based on the reconstructed loss value to update the weights and biases of the network model, so that the entire multi-level Transformer autoencoder network learns the data features of non-slippery data. Step 4: Model testing phase, using the trained model to test data including landslide and non-landslide data.

2. The satellite landslide image detection method based on a multi-level Transformer network according to claim 1, characterized in that, The data preprocessing module in step 1 has the following specific steps: Step 1_1: Divide the dataset into training sets. and test set Data files; Step 1_2 Data Transformation and Enhancement: Transform and enhance image data using methods such as image scaling, padding, rotation, and center cropping.

3. The satellite landslide image detection method based on a multi-level Transformer network according to claim 1, characterized in that, The model testing module in step 4 includes the following steps: Step 4_1 Load the test dataset; Step 4_2: Apply the test data to the test model shown in Figure 4. For example, for the i-th test sample... First, pre-trained features are obtained through a pre-trained ResNet-18 feature extraction network. The features are then reconstructed through a multi-stage Transformer autoencoder to obtain the reconstructed image features. The formula is shown below: Step 4_3 compares the reconstructed features with the true features, calculates the reconstruction error between the features, and treats the reconstruction errors as outlier scores. After standardizing the outlier scores in step 4_4, the AUC, accuracy, recall, precision, and F1 score of landslide detection are calculated using the outlier scores and label values.

Images