Landslide identification method and system based on attention mechanism and multimodal representation learning

Abstract

The invention discloses a landslide identification method and system based on attention mechanism and multimodal representation learning. The method comprises the following steps: dividing positive samples containing landslides and negative samples containing non-landslides into a training set, a validation set and a test set; performing data enhancement on the training set, adjusting the image sizes of the validation set, the test set and the training set after data enhancement, and normalizing the pixel values ​​of each channel of the image; constructing a multi-way convolutional neural network based on attention mechanism and multimodal representation learning; training the multi-way convolutional neural network based on attention mechanism and multimodal representation learning by using a cross entropy loss function; training the multi-way convolutional neural network of attention mechanism and multimodal representation learning after training by using a normalized training set, and verifying by using a normalized validation set, and saving the network model with the best performance on the validation set; and testing the saved network model by using a normalized test set to obtain a landslide identification result, thereby reducing the consumption of computing resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a landslide identification method and system based on attention mechanism and multimodal representation learning. Background Art

[0002] Landslide identification is an image analysis technology that determines the area, scale and distribution of landslides by analyzing the morphology and characteristics of the landslide area. Landslide identification is the basis for landslide hazard assessment, the premise for the rational allocation of monitoring resources and effective early warning, and occupies a key position in the study of landslide hazards.

[0003] However, when identifying landslides, most landslides have a noticeable roughness and prominent texture features. The landslides are often mixed with some vegetation, showing a mixture of soil and vegetation, making their appearance in optical images more complicated. At the same time, due to the obvious bare ground features, the features of landslides are easily confused with those of roads, residential areas and other landforms, making identification somewhat challenging.

[0004] There is a method for dynamic identification and monitoring of mining geological disasters based on multi-source remote sensing data. First, the high-precision optical image information of the area is obtained by using drone technology and the high-precision ground elevation information (DEM) is obtained by using airborne laser radar. The two registered data are used to generate a three-dimensional geological model of the area. The historical data archived in the area is used to measure its deformation, interpret the disaster, and identify the geological disaster. The disadvantage of this method is that the use of high-precision images and elevation data to generate a three-dimensional model will greatly consume the computer's memory and computing resources, which will make this technology unable to be applied in embedded mobile platforms with limited memory and computing resources. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a landslide identification method and system based on attention mechanism and multimodal representation learning in view of the deficiencies in the above-mentioned prior art, which uses hyperspectral RGB data and digital elevation model (DEM) data as input of the landslide identification network model to identify whether there is a landslide in a certain area, thereby improving the recognition accuracy of landslides with less computing resources.

[0006] The present invention adopts the following technical solutions:

[0007] A landslide identification method based on attention mechanism and multimodal representation learning includes the following steps:

[0008] S1, divide the positive samples containing landslides and the negative samples containing non-landslides into training set, validation set and test set;

[0009] S2, perform data enhancement on the training set divided in step S1, then adjust the image sizes of the validation set, the test set divided in step S1, and the training set after data enhancement in step S2, and normalize the pixel values ​​of each channel of the image;

[0010] S3. Construct a multi-way convolutional neural network based on attention mechanism and multimodal representation learning;

[0011] S4, using a cross entropy loss function to train the multi-way convolutional neural network based on attention mechanism and multimodal representation learning constructed in step S3;

[0012] S5, using the training set normalized in step S2 to train the attention mechanism and the multi-way convolutional neural network for multimodal representation learning after training in step S4, and using the validation set normalized in step S2 for validation, and saving the network model with the best performance on the validation set;

[0013] S6. Use the test set normalized in step S2 to test on the network model saved in step S5 to obtain a landslide identification result.

[0014] Specifically, in step S1, the positive samples containing landslides and the negative samples containing non-landslides are randomly divided into 10 parts, and divided into a training set, a validation set and a test set in a ratio of 6:1:3.

[0015] Specifically, step S2 is as follows:

[0016] S201, simultaneously read the RGB data and DEM data of a certain area in the training set, perform data enhancement by horizontally flipping, vertically flipping, and rotating the data by 90 degrees according to probability, and add Gaussian or salt and pepper noise to complete the training set data enhancement;

[0017] S202, using a bilinear interpolation method to adjust the image sizes of the validation set, the test set, and the training set after data enhancement in step S201 to 224×224, and normalize the pixel values ​​of each channel to 0~1.

[0018] Specifically, step S3 is as follows:

[0019] S301, constructing a multi-channel convolutional neural network including an RGB branch, a DEM branch and a multimodal fusion branch, respectively extracting features from the hyperspectral RGB data and the digital elevation DEM data, obtaining an RGB deep feature map, a DEM deep feature map and a deep feature map fused from the two modal data of the RGB deep feature map and the DEM deep feature map, and making classification decisions;

[0020] S302, constructing the input module of the multi-channel convolutional neural network in step S1, wherein the input modules of the RGB branch and the DEM branch are respectively composed of a convolutional layer, a batch normalization layer, and a RELU nonlinear activation layer; inputting the RGB data and the DEM data in the training set after data enhancement in step S2 into the input modules of each branch of the multi-channel convolutional neural network, and obtaining 64 shallow feature maps of RGB data and 64 shallow feature maps of DEM data through the first input convolutional layer;

[0021] S303, construct an attention feature fusion module including a channel attention module and a spatial attention module, input the shallow feature map of 64 RGB data and the shallow feature map of 64 DEM data obtained in step S302 into the spatial attention modules of their respective branches, and obtain the RGB spatial attention feature map and the DEM spatial attention feature map; and input the RGB spatial attention feature map and the DEM spatial attention feature map into the channel attention module after data fusion, and obtain the channel attention feature map of the fused data;

[0022] S304, construct a residual module including 3 convolutional layers, 3 batch normalization layers, and 2 RELU nonlinear activation layers;

[0023] S305, based on the residual module of step S304, construct super-module RGB, DEM and fused data feature extraction branches, input the shallow feature maps of 64 RGB data and 64 DEM data obtained in step S302 into their respective super-module feature extraction branches, respectively, to obtain their respective RGB feature maps and DEM feature maps; input the fused data channel attention feature map obtained in step S303 into the super-module feature extraction branch of the fused data, to obtain the feature map of the fused data;

[0024] S306, construct a deep feature extraction module, input the RGB feature map and the DEM feature map obtained in step S305 into the spatial attention module of each branch in step S303, and obtain an RGB spatial attention feature map and a DEM spatial attention feature map; and perform data fusion on the RGB spatial attention feature map, the DEM spatial attention feature map and the fused data feature map obtained in step S305, and input them into the channel attention module in S303 to obtain a channel attention feature map of the fused data, and input the RGB spatial attention feature map, the DEM spatial attention feature map and the channel attention feature map of the fused data into the super module feature extraction layer of each branch to obtain respective deep feature maps;

[0025] S307, repeat the deep feature extraction module in step S306, and finally obtain 1024 RGB depth feature map, 1024 DEM depth feature map and 1024 The deep feature map of fused data;

[0026] S308, construct a multimodal shared decision module to perform classification decisions on the RGB depth feature map, 1024 DEM depth feature maps, and 1024 fused data depth feature maps obtained in step S307.

[0027] Furthermore, in step S305, the first super module of each branch contains 3 residual modules, and the number of input convolution kernels of the first residual module is 64; in step S306, the first super module of each branch contains 4 residual modules, and the number of input convolution kernels of the first residual module is 128; the second super module contains 6 residual modules, and the number of input convolution kernels of the first residual module is 256; the third super module contains 3 residual modules, and the number of input convolution kernels of the first residual module is 512.

[0028] Further, in step S308, the multimodal shared decision module includes four groups of parallel multilayer perceptron networks and one group of fusion decision networks, each multilayer perceptron network includes two fully connected layers and one softmax layer, the fully connected modules belonging to the RGB branch, the DEM branch and the multimodal fusion branch are the first multilayer perceptron network, the second multilayer perceptron network and the third multilayer perceptron network, respectively, and the number of nodes in each layer is 1024, 512, and 2 respectively. The first layer of the first multilayer perceptron network, the second multilayer perceptron network and the third multilayer perceptron network are spliced ​​as the first layer of the fourth multilayer perceptron network, and the number of nodes in each layer of the fourth multilayer perceptron network is 3072, 512, and 2 respectively; the fusion decision network includes one fully connected layer and one softmax layer, and the number of nodes in each layer is 8 and 2 respectively; the third layers of the four groups of multilayer perceptron networks are spliced ​​and input into the first layer of the fusion decision network to obtain the final landslide recognition result.

[0029] Specifically, in step S4, the cross entropy loss function for:

[0030]

[0031] in, For labels, Predicted probabilities for landslide identification networks.

[0032] Specifically, in step S4, the multi-way convolutional neural network for attention mechanism and multimodal representation learning is trained by using the stochastic gradient descent optimization method. First, the loss function right Find the partial derivative, ,parameter Update in the direction of negative gradient, , are the updated network parameters, are the network parameters before updating, is the learning rate, is the training data input to the network, is the weight of the training set, is the label corresponding to the training set, The number of samples input for each training is a random set of samples from the training set, which are updated according to the gradient descent rule after each training.

[0033] Furthermore, the loss function as follows:

[0034]

[0035] in, For the training set.

[0036] Another technical solution of the present invention is a landslide identification system based on attention mechanism and multimodal representation learning, comprising:

[0037] A partitioning module divides the positive samples containing landslides and the negative samples containing non-landslides into training sets, validation sets, and test sets;

[0038] The preprocessing module performs data augmentation on the training set, adjusts the image sizes of the validation set, test set, and data augmented training set, and normalizes the pixel values ​​of each channel of the image;

[0039] Network module, building a multi-way convolutional neural network based on attention mechanism and multimodal representation learning;

[0040] The training module uses the cross entropy loss function to train the multi-way convolutional neural network based on the attention mechanism and multimodal representation learning of the network module;

[0041] The verification module uses the training set normalized by the preprocessing module to train the attention mechanism of the training module and the multi-way convolutional neural network for multimodal representation learning, and uses the verification set normalized by the preprocessing module for verification, saving the network model with the best performance on the verification set;

[0042] The recognition module uses the test set normalized by the preprocessing module to test on the network model saved by the verification module to obtain the landslide recognition result.

[0043] Compared with the prior art, the present invention has at least the following beneficial effects:

[0044] The present invention discloses a landslide identification method based on attention mechanism and multimodal representation learning. The attention mechanism is used to perform multi-scale fusion of multimodal data to solve the problem that the existing method only performs a single-mode fusion on multimodal data and may lose multimodal high-level semantic features, thereby improving the accuracy of landslide identification.

[0045] Furthermore, the positive samples containing landslides and the negative samples without landslides were randomly divided into 10 parts and then randomly divided into training set, validation set and test set three times in the ratio of 6:1:3. The cross-validation method was used to reduce the chance of too high or too low recognition accuracy due to a single division of the data set.

[0046] Furthermore, data enhancement of training samples can effectively expand the training data and avoid overfitting caused by using small data to train the network; image data normalization normalizes all image data to the same scale, which is conducive to improving the convergence speed and accuracy of the model.

[0047] Furthermore, constructing a landslide recognition network based on attention mechanism and multimodal representation learning is conducive to extracting multimodal data features step by step and fusing them using the attention mechanism. The multimodal shared decision module is conducive to integrating the extracted feature maps of each modality, which is more conducive to the identification of regional landslides.

[0048] Furthermore, the use of a convolutional neural network with a cascade of multiple residual modules is helpful in avoiding the situation where the gradient disappears or explodes due to the network being too deep.

[0049] Furthermore, the multimodal shared decision module includes four groups of multi-layer perceptron structures, which respectively perform classification decisions on the deep feature maps of each modal data obtained in S305, and input the decision results of three groups of multi-layer perceptrons into the fourth group of multi-layer perceptrons. The shared decision can not only integrate the decision results of each modality and fusion data to achieve better and more accurate decisions, but also effectively reduce the number of parameters in the decision layer.

[0050] Furthermore, the cross entropy loss function can measure the difference between two different probability distributions in the same random variable, which is expressed as the difference between the true probability distribution and the predicted probability distribution in machine learning. The smaller the cross entropy value, the better the model prediction effect.

[0051] Furthermore, in the training of the multi-way convolutional neural network with attention mechanism and multimodal representation learning, in order to minimize the time function, the stochastic gradient descent optimization method is used to fine-tune the network parameters in order to find the optimal network parameters so that the value of the loss function is minimized.

[0052] Furthermore, the loss function The performance of the current network is evaluated by calculating the distance between the network prediction results and the label data.

[0053] In summary, the present invention utilizes the attention mechanism and the multimodal shared decision module to perform multi-scale fusion and decision-making on the features of each modality for multiple times, which can effectively avoid the influence of the final classification result due to the difficulty in distinguishing a single modal feature, and realize the effective identification of landslides.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of the present invention;

[0056] Figure 2 is a network model block diagram of the present invention;

[0057] Figure 3 This is a block diagram of the channel attention module in the present invention;

[0058] Figure 4 This is a block diagram of the spatial attention module in the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0061] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0062] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0063] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0064] The present invention provides a landslide identification method based on attention mechanism and multimodal representation learning, firstly, the training set, validation set and test set are divided; then the training set is data enhanced; then a multi-way convolutional neural network based on attention mechanism and multimodal representation learning is constructed; the network model is trained using the training set, and the model with the best performance on the validation set is saved; finally, the test set is tested with the trained model to obtain the results of various indicators of landslide identification. The network result constructed by the present invention can realize the step-by-step feature extraction and fusion of multimodal data, effectively avoiding the loss of deep information of the modality due to premature fusion of multimodal data; and the spatial attention module and the channel attention module are integrated in the multi-level fusion mechanism to emphasize the important area (i.e., landslide) in the feature, and the spatial and channel integration can not only reduce the amount of calculation, but also make full use of the global consistency of space and channel, so that the feature is more separable; the present invention has also made improvements in the decision layer, and the classification of the decision layer integrates the decision results of multiple modal features and fusion features, which reduces the parameter amount of the traditional classifier while also improving the result parameters.

[0065] See also Figure 1 The present invention provides a landslide identification method based on attention mechanism and multimodal representation learning, comprising the following steps:

[0066] S1. Sample set division

[0067] The Bijie dataset was divided into training set, validation set and test set using Matlab software. Different random seeds were set and the dataset was randomly divided three times to obtain three groups of data to be trained.

[0068] The positive samples containing landslides and the negative samples without landslides were randomly divided into 10 parts in the ratio of 6:1:3 as training set, validation set and test set respectively.

[0069] S2. Multimodal data preprocessing

[0070] Set three random numbers to control the horizontal flip, vertical flip and 90-degree rotation of the image respectively. The probability of random Gaussian or salt and pepper noise is set to 0.7. Normalize the image size to , and normalize the pixel values ​​of the multimodal image to .

[0071] S201. Perform data enhancement on training set samples

[0072] At the same time, the RGB data and DEM data of a certain area in the training set divided in step S1 are read, and the data is enhanced by horizontally flipping, vertically flipping, and rotating 90 degrees according to probability, and Gaussian or salt and pepper noise is added to the data;

[0073] S202, adjusting the image sizes of the training set, validation set, and test set samples after data enhancement in step S201 to 224×224 using a bilinear interpolation method, and normalizing the pixel values ​​of each channel to 0-1;

[0074] The normalization formula is as follows:

[0075]

[0076] in, Represents the normalized RGB image or DEM image. I Represents the RGB image or DEM image before normalization. Represents the maximum and minimum operations respectively.

[0077] S3. Construct a multi-way convolutional neural network based on attention mechanism and multimodal representation learning;

[0078] S301, overall framework

[0079] A multi-channel convolutional neural network framework consisting of an RGB branch, a DEM branch, and a multimodal fusion branch is constructed to extract the features of hyperspectral RGB data, digital elevation DEM data, and the feature map after the fusion of the two modal data, and perform multimodal decision fusion on the obtained feature map. Each branch network is composed of an input module, four superblocks containing different numbers of residual modules, and a multimodal shared decision module. The branches are connected by attention modules.

[0080] The learning rate is 0.015; the batch size is 16; the loss function is the cross entropy loss function; and the network optimization algorithm is the stochastic gradient descent algorithm.

[0081] S302, constructing the input module of the multi-channel convolutional neural network in step S1, the input modules of the RGB branch and the DEM branch are respectively composed of a convolution layer, a batch normalization layer and a RELU nonlinear activation layer; inputting the RGB data and DEM data in the training set after data enhancement in step S2 into the input modules of each branch of the multi-channel convolutional neural network, and obtaining 64 shallow feature maps of RGB data and 64 shallow feature maps of DEM data through the first input convolution layer.

[0082] Construct a super-module feature extraction branch, each super-module contains a certain number of basic residual modules constructed in step S303, and input the RGB feature map and DEM feature map obtained in step S304 into the next super-module (SuperBlock) of their respective branches to obtain their respective RGB depth feature maps and DEM depth feature maps; input the fused data channel attention feature map obtained in step S305 into the first super-module of the fused data branch to obtain the depth feature map of the fused data.

[0083] The first supermodule (SuperBlock1) of each branch contains 3 residual modules and 64 convolution kernels; the first supermodule (SuperBlock2) contains 4 residual modules and 128 convolution kernels; the second supermodule (SuperBlock3) contains 6 residual modules and 256 convolution kernels; the third supermodule (SuperBlock4) contains 3 residual modules and 512 convolution kernels.

[0084] S303, construct an attention feature fusion module including a channel attention module and a spatial attention module, input the shallow feature map of 64 RGB data and the shallow feature map of 64 DEM data obtained in step S302 into the spatial attention modules of their respective branches, and obtain the RGB spatial attention feature map and the DEM spatial attention feature map; and input the RGB spatial attention feature map and the DEM spatial attention feature map into the channel attention module after data fusion, and obtain the channel attention feature map of the fused data;

[0085] See also Figure 3 and Figure 4 , the channel attention module and the spatial attention module are specifically:

[0086] Channel attention module: The input of each channel attention module consists of a parallel maximum pooling layer and an average pooling layer, which is then connected to the Sigmoid layer through a shared perceptron (consisting of two fully connected layers);

[0087] Spatial attention module: The input of each spatial attention module consists of a channel-wise max pooling layer and an average pooling layer, followed by a cascaded dilated convolution (four parallel dilated convolutions of different scales) to aggregate information, and then a sigmoid layer.

[0088] The RGB feature map and the DEM feature map are input into the spatial attention modules of their respective branches. The cascaded dilated convolution module in the spatial attention module contains dilated convolutions with four sets of scale dilated rates, and the dilated rates are 1, 6, 12, and 18, respectively. Four sets of feature maps are obtained, which are then restored to the size of the source input feature map through bilinear interpolation. Finally, the feature maps of the four scales are fused together to obtain the RGB spatial attention feature map and the DEM spatial attention feature map, respectively.

[0089] The RGB spatial attention feature map and the DEM spatial attention feature map are fused and input into the channel attention module to obtain the channel attention feature map of the fused data.

[0090] S304, construct a residual module including 3 convolutional layers, 3 batch normalization layers, and 2 RELU nonlinear activation layers;

[0091] The residual module consists of 3 convolutional layers, 3 batch normalization layers, and 2 RELU nonlinear activation layers. The convolution kernel sizes of the 3 convolutional layers are: , , , each convolutional layer is followed by a batch normalization layer and a RELU activation layer, and the final output layer does not have a RELU activation layer. The output layer is residually connected to the input as the final output of the module, and the residual module is used as the basic module in the super module in S304.

[0092] S305, based on the residual module of step S304, construct super-module RGB, DEM and fused data feature extraction branches, input the shallow feature maps of 64 RGB data and 64 DEM data obtained in step S302 into their respective super-module feature extraction branches, respectively, to obtain their respective RGB feature maps and DEM feature maps; input the fused data channel attention feature map obtained in step S303 into the super-module feature extraction branch of the fused data, to obtain the feature map of the fused data;

[0093] S306, construct a deep feature extraction module, input the RGB feature map and the DEM feature map obtained in step S305 into the spatial attention module of each branch in step S303, and obtain an RGB spatial attention feature map and a DEM spatial attention feature map; and perform data fusion on the RGB spatial attention feature map, the DEM spatial attention feature map and the fused data feature map obtained in step S305, and input them into the channel attention module in S303 to obtain a channel attention feature map of the fused data, and input the RGB spatial attention feature map, the DEM spatial attention feature map and the channel attention feature map of the fused data into the super module feature extraction layer of each branch to obtain respective deep feature maps;

[0094] S307, repeat the deep feature extraction module in step S306, and finally obtain 1024 RGB depth feature map, 1024 DEM depth feature map and 1024 The fused data deep feature maps are used as inputs of the three groups of MLPs in S306 respectively.

[0095] S308, multi-modal shared decision module

[0096] The RGB depth feature map, 1024 DEM depth feature maps and 1024 fused data depth feature maps obtained in step S307 are input into the multimodal shared decision module. First, each group of depth feature maps is stretched into a vector and input into the MLP module of the corresponding branch to obtain the decision results of the respective modes of RGB, DEM and fused data; and the results of the first fully connected layer of each branch are spliced, and the decision layer is fused and input into the fused decision MLP to obtain the fused decision result; finally, the four decision results are fused and input into the multimodal shared decision module to obtain the landslide identification result in the area.

[0097] The multimodal shared decision module consists of four parallel multi-layer perceptron networks (MLP) and a fusion decision network (FDN). Each MLP contains two fully connected layers and one softmax layer. The fully connected modules belonging to the RGB branch, DEM branch and multimodal fusion branch are MLP1, MLP2 and MLP3, respectively, and the number of nodes in each layer is 1024, 512 and 2 respectively. The first layers of MLP1, MLP2 and MLP3 are concatenated as the first layer of MLP4, so the number of nodes in each layer of MLP4 is 3072, 512 and 2 respectively. The fusion decision network consists of one fully connected layer and one softmax layer, and the number of nodes in each layer is 8 and 2 respectively. The third layers of the four groups of MLPs are concatenated and input into the first layer of FDN to obtain the final landslide recognition results.

[0098] S4, training a multi-way convolutional neural network with attention mechanism and multimodal representation learning using cross entropy loss function;

[0099] During the training process, the loss function uses the cross entropy loss function and the optimization algorithm uses stochastic gradient descent.

[0100] The cross entropy loss function is defined as:

[0101]

[0102] in, is the label (1 for landslide and 0 for non-landslide), Predicted probabilities for landslide identification networks.

[0103] Define gradient descent as follows:

[0104] Given the loss function:

[0105]

[0106] in, is the number of samples input for each training, is the training set, is the label corresponding to the training set, is the weight of the training set.

[0107] According to the gradient descent rule, we first use right Find the partial derivative, , since we want to minimize the loss function, the parameter Update according to its negative gradient direction, Stochastic gradient descent refers to randomly extracting a group of samples from the training set and updating them according to the gradient descent rule after each training.

[0108] S5, using the training set to train the attention mechanism and the multi-way convolutional neural network for multimodal representation learning after the training in step S4, and verifying it on the validation set, and saving the network model parameters with the best performance on the validation set;

[0109] S6. Use the test set to test on the saved network model to obtain the results of landslide identification.

[0110] In another embodiment of the present invention, a landslide identification system based on attention mechanism and multimodal representation learning is provided. The system can be used to implement the above-mentioned landslide identification method based on attention mechanism and multimodal representation learning. Specifically, the landslide identification system based on attention mechanism and multimodal representation learning includes a division module, a preprocessing module, a network module, a training module, a verification module and an identification module.

[0111] Among them, the division module divides the positive samples containing landslides and the negative samples containing non-landslides into training sets, validation sets and test sets;

[0112] The preprocessing module performs data augmentation on the training set, adjusts the image sizes of the validation set, test set, and data augmented training set, and normalizes the pixel values ​​of each channel of the image;

[0113] Network module, building a multi-way convolutional neural network based on attention mechanism and multimodal representation learning;

[0114] The training module uses the cross entropy loss function to train the multi-way convolutional neural network based on the attention mechanism and multimodal representation learning of the network module;

[0115] The verification module uses the training set normalized by the preprocessing module to train the attention mechanism of the training module and the multi-way convolutional neural network for multimodal representation learning, and uses the verification set normalized by the preprocessing module for verification, saving the network model with the best performance on the verification set;

[0116] The recognition module uses the test set normalized by the preprocessing module to test on the network model saved by the verification module to obtain the landslide recognition result.

[0117] In another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the landslide identification method based on the attention mechanism and multimodal representation learning, including:

[0118] The positive samples containing landslides and the negative samples containing non-landslides are divided into training set, validation set and test set; data enhancement is performed on the training set, and then the image sizes of the validation set, test set and training set after data enhancement are adjusted, and the pixel values ​​of each channel of the image are normalized; a multi-way convolutional neural network based on attention mechanism and multimodal representation learning is constructed; the multi-way convolutional neural network based on attention mechanism and multimodal representation learning is trained using the cross entropy loss function; the multi-way convolutional neural network with attention mechanism and multimodal representation learning after training is trained using the normalized training set, and the normalized validation set is used for verification, and the network model with the best performance on the validation set is saved; the normalized test set is used to test on the saved network model to obtain the landslide recognition result.

[0119] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory.

[0120] One or more instructions stored in a computer-readable storage medium may be loaded and executed by a processor to implement the corresponding steps of the landslide identification method based on attention mechanism and multimodal representation learning in the above embodiment; one or more instructions in the computer-readable storage medium are loaded and executed by the processor as follows:

[0121] The positive samples containing landslides and the negative samples containing non-landslides are divided into training set, validation set and test set; data enhancement is performed on the training set, and then the image sizes of the validation set, test set and training set after data enhancement are adjusted, and the pixel values ​​of each channel of the image are normalized; a multi-way convolutional neural network based on attention mechanism and multimodal representation learning is constructed; the multi-way convolutional neural network based on attention mechanism and multimodal representation learning is trained using the cross entropy loss function; the multi-way convolutional neural network with attention mechanism and multimodal representation learning after training is trained using the normalized training set, and the normalized validation set is used for verification, and the network model with the best performance on the validation set is saved; the normalized test set is used to test on the saved network model to obtain the landslide recognition result.

[0122] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0123] Simulation experiment

[0124] 1. Experimental conditions:

[0125] The simulation experiment platform of the present invention adopts an Intel E5-2600 v3 CPU processor, 2.4GHz main frequency, 64G memory, NVIDIA RTX3090 graphics card, a PC running Ubuntu18.04 version 64-bit system, the software platform is Pytorch, and the programming language is Python.

[0126] 2. Simulation content and result analysis:

[0127] The samples in the simulation experiment of the present invention are from the open remote sensing landslide dataset of Bijie landslide dataset, and the attention convolutional neural network model is constructed, such as Figure 2 As shown, the training set, validation set, and test set are divided to prepare the data.

[0128] The sizes of the images in the Bijie landslide data used in this paper are inconsistent. In this experiment, the sizes are normalized to .

[0129] The present invention compares the classification results of different networks for multimodal data, wherein the network structures of VGG16 and Res_att are single input branches, so the RGB data and the DEM data have been fused when input. The method of the present invention improves the feature extraction method of multimodal data, which is no longer similar to the above-mentioned single fusion method, but divides the DEM and RGB data into two branches of feature extraction networks, and performs multi-level data fusion while extracting features from the two groups of modal data. This network structure avoids the premature fusion of multimodal data and the loss of deep information of the modality, and integrates the spatial attention module and the channel attention module in the multi-level fusion mechanism to emphasize the important areas (i.e., landslides) in the features. The spatial and channel integration can not only reduce the amount of calculation, but also make full use of the global consistency of space and channels, making the features more separable. The present invention has also made improvements in the decision layer, and the classification of the decision layer fuses the decision results of multiple modal features and fused features, which reduces the number of parameters of the traditional classifier and also improves the result parameters.

[0130] The landslide identification problem is essentially a binary classification problem of images, that is, the positive samples containing landslides and the negative samples containing non-landslides are classified by a classifier, and the evaluation index is calculated using the prediction results of the classifier on the test set.

[0131] The following four evaluation indicators, namely recall rate, precision rate, F1 coefficient and accuracy rate, are used to quantitatively evaluate the recognition performance of the convolutional neural network model based on multimodal data and attention mechanism proposed in the present invention.

[0132] The following is the specific meaning of each indicator:

[0133] TP: predicting landslides as landslide numbers (True Positive);

[0134] FN: The number of landslides predicted as non-landslides (False Negative);

[0135] FP: the number of non-landslides predicted as landslides (False Positive);

[0136] TN: The number of non-landslides predicted as non-landslides (True Negative).

[0137] Recall refers to the original sample, which indicates how many positive examples in the sample are predicted correctly. There are two possibilities, one is to predict the original positive class as positive class (TP), and the other is to predict the original positive class as negative class (FN):

[0138]

[0139] Precision refers to the prediction results, which indicates how many of the samples predicted as positive are actually positive samples. There are two possibilities for predicting positive, one is to predict the positive class as positive (TP), and the other is to predict the negative class as positive (FP):

[0140]

[0141] The F1 score is an indicator used to measure the accuracy of a binary classification model, taking into account both the accuracy and recall of the classification model. It can be seen as a weighted average of the accuracy and recall:

[0142]

[0143] Classification accuracy (Accuracy, Acc) is the ratio of the number of samples correctly classified by the classifier to the total number of samples:

[0144]

[0145] Table 1 List of landslide recognition results obtained from the simulation experiment of the present invention (wherein the 3D attention mechanism module is deployed in Resnet)

[0146]

[0147] Combined with the results in Table 1, it can be seen that our method achieves better results in four indicators with a lower number of parameters. This also proves that the multi-scale data fusion method proposed in the present invention has better effects than traditional fusion methods when processing multimodal data.

[0148] In summary, the present invention provides a landslide identification method and system based on attention mechanism and multimodal representation learning, which utilizes a novel multimodal data fusion network combined with an attention mechanism. In the training process, multimodal data is subjected to multi-scale data feature fusion in the feature extraction layer, and multimodal data is subjected to data decision fusion in the classification layer, which solves the problem of losing some modal data due to a single fusion method in the existing multimodal data fusion, and the data fusion method of the decision layer also greatly reduces the number of parameters. Under the premise of ensuring high-precision landslide identification, the present invention reduces the scale of the model, promotes the application of the landslide identification algorithm in embedded platforms such as drones, and is an effective multimodal landslide identification method.

[0149] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A landslide identification method based on attention mechanism and multimodal representation learning, characterized in that: The following steps are involved: S1, divide the positive samples containing landslides and the negative samples containing non-landslides into training set, validation set and test set; S2, perform data enhancement on the training set divided in step S1, then adjust the image sizes of the validation set, the test set divided in step S1, and the training set after data enhancement in step S2, and normalize the pixel values ​​of each channel of the image; S3. Construct a multi-way convolutional neural network based on attention mechanism and multimodal representation learning, specifically: S301, constructing a multi-channel convolutional neural network including an RGB branch, a DEM branch and a multimodal fusion branch, respectively extracting features from the hyperspectral RGB data and the digital elevation DEM data, obtaining an RGB deep feature map, a DEM deep feature map and a deep feature map fused from the two modal data of the RGB deep feature map and the DEM deep feature map, and making classification decisions; S302, constructing the input module of the multi-channel convolutional neural network in step S1, wherein the input modules of the RGB branch and the DEM branch are respectively composed of a convolutional layer, a batch normalization layer, and a RELU nonlinear activation layer; inputting the RGB data and the DEM data in the training set after data enhancement in step S2 into the input modules of each branch of the multi-channel convolutional neural network, and obtaining 64 shallow feature maps of RGB data and 64 shallow feature maps of DEM data through the first input convolutional layer; S303, construct an attention feature fusion module including a channel attention module and a spatial attention module, input the shallow feature maps of 64 RGB data and the shallow feature maps of 64 DEM data obtained in step S302 into the spatial attention modules of their respective branches, and obtain the RGB spatial attention feature map and the DEM spatial attention feature map; The RGB spatial attention feature map and the DEM spatial attention feature map are fused and input into the channel attention module to obtain the channel attention feature map of the fused data; S304, construct a residual module including 3 convolutional layers, 3 batch normalization layers, and 2 RELU nonlinear activation layers; S305, based on the residual module of step S304, construct super-module RGB, DEM and fused data feature extraction branches, input the shallow feature maps of 64 RGB data and 64 DEM data obtained in step S302 into their respective super-module feature extraction branches, and obtain their respective RGB feature maps and DEM feature maps; input the fused data channel attention feature map obtained in step S303 into the super-module feature extraction branch of the fused data to obtain the feature map of the fused data, the first super-module of each branch contains 3 residual modules, and the number of input convolution kernels of the first residual module is 64; on each branch in step S306, the first super-module contains 4 residual modules, and the number of input convolution kernels of the first residual module is 128; the second super-module contains 6 residual modules, and the number of input convolution kernels of the first residual module is 256; the third super-module contains 3 residual modules, and the number of input convolution kernels of the first residual module is 512; S306, constructing a deep feature extraction module, inputting the RGB feature map and the DEM feature map obtained in step S305 into the spatial attention modules of their respective branches in step S303, and obtaining an RGB spatial attention feature map and a DEM spatial attention feature map; The RGB spatial attention feature map, the DEM spatial attention feature map and the fused data feature map obtained in step S305 are subjected to data fusion, and input into the channel attention module in S303 to obtain the channel attention feature map of the fused data, and the RGB spatial attention feature map, the DEM spatial attention feature map and the channel attention feature map of the fused data are input into the super-module feature extraction layer of each branch to obtain their respective deep feature maps; S307, repeat the deep feature extraction module in step S306, and finally obtain 1024 RGB depth feature map, 1024 DEM depth feature map and 1024 The deep feature map of fused data; S308, construct a multimodal shared decision module, and make classification decisions on the RGB depth feature map, 1024 DEM depth feature maps and 1024 fusion data depth feature maps obtained in step S307. The multimodal shared decision module includes four parallel multilayer perceptron networks and one fusion decision network. Each multilayer perceptron network includes two fully connected layers and one softmax layer. The fully connected modules belonging to the RGB branch, the DEM branch and the multimodal fusion branch are the first multilayer perceptron network, the second multilayer perceptron network and the third multilayer perceptron network. The number of nodes in each layer of the network is 1024, 512, and 2 respectively. The first layer of the first multilayer perceptron network, the second multilayer perceptron network, and the first layer of the third multilayer perceptron network are spliced ​​as the first layer of the fourth multilayer perceptron network. The number of nodes in each layer of the fourth multilayer perceptron network is 3072, 512, and 2 respectively. The fusion decision network includes a fully connected layer and a softmax layer. The number of nodes in each layer is 8 and 2 respectively. The third layers of the four groups of multilayer perceptron networks are spliced ​​and input into the first layer of the fusion decision network to obtain the final landslide recognition result. S4, using a cross entropy loss function to train the multi-way convolutional neural network based on attention mechanism and multimodal representation learning constructed in step S3; S5, using the training set normalized in step S2 to train the attention mechanism and the multi-way convolutional neural network for multimodal representation learning after training in step S4, and using the validation set normalized in step S2 for validation, and saving the network model with the best performance on the validation set; S6. Use the test set normalized in step S2 to test on the network model saved in step S5 to obtain a landslide identification result.

2. The landslide identification method based on attention mechanism and multimodal representation learning according to claim 1 is characterized in that: In step S1, the positive samples containing landslides and the negative samples without landslides are randomly divided into 10 parts, and divided into training set, validation set and test set in the ratio of 6:1:

3.

3. The landslide identification method based on attention mechanism and multimodal representation learning according to claim 1 is characterized in that: Step S2 is specifically as follows: S201, simultaneously read the RGB data and DEM data of a certain area in the training set, perform data enhancement by horizontally flipping, vertically flipping, and rotating the data by 90 degrees according to probability, and add Gaussian or salt and pepper noise to complete the training set data enhancement; S202, using a bilinear interpolation method to adjust the image sizes of the validation set, the test set, and the training set after data enhancement in step S201 to 224×224, and normalize the pixel values ​​of each channel to 0~1.

4. The landslide identification method based on attention mechanism and multimodal representation learning according to claim 1 is characterized in that: In step S4, the cross entropy loss function for: in, For labels, Predicted probabilities for landslide identification networks.

5. The landslide identification method based on attention mechanism and multimodal representation learning according to claim 1 is characterized in that: In step S4, the multi-way convolutional neural network for attention mechanism and multimodal representation learning is trained using a stochastic gradient descent optimization method. First, the loss function right Find the partial derivative, ,parameter Update in the direction of negative gradient, , are the updated network parameters, are the network parameters before updating, is the learning rate, is the training data input to the network, is the weight of the training set, is the label corresponding to the training set, The number of samples input for each training is a random set of samples from the training set, which are updated according to the gradient descent rule after each training.

6. The landslide identification method based on attention mechanism and multimodal representation learning according to claim 5 is characterized in that: Loss Function as follows: in, For the training set.

7. A landslide identification system based on attention mechanism and multimodal representation learning, characterized in that: include: A partitioning module divides the positive samples containing landslides and the negative samples containing non-landslides into training sets, validation sets, and test sets; The preprocessing module performs data augmentation on the training set, adjusts the image sizes of the validation set, test set, and data augmented training set, and normalizes the pixel values ​​of each channel of the image; The network module builds a multi-way convolutional neural network based on attention mechanism and multimodal representation learning, specifically: A multi-channel convolutional neural network including RGB branch, DEM branch and multimodal fusion branch is constructed to extract features from hyperspectral RGB data and digital elevation DEM data respectively, and obtain RGB deep feature map, DEM deep feature map and deep feature map of fusion of two modal data, RGB deep feature map and DEM deep feature map, to make classification decisions; Construct the input module of the multi-channel convolutional neural network. The input modules of the RGB branch and the DEM branch are composed of a convolution layer, a batch normalization layer, and a RELU nonlinear activation layer respectively. Input the RGB data and DEM data in the training set after data enhancement into the input modules of each branch of the multi-channel convolutional neural network. After the first input convolution layer, 64 shallow feature maps of RGB data and 64 shallow feature maps of DEM data are obtained. Construct an attention feature fusion module including a channel attention module and a spatial attention module, input the shallow feature maps of 64 RGB data and the shallow step feature maps of 64 DEM data into the spatial attention modules of their respective branches, and obtain the RGB spatial attention feature map and the DEM spatial attention feature map; The RGB spatial attention feature map and the DEM spatial attention feature map are fused and input into the channel attention module to obtain the channel attention feature map of the fused data; Construct a residual module consisting of 3 convolutional layers, 3 batch normalization layers, and 2 RELU nonlinear activation layers; Based on the residual module, super-module RGB, DEM and fusion data feature extraction branches are constructed, and the shallow feature maps of 64 RGB data and 64 DEM data are respectively input into their respective super-module feature extraction branches to obtain their respective RGB feature maps and DEM feature maps; The obtained fused data channel attention feature map is input into the super-module feature extraction branch of the fused data to obtain the feature map of the fused data. The first super-module of each branch contains 3 residual modules, and the number of input convolution kernels of the first residual module is 64; on each branch, the first super-module contains 4 residual modules, and the number of input convolution kernels of the first residual module is 128; the second super-module contains 6 residual modules, and the number of input convolution kernels of the first residual module is 256; the third super-module contains 3 residual modules, and the number of input convolution kernels of the first residual module is 512; Construct a deep feature extraction module, input the obtained RGB feature map and DEM feature map into the spatial attention module of each branch, and obtain the RGB spatial attention feature map and the DEM spatial attention feature map; The RGB spatial attention feature map, the DEM spatial attention feature map and the fused data feature map are fused and input into the channel attention module to obtain the channel attention feature map of the fused data. The RGB spatial attention feature map, the DEM spatial attention feature map and the channel attention feature map of the fused data are input into the super-module feature extraction layer of each branch to obtain their respective deep feature maps. Repeat the deep feature extraction module to finally get 1024 RGB depth feature map, 1024 DEM depth feature map and 1024 The deep feature map of fused data; A multimodal shared decision module is constructed to make classification decisions on the obtained RGB depth feature map, 1024 DEM depth feature maps and 1024 fusion data depth feature maps. The multimodal shared decision module includes four parallel multilayer perceptron networks and one fusion decision network. Each multilayer perceptron network contains two fully connected layers and one softmax layer. The fully connected modules belonging to the RGB branch, DEM branch and multimodal fusion branch are the first multilayer perceptron network, the second multilayer perceptron network and the third multilayer perceptron network, respectively. The number of nodes in each layer is 1024, 512 and 2 respectively. The first layer of the first multilayer perceptron network, the second multilayer perceptron network and the third multilayer perceptron network are spliced ​​as the first layer of the fourth multilayer perceptron network. The number of nodes in each layer of the fourth multilayer perceptron network is 3072, 512 and 2 respectively. The fusion decision network includes one fully connected layer and one softmax layer. The number of nodes in each layer is 8 and 2 respectively. The third layers of the four multilayer perceptron networks are spliced ​​and input into the first layer of the fusion decision network to obtain the final landslide recognition result. The training module uses the cross entropy loss function to train the multi-way convolutional neural network based on the attention mechanism and multimodal representation learning of the network module; The verification module uses the training set normalized by the preprocessing module to train the attention mechanism of the training module and the multi-way convolutional neural network for multimodal representation learning, and uses the verification set normalized by the preprocessing module for verification, saving the network model with the best performance on the verification set; The recognition module uses the test set normalized by the preprocessing module to test on the network model saved by the verification module to obtain the landslide recognition result.

8. The landslide identification system based on attention mechanism and multimodal representation learning according to claim 7 is characterized in that: In the preprocessing module, the RGB data and DEM data of a certain area in the training set are read at the same time, and the data is enhanced by horizontally and vertically flipping and rotating 90 degrees according to probability, and Gaussian or salt and pepper noise is added to complete the training set data enhancement; the image size of the validation set, test set and training set after data enhancement is adjusted to 224×224 using the bilinear interpolation method, and the pixel value of each channel is normalized to 0~1.

9. The landslide identification system based on attention mechanism and multimodal representation learning according to claim 7, characterized in that: In the training module, the multi-way convolutional neural network for attention mechanism and multimodal representation learning is trained using the stochastic gradient descent optimization method. First, the loss function right Find the partial derivative, ,parameter Update in the direction of negative gradient, , are the updated network parameters, are the network parameters before updating, is the learning rate, is the training data input to the network, is the weight of the training set, is the label corresponding to the training set, The number of samples input for each training is a random set of samples from the training set, which are updated according to the gradient descent rule after each training.

10. The landslide identification system based on attention mechanism and multimodal representation learning according to claim 9, characterized in that: Loss Function as follows: in, For the training set.

Images