A multi-modal dike landslide identification method based on direction perception enhancement

Abstract

The application discloses a kind of multi-modal embankment landslide identification methods based on direction perception enhancement, and unmanned aerial vehicle carries RGB camera and laser radar sensor synchronous collection dam area data and pre-process, obtain laser radar depth map and the depth map of RGB, and pixel by pixel label and divide training set and test set;Direction perception enhancement is constructed to the embankment landslide identification model, and training test environment is built, and open source dataset and self-built dam landslide identification dataset are pre-trained, and the final model is trained using the training set and test set;Two kinds of depth maps are inputted to carry out segmentation prediction, and the identification result is pre-processed and outputted in visual form.The application realizes the high-precision fusion of data by unmanned aerial vehicle carrying RGB camera and laser radar sensor synchronous collection dam area data, provides comprehensive and accurate information basis for landslide identification after pre-processing.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing and geological disaster monitoring, and particularly relates to a multi-modal embankment landslide identification method based on direction perception enhancement. BACKGROUND

[0002] As important water conservancy infrastructure, the safety of embankments is directly related to people's life and property safety and social stability. In recent years, affected by extreme climate and geological activities, embankment landslide disasters have shown a multi-incident trend. Traditional manual inspection methods have been difficult to meet the real-time monitoring needs. Although existing remote sensing monitoring technologies can achieve wide coverage, they still have significant limitations in identifying embankment landslides under complex topographic conditions. Optical images are easily disturbed by light conditions and vegetation coverage, and single data sources are difficult to penetrate the surface vegetation to obtain three-dimensional features of landslide bodies. Traditional machine learning methods rely on artificial feature design and lack robustness for subtle changes in landslide boundaries and morphological diversity. Existing deep learning models have problems such as fixed receptive field and insufficient direction sensitivity when processing multi-scale landslide features, resulting in a high rate of missed detection of small-scale landslides and linear cracks.

[0003] In the current technology, although the improved model based on U-Net has made certain progress in the field of semantic segmentation, it still has obvious shortcomings in dealing with the directionality structure and multi-scale features specific to embankment landslides. Conventional attention mechanisms are difficult to achieve efficient processing while maintaining accuracy due to high computational complexity. In addition, existing methods are mostly limited to single data sources and fail to fully utilize the complementary advantages between the three-dimensional geometric information provided by LiDAR point cloud data and the texture features of optical images. Especially in areas with dense vegetation, relying solely on RGB images can lead to blurred landslide boundaries, while point cloud data can penetrate vegetation but lack surface texture details. This bottleneck of multi-modal data fusion seriously hinders the improvement of landslide identification accuracy.

[0004] To address the above technical defects, the present application constructs a UNetFormer neural network model with direction perception enhancement, innovatively introduces a multi-scale sliding window axial attention mechanism, and realizes deep feature fusion of RGB images and LiDAR point cloud data. This technical solution effectively solves the core problems of traditional methods in embankment landslide identification, such as insufficient multi-scale feature capture, lack of direction sensitivity, and insufficient multi-source data fusion, providing a new technical means for embankment safety monitoring. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a multi-modal embankment landslide identification method based on direction perception enhancement, which aims to solve the problems in the background art.

[0006] To achieve the above object, the application provides the following technical scheme: a multi-modal embankment landslide identification method based on direction perception enhancement, comprising:

[0007] Step S1: using a UAV to carry an RGB camera and a laser radar sensor to synchronously collect embankment area data and perform preprocessing, to obtain a laser radar depth map and an RGB depth map;

[0008] Step S2: using a labeling tool to label and divide the laser radar depth map and the RGB depth map pixel by pixel into a training set and a test set;

[0009] Step S3: constructing a direction perception enhancement-based embankment landslide identification model;

[0010] Step S4: building a training and test environment for the direction perception enhancement-based embankment landslide identification model, training the direction perception enhancement-based embankment landslide identification model, and obtaining an embankment landslide identification model;

[0011] Step S5: using an open-source data set and a self-built embankment landslide identification data set to pre-train the embankment landslide identification model, obtaining a pre-trained model, training the pre-trained model using the training set and the test set of step S2, and obtaining a final embankment landslide identification model;

[0012] Step S6: inputting the laser radar depth map and the RGB depth map into the final embankment landslide identification model for segmentation prediction, and obtaining an embankment landslide identification result;

[0013] Step S7: preprocessing the embankment landslide identification result, obtaining a processed detection result, and outputting the processed detection result in a visualized form.

[0014] Further, the specific process of step S1 is as follows:

[0015] Using a UAV to carry an RGB camera and a laser radar sensor to synchronously collect embankment area data, to obtain RGB images and point cloud data in the embankment area data;

[0016] Using a statistical outlier removal algorithm and a radius filtering method to process the point cloud data in the embankment area data, and rasterizing the point cloud data into a laser radar depth map matching the resolution of the RGB images;

[0017] Performing radiation correction and geometric orthographic correction on the RGB images, to obtain corrected RGB images;

[0018] Dividing the corrected RGB images into image blocks according to a sliding window;

[0019] Inputting the image blocks into a Swin vision model for processing, and outputting the landslide confidence of the image blocks, when the landslide confidence exceeds a set threshold value 0.7, the image block is the RGB depth map.

[0020] Further, the final dam landslide identification model is composed of a dual-modal encoder and a direction perception enhanced decoder;

[0021] The dual-modal encoder is composed of an RGB encoder, a lidar depth map encoder, and a deep feature fusion module;

[0022] The RGB encoder is composed of a first convolutional block, a first residual module, a first shallow feature fusion module, a second residual module, a second shallow feature fusion module, a third residual module, a third shallow feature fusion module, and a fourth residual module;

[0023] The lidar depth map encoder is composed of a second convolutional block, a third convolutional block, a fifth residual module, a sixth residual module, a seventh residual module, and an eighth residual module;

[0024] The direction perception enhanced decoder is composed of a first upsampling, a first direction perception module, a second upsampling, two second direction perception modules, a third upsampling, two third direction perception modules, a fourth upsampling, two fourth direction perception modules, and a detection module;

[0025] The depth map of the RGB is input into the first convolutional block and the first residual module for processing to obtain the output of the first residual module;

[0026] The lidar depth map is sequentially input into the second convolutional block, the third convolutional block, and the fifth residual module for processing to obtain the output of the fifth residual module;

[0027] The output of the first residual module and the output of the fifth residual module are input into the first shallow feature fusion module for processing to obtain the output of the first shallow feature fusion module;

[0028] The output of the first shallow feature fusion module is input into the second residual module for processing to obtain the output of the second residual module;

[0029] The output of the fifth residual module is input into the sixth residual module for processing to obtain the output of the sixth residual module;

[0030] The output of the sixth residual module and the output of the second residual module are input into the second shallow feature fusion module for processing to obtain the output of the second shallow feature fusion module;

[0031] The output of the second shallow feature fusion module is input into the third residual module for processing to obtain the output of the third residual module;

[0032] The output of the sixth residual module is input into the seventh residual module for processing to obtain the output of the seventh residual module;

[0033] The output of the seventh residual module and the output of the third residual module are input into a third shallow feature fusion module for processing to obtain an output of the third shallow feature fusion module;

[0034] The output of the third shallow feature fusion module is input into a fourth residual module for processing to obtain an output of the fourth residual module;

[0035] The output of the seventh residual module is input into an eighth residual module for processing to obtain an output of the eighth residual module;

[0036] The output of the eighth residual module and the output of the fourth residual module are input into a deep feature fusion module for processing to obtain an output of the deep feature fusion module;

[0037] The output of the deep feature fusion module is input into a first up-sampling, and the output of the deep feature fusion module is input into a first direction perception module in a first direction for processing through the first up-sampling to obtain an output of the first direction perception module;

[0038] The output of the first direction perception module and the output of the third shallow feature fusion module are input into a second up-sampling, and the output of the first direction perception module and the output of the third shallow feature fusion module are input into two second direction perception modules for processing through the second up-sampling to obtain outputs of the two second direction perception modules;

[0039] The outputs of the two second direction perception modules and the output of the second shallow feature fusion module are input into a third up-sampling, and the outputs of the two second direction perception modules and the output of the second shallow feature fusion module are input into two third direction perception modules for processing through the third up-sampling to obtain outputs of the two third direction perception modules;

[0040] The outputs of the two third direction perception modules and the output of the first shallow feature fusion module are input into a fourth up-sampling, and the outputs of the two third direction perception modules and the output of the first shallow feature fusion module are input into two fourth direction perception modules for processing through the fourth up-sampling to obtain outputs of the two fourth direction perception modules;

[0041] The outputs of the two fourth direction perception modules are input into a detection module for processing to obtain an output of the detection module, which is a dam landslide recognition result.

[0042] Further, the specific process of obtaining the output of the first shallow feature fusion module is as follows:

[0043] The first shallow feature fusion module is composed of a first feature processing module, a first activation function, a second feature processing module, a second activation function, and a fourth convolution block;

[0044] The first shallow feature fusion module, the second shallow feature fusion module, and the third shallow feature fusion module have the same structure.

[0045] The output of the first residual module is sequentially input into the first feature processing module and the first activation function for processing to obtain the output of the first activation function;

[0046] The output of the fifth residual module is sequentially input into the second feature processing module and the second activation function for processing to obtain the output of the second activation function;

[0047] The output of the first activation function is multiplied by the output of the fifth residual module to obtain the output of the first multiplication;

[0048] The output of the second activation function is multiplied by the output of the first residual module to obtain the output of the second multiplication;

[0049] The output of the first multiplication and the output of the second multiplication are feature spliced to obtain the output of the first feature splicing, and the output of the first feature splicing is input into the first convolution block for processing to obtain the output of the first convolution block, which is the output of the first shallow feature fusion module.

[0050] Further, the output of the deep feature fusion module is obtained in the following specific process:

[0051] The deep feature fusion module is composed of a first average pooling block, a second average pooling block, a multi-layer perception block, a third activation function, and a fifth convolution block;

[0052] The output of the fourth residual module is input into the first average pooling block for processing to obtain the output of the first average pooling block;

[0053] The output of the eighth residual module is input into the second average pooling block for processing to obtain the output of the second average pooling block;

[0054] The output of the first average pooling block and the output of the second average pooling block are added and sequentially input into the multi-layer perception block and the third activation function for processing to obtain the output of the third activation function;

[0055] The output of the third activation function is multiplied by the output of the fourth residual module to obtain the output of the third multiplication;

[0056] The output of the third activation function is multiplied by the output of the eighth residual module to obtain the output of the fourth multiplication;

[0057] The output of the third multiplication and the output of the fourth multiplication are feature spliced to obtain the output of the second feature splicing, and the output of the second feature splicing is input into the fifth convolution block for processing to obtain the output of the fifth convolution block, which is the output of the deep feature fusion module.

[0058] Further, the output of a first direction perception module is obtained in the following specific process:

[0059] The first direction perception module is composed of first normalization, multi-scale sliding window axial attention mechanism, second normalization and full connection neural network;

[0060] The first direction perception module, the two second direction perception modules, the two third direction perception modules and the two fourth direction perception modules have the same structure.

[0061] The output of the deep feature fusion module is input into the first normalization for processing to obtain the output of the first normalization, and the output of the first normalization is input into the multi-scale sliding window axial attention mechanism for processing to obtain the output of the multi-scale sliding window axial attention mechanism.

[0062] The output of the multi-scale sliding window axial attention mechanism and the output of the deep feature fusion module are added to obtain the output of the first addition, and the output of the first addition is sequentially input into the second normalization and the full connection neural network for processing to obtain the output of the full connection neural network, and the output of the full connection neural network and the output of the first addition are added to obtain the output of the second addition, which is the output of the first direction perception module.

[0063] Further, the output of the multi-scale sliding window axial attention mechanism is obtained by the following process:

[0064] The multi-scale sliding window axial attention mechanism is composed of sliding window axial attention and a sixth convolutional block.

[0065] The output input into the first normalization is evenly divided into n preset sliding window numbers, and the first preset sliding window number is the first sub-feature, the second preset sliding window number is the second sub-feature, and the nth preset sliding window number is the nth sub-feature.

[0066] The nth sub-feature is input into the sliding window axial attention for processing to obtain the output of the nth sliding window axial attention, the output of the nth sliding window axial attention is spliced to obtain the spliced output, and the spliced output is processed by the sixth convolutional block to obtain the output of the sixth convolutional block, which is the output of the multi-scale sliding window axial attention mechanism.

[0067] Further, the output of the nth sliding window axial attention is obtained by the following process:

[0068] The sliding window axial attention is composed of horizontal axial attention, vertical axial attention and a fourth activation function.

[0069] ​The nth sub-feature is input into horizontal axial attention and vertical axial attention respectively for processing to obtain the output of the nth horizontal axial attention and the output of the nth vertical axial attention, the output of the nth horizontal axial attention and the output of the nth vertical axial attention are spliced to obtain the output of the nth third feature splicing, and the output of the nth third feature splicing is input into a fourth activation function for processing to obtain the output of the nth fourth activation function.

[0070] The output of the nth fourth activation function is multiplied by the output of the nth horizontal axial attention to obtain the output of the nth fifth multiplication.

[0071] The output of the nth fourth activation function is multiplied by the output of the nth vertical axial attention to obtain the output of the nth sixth multiplication.

[0072] The output of the nth fifth multiplication and the output of the nth sixth multiplication are added to obtain the output of the nth third addition, which is the output of the nth sliding window axial attention.

[0073] Further, the output of the nth horizontal axial attention is obtained by the following process:

[0074] The nth sub-feature is divided into the nth G row vector along the horizontal direction, the corresponding nth query vector, the nth key vector and the nth value vector are generated by applying a linear layer to the nth G row vector, the nth query vector and the nth key vector are multiplied to obtain the output of the nth seventh multiplication, the output of the nth seventh multiplication is processed by a Softmax function to obtain the nth attention graph, and the nth attention graph is weighted and summed with the nth value vector to obtain the output of the nth horizontal axial attention.

[0075] Compared with the prior art, the present application has the following advantages:

[0076] (1) The present application synchronously collects dam area data by mounting an RGB camera and a laser radar sensor on an unmanned aerial vehicle, realizes high-precision fusion of data after preprocessing, and provides a comprehensive and accurate information basis for landslide identification. The direction perception enhanced dam landslide identification model is constructed by introducing a multi-scale sliding window axial attention mechanism, which can effectively capture multi-scale features and directional structure information of dam landslides, solve the problems of fixed receptive field and insufficient direction sensitivity of traditional methods, significantly improve the detection capability of small-scale landslides and linear cracks, and reduce the missed detection rate.

[0077] (2) The application improves the generalization ability and pertinence of the direction perception enhanced dam landslide identification model by utilizing the advantages of the open source data set and the self-built dam landslide identification data set in the training process of the direction perception enhanced dam landslide identification model, determines the optimal super parameter combination through super parameter optimization methods such as enumeration method, ensures the stability and convergence of the direction perception enhanced dam landslide identification model training, and further improves the identification performance and efficiency. The final output dam landslide identification result includes detailed labeling and statistical data, and is presented in a visual form, which provides an intuitive basis for dam safety evaluation and early warning, helps to discover hidden dangers in time, and ensures the safety of people's life and property. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 The method flowchart of the application.

[0079] Figure 2 The dam landslide identification model structure diagram of the application.

[0080] Figure 3 The shallow feature fusion module structure diagram of the application.

[0081] Figure 4 The deep feature fusion module structure diagram of the application.

[0082] Figure 5 The direction perception module structure diagram of the application.

[0083] Figure 6 The multi-scale sliding window axial attention mechanism structure diagram of the application. DETAILED DESCRIPTION

[0084] As shown in Figure 1 The application provides a technical solution: a multi-modal dam landslide identification method based on direction perception enhancement, comprising:

[0085] Step S1: using a UAV to carry an RGB camera and a laser radar sensor to synchronously collect dam area data and perform preprocessing, to obtain a laser radar depth map and an RGB depth map;

[0086] Step S2: pixel-by-pixel labeling the laser radar depth map and the RGB depth map by using a labeling tool and dividing them into a training set and a test set;

[0087] Step S3: constructing a direction perception enhanced dam landslide identification model;

[0088] Step S4: building a training and test environment for the direction perception enhanced dam landslide identification model, training the direction perception enhanced dam landslide identification model, and obtaining a dam landslide identification model;

[0089] Step S5: Pre-training the dam landslide identification model using open source datasets and self-built dam landslide identification datasets, obtaining a pre-trained model, training the pre-trained model using the training set and test set of step S2, and obtaining the final dam landslide identification model;

[0090] Step S6: Input the laser radar depth map and the RGB depth map into the final dam landslide identification model for segmentation prediction to obtain the dam landslide identification result;

[0091] Step S7: Preprocessing the dam landslide identification result to obtain the processed detection result, and outputting the processed detection result in a visual form.

[0092] The specific process of step S1 is as follows:

[0093] An unmanned aerial vehicle is used to carry an RGB camera and a laser radar sensor to synchronously collect dam area data, obtaining RGB images and point cloud data in the dam area data, wherein the resolution of the RGB camera is not less than 0.1 m, and the point cloud density of the laser radar sensor (LiDAR) is not less than 50 points per square meter;

[0094] The point cloud data in the dam area data is processed using a statistical outlier removal algorithm and a radius filtering method, and rasterized into a laser radar depth map matching the resolution of the RGB image;

[0095] The RGB image is subjected to radiometric correction and geometric orthographic correction to obtain a corrected RGB image;

[0096] A unified spatial reference coordinate system for the laser radar depth map and the corrected RGB image is established, and the specific process is as follows:

[0097] At least 15 ground control points are arranged using the WGS-84 coordinate system to realize pixel-level registration of the laser radar depth map and the corrected RGB image;

[0098] Select an adaptive resolution according to the requirements of the laser radar depth map and the corrected RGB image.

[0099] The corrected RGB image is segmented into 256x256 pixel image blocks according to a sliding window, and the window step is ≤244 pixels;

[0100] The image block is input into a Swin-Transformer model for processing, and the landslide confidence of the image block is output. When the landslide confidence exceeds a set threshold value of 0.7, the image block is the RGB depth map.

[0101] The specific process of step S2 is as follows:

[0102] The laser radar depth map and the RGB depth map are labeled pixel by pixel by the labeling tool to generate three types of labels (landslide body, landslide boundary, and background), and the three types of labels need to meet the intersection over union IoU≥0.85 in the back-to-back verification of two persons;

[0103] Based on the three types of labels, the training set and the verification set are divided in a ratio of 8:2 to ensure the balance of the landslide area proportion;

[0104] The training set is subjected to random rotation (0-360°), flipping and brightness adjustment (±15%), and the landslide boundary area is subjected to targeted enhancement.

[0105] The specific process of step S4 is as follows:

[0106] The open source machine learning library Pytorch and the direction perception enhancement platform and the programming model CUDA are used to build the training and testing environment of the direction perception enhanced dam landslide recognition model (UNetFormer);

[0107] In the built training and testing environment, the enumeration method is used to test the performance of the direction perception enhanced dam landslide recognition model under different learning rates and batch sizes in the [0.0005, 0.005] interval with a step size of 0.0005 and using batch sizes of 4, 10 and 16 respectively. Through horizontal comparison of the detection results, 0.001 and 10 are finally selected as the optimal hyperparameters of the learning rate and the small batch size.

[0108] Based on the optimal hyperparameters, different direction perception enhanced dam landslide recognition models are selected, and the recognition performance, training and prediction efficiency of different direction perception enhanced dam landslide recognition models are compared, and finally the configuration from low to high [1, 2, 2, 2] is used as the dam landslide recognition model.

[0109] The specific process of step S5 is as follows:

[0110] The open source dataset (ISPRS Vaihingen) and the self-built dam landslide recognition dataset (ISPRS Potsdam) are used to pretrain the dam landslide recognition model to obtain a pretraining model.

[0111] Wherein, the cross-entropy loss function Cross-entropy Loss is used as the loss function and the Adam optimizer is used, and the dam landslide recognition model is further optimized by iterating the training epoch=100 times.

[0112] As shown in Figure 2 , the final dam landslide recognition model is composed of a dual-modal encoder and a direction perception enhanced decoder;

[0113] The dual-modal encoder consists of an RGB encoder, a LiDAR depth map encoder, and a deep feature fusion module;

[0114] The RGB encoder consists of the first convolutional block ( It consists of (Pre_conv), the first residual module (ResBlock), the first shallow feature fusion module (SFF), the second residual module, the second shallow feature fusion module, the third residual module, the third shallow feature fusion module, and the fourth residual module;

[0115] The lidar depth map encoder consists of a second convolutional block ( ), third volume block ( It consists of the fifth residual module, the sixth residual module, the seventh residual module, and the eighth residual module;

[0116] The direction-aware enhanced decoder consists of a first upsampling module, a first direction-aware module (DA Block), a second upsampling module, two second direction-aware modules, a third upsampling module, two third direction-aware modules, a fourth upsampling module, two fourth direction-aware modules, and a detection module.

[0117] The RGB depth map is input into the first convolutional block and the first residual module for processing, and the output of the first residual module is obtained.

[0118] The LiDAR depth map is sequentially input into the second convolutional block, the third convolutional block, and the fifth residual module for processing, and the output of the fifth residual module is obtained.

[0119] The outputs of the first residual module and the fifth residual module are input into the first shallow feature fusion module for processing to obtain the output of the first shallow feature fusion module.

[0120] The output of the first shallow feature fusion module is input into the second residual module for processing to obtain the output of the second residual module;

[0121] The output of the fifth residual module is input into the sixth residual module for processing to obtain the output of the sixth residual module;

[0122] The output of the sixth residual module and the output of the second residual module are input into the second shallow feature fusion module for processing to obtain the output of the second shallow feature fusion module.

[0123] The output of the second shallow feature fusion module is input into the third residual module for processing to obtain the output of the third residual module;

[0124] The output of the sixth residual module is input into the seventh residual module for processing to obtain the output of the seventh residual module;

[0125] The outputs of the seventh residual module and the third residual module are input into the third shallow feature fusion module for processing to obtain the output of the third shallow feature fusion module.

[0126] The output of the third shallow feature fusion module is input into the fourth residual module for processing to obtain the output of the fourth residual module;

[0127] The output of the seventh residual module is input into the eighth residual module for processing to obtain the output of the eighth residual module;

[0128] The outputs of the eighth residual module and the fourth residual module are input into the deep feature fusion module for processing to obtain the output of the deep feature fusion module.

[0129] The output of the deep feature fusion module is input into the first upsampling, and the output of the deep feature fusion module is input into a first direction perception module for processing through the first upsampling to obtain the output of the first direction perception module;

[0130] The output of a first direction perception module and the output of a third shallow feature fusion module are input into a second upsampling. The output of a first direction perception module and the output of a third shallow feature fusion module are then input into two second direction perception modules for processing through the second upsampling to obtain the outputs of the two second direction perception modules.

[0131] The outputs of the two second-direction perception modules and the output of the second shallow feature fusion module are input into the third upsampling. The outputs of the two second-direction perception modules and the output of the second shallow feature fusion module are then input into the two third-direction perception modules for processing through the third upsampling to obtain the outputs of the two third-direction perception modules.

[0132] The outputs of the two third-direction perception modules and the output of the first shallow feature fusion module are input into the fourth upsampling. The outputs of the two third-direction perception modules and the output of the first shallow feature fusion module are then input into the two fourth-direction perception modules for processing through the fourth upsampling to obtain the outputs of the two fourth-direction perception modules.

[0133] The outputs of the two fourth-direction sensing modules are input into the detection module for processing, and the output of the detection module is obtained, which is the dam landslide identification result.

[0134] like Figure 3 As shown, the specific process of obtaining the output of the first shallow feature fusion module is as follows:

[0135] The first shallow feature fusion module consists of a first feature processing module, a first activation function, a second feature processing module, a second activation function, and a fourth convolutional block. )composition;

[0136] The first shallow feature fusion module, the second shallow feature fusion module, and the third shallow feature fusion module are of the same structure.

[0137] The output of the first residual module is sequentially input into the first feature processing module and the first activation function for processing to obtain the output of the first activation function.

[0138] The output of the fifth residual module is sequentially input into the second feature processing module and the second activation function for processing to obtain the output of the second activation function.

[0139] The output of the first activation function is multiplied by the output of the fifth residual module to obtain the output of the first multiplication.

[0140] The output of the second activation function is multiplied by the output of the first residual module to obtain the output of the second multiplication.

[0141] The output of the first multiplication and the output of the second multiplication are feature spliced to obtain the output of the first feature splicing, and the output of the first feature splicing is input into the first convolution block for processing to obtain the output of the first convolution block, which is the output of the first shallow feature fusion module.

[0142] As shown in Figure 4 , the output of the deep feature fusion module is obtained in the following specific process:

[0143] The deep feature fusion module is composed of a first average pooling block, a second average pooling block, a multi-layer perception block, a third activation function, and a fifth convolution block.

[0144] The output of the fourth residual module is input into the first average pooling block for processing to obtain the output of the first average pooling block.

[0145] The output of the eighth residual module is input into the second average pooling block for processing to obtain the output of the second average pooling block.

[0146] The output of the first average pooling block and the output of the second average pooling block are added and sequentially input into the multi-layer perception block and the third activation function for processing to obtain the output of the third activation function.

[0147] The output of the third activation function is multiplied by the output of the fourth residual module to obtain the output of the third multiplication.

[0148] The output of the third activation function is multiplied by the output of the eighth residual module to obtain the output of the fourth multiplication.

[0149] The output of the third multiplication and the output of the fourth multiplication are feature spliced to obtain the output of the second feature splicing, and the output of the second feature splicing is input into the fifth convolution block for processing to obtain the output of the fifth convolution block, which is the output of the deep feature fusion module.​

[0150] As shown in Figure 5 , wherein the specific process of obtaining the output of the first direction perception module is:

[0151] The first direction perception module is composed of first normalization, multi-scale sliding window axial attention mechanism, second normalization and fully connected neural network.

[0152] The first direction perception module, the two second direction perception modules, the two third direction perception modules and the two fourth direction perception modules have the same structure.

[0153] The output of the deep feature fusion module is input into the first normalization for processing to obtain the output of the first normalization, and the output of the first normalization is input into the multi-scale sliding window axial attention mechanism for processing to obtain the output of the multi-scale sliding window axial attention mechanism.

[0154] The output of the multi-scale sliding window axial attention mechanism and the output of the deep feature fusion module are added to obtain the output of the first addition; the output of the first addition is sequentially input into the second normalization and the fully connected neural network for processing to obtain the output of the fully connected neural network, and the output of the fully connected neural network and the output of the first addition are added to obtain the output of the second addition, which is the output of the first direction perception module.

[0155] As shown in Figure 6 , wherein the specific process of obtaining the output of the multi-scale sliding window axial attention mechanism is:

[0156] The multi-scale sliding window axial attention mechanism is composed of sliding window axial attention and a sixth convolutional block .

[0157] The preset number of sliding windows is n, and the output input into the first normalization is evenly divided into ; 1 is the first sub-feature; 2 is the second sub-feature; and n is the nth sub-feature.

[0158] The nth sub-feature is input into the sliding window axial attention for processing to obtain the output of the nth sliding window axial attention, the output of the nth sliding window axial attention is spliced to obtain the spliced output, and the spliced output is processed by the sixth convolutional block to obtain the output of the sixth convolutional block, which is the output of the multi-scale sliding window axial attention mechanism.

[0159] The specific process of obtaining the output of the nth sliding window axial attention is:

[0160] The sliding window axial attention is composed of horizontal axial attention, vertical axial attention and a fourth activation function.

[0161] The nth sub-feature is input into horizontal axial attention and vertical axial attention respectively for processing to obtain the output of the nth horizontal axial attention and the output of the nth vertical axial attention, the output of the nth horizontal axial attention and the output of the nth vertical axial attention are spliced to obtain the output of the nth third feature splicing, and the output of the nth third feature splicing is input into a fourth activation function for processing to obtain the output of the nth fourth activation function;

[0162] The output of the nth fourth activation function is multiplied by the output of the nth horizontal axial attention to obtain the output of the nth fifth multiplication;

[0163] The output of the nth fourth activation function is multiplied by the output of the nth vertical axial attention to obtain the output of the nth sixth multiplication;

[0164] The output of the nth fifth multiplication and the output of the nth sixth multiplication are added to obtain the output of the nth third addition, which is the output of the nth sliding window axial attention.

[0165] The output of the nth horizontal axial attention is obtained by the following specific process:

[0166] The nth sub-feature is divided into an nth G row vector along the horizontal direction, a linear layer is applied to the nth G row vector to generate a corresponding nth query vector, an nth key vector and an nth value vector, the nth query vector and the nth key vector are multiplied to obtain the output of the nth seventh multiplication, and the output of the nth seventh multiplication is processed by a Softmax function to obtain an nth attention map, the nth attention map is weighted and summed with the nth value vector to obtain the output of the nth horizontal axial attention.

[0167] The specific process of step S7 is as follows:

[0168] The dam landslide recognition result is processed to obtain a processed detection result, the processing including removing isolated noise regions with an area less than 50 pixels (preferably in the range of 30-100 pixels) in the dam landslide recognition result through connected component analysis; filling the internal cavities of the dam landslide recognition result based on an adaptive threshold (e.g., when the hollow area is less than 5% of the total landslide area) to preserve the true topographic features; applying Gaussian filtering to optimize the sawtooth phenomenon of the dam landslide recognition result landslide boundary to improve the credibility of the detection result;

[0169] The processed detection result is output in a visual form to generate a report containing landslide region labeling, boundary information and related statistical data, providing intuitive basis for dam safety assessment and early warning.

[0170] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A multimodal dam landslide identification method based on direction-aware enhancement, characterized in that, include: Step S1: Use a drone equipped with an RGB camera and a LiDAR sensor to simultaneously collect data of the dam area and perform preprocessing to obtain a LiDAR depth map and an RGB depth map; Step S2: Label the LiDAR depth map and RGB depth map pixel by pixel using the annotation tool and divide them into training set and test set; Step S3: Construct a direction-aware enhanced dam landslide identification model; Step S4: Build a training and testing environment for the orientation-aware enhanced dam landslide recognition model, train the orientation-aware enhanced dam landslide recognition model, and obtain the dam landslide recognition model. Step S5: Use open-source datasets and self-built dam landslide recognition datasets to pre-train the dam landslide recognition model to obtain a pre-trained model. Use the training set and test set from step S2 to train the pre-trained model to obtain the final dam landslide recognition model. Step S6: Input the lidar depth map and RGB depth map into the final dam landslide identification model for segmentation prediction to obtain the dam landslide identification result; Step S7: Preprocess the landslide identification results of the embankment to obtain the processed detection results, and output the processed detection results in a visual form; The final landslide identification model consists of a dual-modal encoder and a direction-aware enhanced decoder; the dual-modal encoder consists of an RGB encoder, a LiDAR depth map encoder, and a deep feature fusion module. The direction-aware enhanced decoder consists of a first upsampling module, a first direction-aware module, a second upsampling module, two second direction-aware modules, a third upsampling module, two third direction-aware modules, a fourth upsampling module, two fourth direction-aware modules, and a detection module. The first direction-aware module consists of a first normalization, a multi-scale sliding window axial attention mechanism, a second normalization, and a fully connected neural network. The first, second, third, and fourth direction-aware modules have the same structure. The output of the multi-scale sliding window axial attention mechanism includes: a preset number of sliding windows of n, uniformly dividing the first normalized output into n sub-features; inputting the n sub-features into the sliding window axial attention for processing to obtain the output of n sliding window axial attention; concatenating the outputs of n sliding window axial attention; and processing the concatenated output through the sixth convolutional block to obtain the output of the sixth convolutional block. The process of obtaining the output of the nth sliding window axial attention includes: performing horizontal and vertical axial attention processing on the nth sub-feature, concatenating the outputs of the horizontal and vertical axial attention and inputting them into the fourth activation function to obtain the output of the nth fourth activation function; multiplying the output of the nth fourth activation function with the outputs of the horizontal and vertical axial attention respectively, and adding the two multiplication results as the output of the nth sliding window axial attention.

2. The method for identifying multimodal dam landslides based on direction-aware enhancement according to claim 1, characterized in that: The specific process of step S1 is as follows: A drone equipped with an RGB camera and a LiDAR sensor was used to simultaneously collect data on the dam area, resulting in RGB images and point cloud data of the dam area. The point cloud data in the dam area data was processed using a statistical outlier removal algorithm and a radius filtering method, and then rasterized into a LiDAR depth map that matches the resolution of the RGB image. Radiometric and geometric orthorectification are performed on the RGB image to obtain the corrected RGB image; The corrected RGB image is divided into image blocks using a sliding window; The image patch is input into the Swin-Transformer for processing, and the landslide confidence of the image patch is output. When the landslide confidence exceeds the set threshold of 0.7, the image patch is the RGB depth map.

3. The method for identifying multimodal dam landslides based on direction-aware enhancement according to claim 2, characterized in that: The RGB encoder consists of a first convolutional block, a first residual module, a first shallow feature fusion module, a second residual module, a second shallow feature fusion module, a third residual module, a third shallow feature fusion module, and a fourth residual module. The lidar depth map encoder consists of a second convolutional block, a third convolutional block, a fifth residual module, a sixth residual module, a seventh residual module, and an eighth residual module. The RGB depth map is input into the first convolutional block and the first residual module for processing, and the output of the first residual module is obtained. The LiDAR depth map is sequentially input into the second convolutional block, the third convolutional block, and the fifth residual module for processing, and the output of the fifth residual module is obtained. The outputs of the first residual module and the fifth residual module are input into the first shallow feature fusion module for processing to obtain the output of the first shallow feature fusion module. The output of the first shallow feature fusion module is input into the second residual module for processing to obtain the output of the second residual module; The output of the fifth residual module is input into the sixth residual module for processing to obtain the output of the sixth residual module; The output of the sixth residual module and the output of the second residual module are input into the second shallow feature fusion module for processing to obtain the output of the second shallow feature fusion module. The output of the second shallow feature fusion module is input into the third residual module for processing to obtain the output of the third residual module; The output of the sixth residual module is input into the seventh residual module for processing to obtain the output of the seventh residual module; The outputs of the seventh residual module and the third residual module are input into the third shallow feature fusion module for processing to obtain the output of the third shallow feature fusion module. The output of the third shallow feature fusion module is input into the fourth residual module for processing to obtain the output of the fourth residual module; The output of the seventh residual module is input into the eighth residual module for processing to obtain the output of the eighth residual module; The outputs of the eighth residual module and the fourth residual module are input into the deep feature fusion module for processing to obtain the output of the deep feature fusion module. The output of the deep feature fusion module is input into the first upsampling, and the output of the deep feature fusion module is input into a first direction perception module for processing through the first upsampling to obtain the output of the first direction perception module; The output of a first direction perception module and the output of a third shallow feature fusion module are input into a second upsampling. The output of a first direction perception module and the output of a third shallow feature fusion module are then input into two second direction perception modules for processing through the second upsampling to obtain the outputs of the two second direction perception modules. The outputs of the two second-direction perception modules and the output of the second shallow feature fusion module are input into the third upsampling. The outputs of the two second-direction perception modules and the output of the second shallow feature fusion module are then input into the two third-direction perception modules for processing through the third upsampling to obtain the outputs of the two third-direction perception modules. The outputs of the two third-direction perception modules and the output of the first shallow feature fusion module are input into the fourth upsampling. The outputs of the two third-direction perception modules and the output of the first shallow feature fusion module are then input into the two fourth-direction perception modules for processing through the fourth upsampling to obtain the outputs of the two fourth-direction perception modules. The outputs of the two fourth-direction sensing modules are input into the detection module for processing, and the output of the detection module is obtained, which is the dam landslide identification result.

4. The method for identifying multimodal dam landslides based on direction-aware enhancement according to claim 3, characterized in that: The specific process of obtaining the output of the first shallow feature fusion module is as follows: The first shallow feature fusion module consists of a first feature processing module, a first activation function, a second feature processing module, a second activation function, and a fourth convolutional block; Among them, the first shallow feature fusion module, the second shallow feature fusion module, and the third shallow feature fusion module have the same structure; The output of the first residual module is sequentially input into the first feature processing module and the first activation function for processing to obtain the output of the first activation function. The output of the fifth residual module is sequentially input into the second feature processing module and the second activation function for processing, to obtain the output of the second activation function. Multiply the output of the first activation function by the output of the fifth residual module to obtain the output of the first multiplication; The output of the second activation function is multiplied by the output of the first residual module to obtain the output of the second multiplication. The outputs of the first multiplication and the second multiplication are concatenated to obtain the output of the first feature concatenation. The output of the first feature concatenation is then input into the first convolutional block for processing to obtain the output of the first convolutional block, which is the output of the first shallow feature fusion module.

5. The method for identifying multimodal dam landslides based on direction-aware enhancement according to claim 4, characterized in that: The specific process of obtaining the output of the deep feature fusion module is as follows: The deep feature fusion module consists of a first average pooling block, a second average pooling block, a multilayer perceptron block, a third activation function, and a fifth convolutional block. The output of the fourth residual module is input into the first average pooling block for processing to obtain the output of the first average pooling block; The output of the eighth residual module is processed in the second average pooling block to obtain the output of the second average pooling block; The outputs of the first average pooling block and the second average pooling block are added together and then fed into the multilayer perceptron block and the third activation function for processing to obtain the output of the third activation function. The output of the third activation function is multiplied by the output of the fourth residual module to obtain the output of the third multiplication; The output of the third activation function is multiplied by the output of the eighth residual module to obtain the output of the fourth multiplication; The outputs of the third and fourth multiplications are concatenated to obtain the output of the second feature concatenation. The output of the second feature concatenation is then input into the fifth convolutional block for processing, resulting in the output of the fifth convolutional block, which is the output of the deep feature fusion module.

6. The method for identifying multimodal dam landslides based on direction-aware enhancement according to claim 5, characterized in that: The specific process of obtaining the output of a first-direction sensing module is as follows: The output of the deep feature fusion module is processed by the first normalization to obtain the first normalized output. The first normalized output is then processed by the multi-scale sliding window axial attention mechanism to obtain the output of the multi-scale sliding window axial attention mechanism. The output of the multi-scale sliding window axial attention mechanism and the output of the deep feature fusion module are added together to obtain the first summed output. The first summed output is then sequentially input into the second normalization and fully connected neural network for processing to obtain the output of the fully connected neural network. The output of the fully connected neural network is then added together with the first summed output to obtain the second summed output, which is the output of a first direction perception module.

7. The method for identifying multimodal dam landslides based on direction-aware enhancement according to claim 6, characterized in that: The specific process for obtaining the output of the nth horizontal axis attention is as follows: The nth sub-feature is divided into nth G-row vectors along the horizontal direction. A linear layer is applied to the nth G-row vector to generate the corresponding nth query vector, nth key vector, and nth value vector. The nth query vector and the nth key vector are multiplied to obtain the nth seventh multiplication output. The nth seventh multiplication output is processed by the Softmax function to obtain the nth attention map. The nth attention map and the nth value vector are weighted and summed to obtain the nth horizontal axis attention output.

Images