Landslide classification method and system based on visual language model and cross attention mechanism

Abstract

The invention provides a landslide classification method and system based on a visual language model and a cross attention mechanism. The method and the system specifically comprise the following steps: data preprocessing: carrying out Canny edge detection on an RGB image, and calculating terrain attributes such as a gradient and a slope direction for a DEM (Digital Elevation Model); feature extraction: capturing local features by adopting a reflection filling convolution layer and multi-scale residual connection; a visual language model is introduced, wherein semantic enhancement features are extracted through image-text alignment by means of the visual language model; cross self-attention fusion: capturing a global context through self-attention, and focusing heterogenous data complementary information by cross attention; and classifying and outputting: outputting a result by using global average pooling and a linear classifier. Through the visual language model and the cross self-attention mechanism, the landslide recognition capability under the complex terrain is effectively improved, an efficient and reliable technical means is provided for geological disaster monitoring, and the method can be widely applied to the fields of landslide recognition, risk assessment and the like.

Description

Technical Field

[0001] This invention proposes a landslide classification method and system based on a visual language model and a cross-attention mechanism, which relates to the fields of computer vision and geological disaster monitoring technology. Background Technology

[0002] Landslides, a common geological hazard, seriously threaten the lives and property of residents in mountainous areas. Traditional landslide classification methods mainly rely on field surveys and manual analysis, which are not only time-consuming and labor-intensive but also have limited coverage, making them unsuitable for real-time monitoring. With the continuous development of remote sensing technology, landslide identification using high-resolution satellite imagery and digital elevation models (DEMs) has become a research hotspot. Early methods, such as support vector machines (SVMs) and random forests (RFs), while capable of extracting shallow features like terrain and texture, suffer from insufficient feature fusion capabilities when processing multi-source data under complex geological conditions.

[0003] In recent years, deep learning technology has made significant progress in the field of remote sensing image analysis. Convolutional Neural Networks (CNNs) and Transformers have been widely used in landslide detection, improving recognition accuracy by automatically extracting deep features. However, most existing methods rely solely on a single data type (such as RGB images or DEMs), neglecting the complementarity between multiple data sources. For example, RGB images provide rich information on land cover but are insensitive to topographic changes; while DEMs can reflect topographic undulations, they lack detailed texture features. Furthermore, traditional feature fusion methods, such as simple channel stitching or element-wise addition, cannot effectively capture the semantic relationships between different data types, limiting the model's generalization ability in complex terrains.

[0004] Currently, a key challenge in multi-source data fusion lies in designing an efficient fusion mechanism to fully leverage the complementary information from different data sources. Existing attention mechanisms, such as self-attention, while capable of capturing long-distance dependencies, lack explicit modeling of interactions between different features when processing multimodal data. Therefore, constructing a landslide classification model that can effectively fuse multiple types of remote sensing datasets and enhance feature representation capabilities has become a pressing technical challenge in this field. Summary of the Invention

[0005] With the development of multimodal learning, the ability of visual language models (such as CLIP and BLIP) to combine image and text data offers new possibilities for landslide identification. These models, through pre-training on massive image-text pairs, learn cross-modal semantic alignment, enabling them to extract visual features from RGB images and match them with text describing landslide characteristics (such as "exposed slope" or "vegetation broken"), thereby improving the understanding of complex scenes. The challenge lies in integrating visual language models with multi-source remote sensing data, utilizing textual cues to guide the semantic extraction of landslide features, and enhancing the interactive modeling of heterogeneous data through attention mechanisms to adapt to the landslide classification needs in complex terrain. This technological direction offers significant potential for improving the generalization ability of models and the real-time performance of disaster monitoring.

[0006] In view of this, in order to fill the gaps and deficiencies in the existing technology, this invention proposes a landslide classification method and system based on a visual language model and a cross-attention mechanism, which can realize the interactive modeling of heterogeneous data through the attention mechanism to adapt to the landslide classification needs under complex terrain.

[0007] This invention proposes a landslide classification method and system based on a visual language model and a cross-attention mechanism, including the following:

[0008] This invention proposes a landslide classification method based on a visual language model and a cross-attention mechanism, characterized by the following:

[0009] Step S1: Construct multi-type remote sensing datasets, including the acquisition of RGB images and DEM data;

[0010] Step S2: The RGB image is processed using the Canny edge detection algorithm to generate an edge feature map to enhance the landslide boundary and texture features; the topographic attributes of the DEM data are calculated and the topographic structure representation is refined; then, data cleaning is performed on the multi-type remote sensing datasets to remove outliers and missing values, and standardization is performed to eliminate the dimensional differences between the multi-type remote sensing data.

[0011] Step S3: Use a dual-branch convolutional network to process RGB images and DEM data respectively; including initial feature extraction through the Stem module, followed by the use of the encoder module, combined with reflection-filled convolutional layers and multi-scale residual connections, to capture local features and preserve boundary information, and generate standardized feature maps;

[0012] Step S4: Introduce a visual language model, combining the visual features of RGB images with predefined text prompts, and enhance multimodal semantic understanding through a visual unit attention mechanism; the visual language model aligns feature maps with text embeddings, captures the semantic context of landslide features in the image, and provides richer multimodal representations for subsequent cross-self-attention fusion;

[0013] Step S5: Capture global contextual information from RGB images, DEM data, and visual language model outputs using a self-attention module, and fuse heterogeneous features using a cross-self-attention module to achieve deep semantic integration of multiple types of remote sensing datasets;

[0014] Step S6: Input the fused features into a global average pooling layer for spatial dimension compression, and then output the landslide classification results through a linear classifier; evaluate the model performance in multiple dimensions to verify its ability to identify landslide samples and its generalization.

[0015] Further, step S1 includes the following:

[0016] Step S11: The data sources for the multi-type remote sensing datasets include satellite remote sensing platforms, UAV remote sensing equipment, and geographic information monitoring systems; among them, RGB images are used to capture surface texture and color information, DEM data are obtained through geographic information systems, and derived terrain attributes such as slope, aspect, and curvature are calculated to deeply explore terrain structure features.

[0017] Further, step S2 includes the following:

[0018] Step S2, which calculates the terrain attributes of the DEM data and refines the terrain structure representation, includes the following:

[0019] The slope α is calculated using the finite difference formula:

[0020]

[0021] Where f x f y The slope change rate in the x and y directions of the slope to be calculated in the DEM data is used, and then the surface inclination is calculated point by point to quantify the steepness of the terrain.

[0022] Calculate slope aspect β:

[0023]

[0024] Use slope aspect to determine the orientation angle of terrain cells in DEM data;

[0025] Calculate curvature r:

[0026]

[0027] in, and These are the second partial derivatives along the x-axis and y-axis, respectively. The mixed second partial derivative reflects the geometric characteristics of the Earth's surface.

[0028] Further, step S3 includes the following:

[0029] Step S31: Use a dual-branch convolutional network to process the RGB image and DEM data respectively; the dual-branch convolutional network includes the following:

[0030] Reflection-filled convolutional layers: Employ a mirror-filling strategy to preserve boundary information;

[0031] Multi-scale feature fusion: fusing feature maps from different levels through skip connections;

[0032] Channel standardization: Maps bi-branch features uniformly to a 16-channel dimension;

[0033] Step S32: Perform hierarchical feature learning on the encoder module with a 3-layer residual structure that has processed RGB image and DEM data respectively through a dual-branch convolutional network. Each encoder module contains a convolutional layer and a ReLU activation function, and gradually extracts multi-level features from edge texture to terrain structure; thus obtaining the dual features of RGB image and DEM data.

[0034] Step S33: The dual features of the RGB image and DEM data are uniformly mapped to a 16-channel dimension through channel normalization.

[0035] Further, step S4 includes the following:

[0036] The visual language model in step S4 includes the following:

[0037] Visual encoding unit: A Transformer-based visual encoder is used to process RGB images, and the initial feature dimensions are related to the input image resolution;

[0038] Text encoding unit: Encodes predefined landslide-related text prompts using a pre-trained language model, generates semantic embeddings, aligns with visual features, and unifies the dimensions to preset values;

[0039] Multimodal alignment module: Through the visual unit attention mechanism, it achieves cross-alignment of visual features and text embeddings, calculates the similarity of image-text pairs, enhances the semantic representation of landslide features, and outputs a fused multimodal feature map.

[0040] Further, step S5 includes the following:

[0041] The fork self-attention module in step S5 includes the following:

[0042] Self-attention units handle long-distance dependencies of single-modal features;

[0043] Cross-attention units calculate complementary weights between heterogeneous features;

[0044] The feature fusion unit achieves multimodal feature fusion through element-level addition.

[0045] The deep semantic integration of multiple types of remote sensing datasets mentioned in step S5 includes RGB image + DEM data, RGB image + Canny, and RGB + terrain attribute combination.

[0046] Further, step S6 includes the following:

[0047] Step S61: Verify the model's ability to identify landslide samples, including the following:

[0048] The accuracy, recall, and F1 score were used to verify the model's ability to identify landslide samples.

[0049] Step S62: Deploy the trained landslide classification model to a real-world application scenario to build an intelligent monitoring system.

[0050] This invention proposes a landslide classification system based on a visual language model and a cross-attention mechanism, used to execute a landslide classification method based on a visual language model and a cross-attention mechanism as described in any one of the claims of this invention. The system is characterized by including a landslide classification model, wherein the landslide classification model comprises the following:

[0051] Data preprocessing module: performs edge detection on RGB images and calculates terrain attributes on DEM data;

[0052] Feature extraction module: A dual-branch convolutional network is used to process RGB and DEM data separately;

[0053] Cross-self-attention module: Enables deep fusion of heterogeneous features;

[0054] Visual language model module: performs interpretive feature extraction based on text and images;

[0055] Classification and evaluation module: Outputs classification results and performs performance evaluation.

[0056] According to a second aspect of the present invention, a landslide classification system based on a visual language model and a cross-attention mechanism includes an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements a landslide classification method based on a visual language model and a cross-attention mechanism as described in any one of the present invention.

[0057] According to a third aspect of the present invention, a landslide classification system based on a visual language model and a cross-attention mechanism includes a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements any of the landslide classification methods based on a visual language model and a cross-attention mechanism of the present invention.

[0058] The present invention has the following advantages:

[0059] (1) Enhancing the complementarity of multi-source data: By fusing RGB images and DEM data, surface texture information and topographic structure features are fully integrated, overcoming the limitations of single data types in adapting to complex geological environments. Experiments show that combining multiple types of remote sensing datasets significantly improves classification accuracy compared to single data, effectively enhancing the model's ability to identify landslide boundaries and topographic changes.

[0060] (2) Improved feature fusion efficiency: The attention mechanism explicitly models the semantic relationships between heterogeneous data and focuses on non-redundant features through a negative similarity calculation strategy. Compared with the traditional channel concatenation method, it reduces the interference of redundant information and enhances the discriminativeness of feature representation. In complex scenarios, the cross-attention module effectively improves the model's efficiency in utilizing multimodal features.

[0061] (3) Enhanced model robustness: The reflection-filled convolutional layer and multi-scale residual connection design effectively preserve image boundary information and alleviate overfitting. In tests under different geological conditions, the model exhibits stronger stability, is suitable for diverse landslide scenarios, and significantly improves generalization ability.

[0062] (4) Enhance semantic understanding: The visual language model combines RGB images with landslide-related text prompts (such as “exposed slope” or “vegetation broken”) to achieve multimodal alignment of images and text, thereby improving the semantic recognition of landslide features in complex terrain.

[0063] (5) Optimized engineering application effects: Through the design of a lightweight classifier, the model inference speed is significantly improved, supporting real-time monitoring needs. The visualization heat map function provides an intuitive basis for geological hazard assessment, which can greatly reduce the cost of manual annotation in actual engineering and has higher practical value. Attached Figure Description

[0064] Figure 1 This is a flowchart of the steps of the system of the present invention.

[0065] Figure 2 This is a schematic diagram of the system of the present invention.

[0066] Figure 3 This is a pseudocode diagram illustrating the computation using the dual-stream branch network of this invention.

[0067] Figure 4 This is a pseudocode diagram of the cross-attention mechanism algorithm of the present invention.

[0068] Figure 5 This is a schematic diagram of the prompt text setting for the visual language model of the present invention. Detailed Implementation

[0069] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0070] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0071] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0072] like Figure 1 As shown, this invention proposes a landslide classification method based on a visual language model and a cross-attention mechanism, characterized by the following:

[0073] Step S1: Construct multi-type remote sensing datasets, including the acquisition of RGB images and DEM data;

[0074] Step S2: The RGB image is processed using the Canny edge detection algorithm to generate an edge feature map to enhance the landslide boundary and texture features; the topographic attributes of the DEM data are calculated and the topographic structure representation is refined; then, data cleaning is performed on the multi-type remote sensing datasets to remove outliers and missing values, and standardization is performed to eliminate the dimensional differences between the multi-type remote sensing data.

[0075] Step S3: Use a dual-branch convolutional network to process RGB images and DEM data respectively; including initial feature extraction through the Stem module, followed by the use of the encoder module, combined with reflection-filled convolutional layers and multi-scale residual connections, to capture local features and preserve boundary information, and generate standardized feature maps;

[0076] Step S4: Introduce a visual language model, combining the visual features of RGB images with predefined text prompts, and enhance multimodal semantic understanding through a visual unit attention mechanism; the visual language model aligns feature maps with text embeddings, captures the semantic context of landslide features in the image, and provides richer multimodal representations for subsequent cross-self-attention fusion;

[0077] Step S5: Capture global contextual information from RGB images, DEM data, and visual language model outputs using a self-attention module, and fuse heterogeneous features using a cross-self-attention module to achieve deep semantic integration of multiple types of remote sensing datasets;

[0078] Step S6: Input the fused features into a global average pooling layer for spatial dimension compression, and then output the landslide classification results through a linear classifier; evaluate the model performance in multiple dimensions to verify its ability to identify landslide samples and its generalization.

[0079] In one embodiment of the present invention, step S1 includes the following:

[0080] Step S11: The data sources for the multi-type remote sensing datasets include satellite remote sensing platforms, UAV remote sensing equipment, and geographic information monitoring systems; among them, RGB images are used to capture surface texture and color information, DEM data are obtained through geographic information systems, and derived terrain attributes such as slope, aspect, and curvature are calculated to deeply explore terrain structure features.

[0081] Furthermore, in one embodiment of the present invention, the multi-type remote sensing dataset not only includes the acquisition of RGB images and DEM data, but also includes data cleaning of Canny indicators and DEM-derived data.

[0082] Furthermore, in one embodiment of the present invention, DEM data represents digital elevation model data.

[0083] Furthermore, in one embodiment of the present invention, step S1 further includes the following:

[0084] First, various types of remote sensing datasets required for landslide classification were collected, including RGB images acquired by remote sensing satellites to capture surface texture and color information; Digital Elevation Model (DEM) data were obtained using a Geographic Information System (GIS) to reflect topographic relief characteristics; and topographic attribute data such as slope, aspect, and curvature derived from the DEM were extracted to further explore topographic structural features. Data acquisition was carried out using satellite remote sensing platforms, UAV remote sensing equipment, and geographic information monitoring systems.

[0085] Furthermore, in one embodiment of the present invention, step S2 further includes the following:

[0086] Data cleaning was performed on RGB images, DEM data, Canny indices, and DEM-derived data to remove outliers and missing values, and standardization was performed to eliminate dimensional differences between different types of remote sensing datasets.

[0087] In one embodiment of the present invention, step S2 includes the following:

[0088] Step S2, which calculates the terrain attributes of the DEM data and refines the terrain structure representation, includes the following:

[0089] The slope α is calculated using the finite difference formula:

[0090]

[0091] Where f x f y The slope change rate in the x and y directions of the slope to be calculated in the DEM data is used, and then the surface inclination is calculated point by point to quantify the steepness of the terrain.

[0092] Calculate slope aspect β:

[0093]

[0094] Use slope aspect to determine the orientation angle of terrain cells in DEM data;

[0095] Calculate curvature r:

[0096]

[0097] in, and These are the second partial derivatives along the x-axis and y-axis, respectively. The mixed second partial derivative reflects the geometric characteristics of the Earth's surface.

[0098] Furthermore, in one embodiment of the present invention, data cleaning is performed uniformly on the multiple types of remote sensing datasets to remove outliers and missing values, and the multiple types of remote sensing datasets are normalized to eliminate differences in units, thus laying a high-quality data foundation for subsequent feature extraction and model construction.

[0099] Further, step S3 includes the following:

[0100] Step S31: Use a dual-branch convolutional network to process the RGB image and DEM data respectively; the dual-branch convolutional network includes the following:

[0101] Reflection-filled convolutional layers: Employ a mirror-filling strategy to preserve boundary information;

[0102] Multi-scale feature fusion: fusing feature maps from different levels through skip connections;

[0103] Channel standardization: Maps bi-branch features uniformly to a 16-channel dimension;

[0104] Step S32: Perform hierarchical feature learning on the encoder module with a 3-layer residual structure that has processed RGB image and DEM data respectively through a dual-branch convolutional network. Each encoder module contains a convolutional layer and a ReLU activation function, and gradually extracts multi-level features from edge texture to terrain structure; thus obtaining the dual features of RGB image and DEM data.

[0105] Step S33: The dual features of the RGB image and DEM data are uniformly mapped to a 16-channel dimension through channel normalization.

[0106] Furthermore, in one embodiment of the present invention, step S3 further includes the following:

[0107] First, for the RGB image branch, the input feature map detected by Canny edges is used. A reflection-filled convolutional layer is applied to maintain spatial dimensionality, and a mirror-filling strategy is employed to enhance boundary information preservation. The DEM branch takes in terrain attribute data such as slope, aspect, and curvature, and processes it using the same reflection-filled convolutional layer. Subsequently, both branches undergo hierarchical feature learning through a 3-layer residual encoder (EB) module. Each EB layer contains a convolutional layer and a ReLU activation function, progressively extracting multi-level features from edge texture to terrain structure. Skip-layer connections are used to fuse feature maps from different levels at multiple scales, effectively integrating local details and global contextual information. Finally, the dual-branch features are uniformly mapped to a 16-channel dimension through channel normalization, providing structured feature input for subsequent cross-attention fusion. This design significantly improves the model's accuracy in capturing landslide features by preserving the original data characteristics and enhancing feature representation capabilities.

[0108] In one embodiment of the present invention, step S4 includes the following:

[0109] The visual language model in step S4 includes the following:

[0110] Visual encoding unit: A Transformer-based visual encoder is used to process RGB images, and the initial feature dimensions are related to the input image resolution;

[0111] Text encoding unit: Encodes predefined landslide-related text prompts using a pre-trained language model, generates semantic embeddings, aligns with visual features, and unifies the dimensions to preset values;

[0112] Multimodal alignment module: Through the visual unit attention mechanism, it achieves cross-alignment of visual features and text embeddings, calculates the similarity of image-text pairs, enhances the semantic representation of landslide features, and outputs a fused multimodal feature map.

[0113] Furthermore, in one embodiment of the present invention, step S4 further includes the following:

[0114] A CLIP-based visual language model is introduced to extract features. The Vision Transformer (ViT) is used as the visual encoding unit to process RGB images, generating 768-dimensional multi-scale visual features. Simultaneously, a pre-trained Transformer language model is used to encode landslide-related text cues (such as "exposed slope" or "vegetation broken"), generating semantic embeddings for the corresponding dimensions. The resulting text cues are as follows: Figure 5 As shown, the model achieves cross-matching of image-text features through a multimodal alignment mechanism, calculates similarity scores using a visual unit attention mechanism, and extracts semantically enhanced features to highlight the semantic relevance of landslide features. Features at each stage are normalized to ensure stability and consistency, fusing visual and textual information to capture subtle features in complex terrain. By pre-training the CLIP model on massive image-text pairs, the model significantly improves its ability to identify atypical or ambiguous landslide samples, providing high-quality multimodal feature input for subsequent fusion steps. This design enhances the model's accuracy in representing geological hazard features by combining the complementarity of vision and language.

[0115] In one embodiment of the present invention, step S5 includes the following:

[0116] The fork self-attention module in step S5 includes the following:

[0117] Self-attention units handle long-distance dependencies of single-modal features;

[0118] Cross-attention units calculate complementary weights between heterogeneous features;

[0119] The feature fusion unit achieves multimodal feature fusion through element-level addition.

[0120] Furthermore, in one embodiment of the present invention, the deep semantic integration of the multi-type remote sensing datasets in step S5 includes RGB image + DEM data, RGB image + Canny, and RGB + terrain attribute combination, wherein the RGB image + DEM data combination achieves the best classification effect.

[0121] Furthermore, in one embodiment of the present invention, step S5 further includes the following:

[0122] To perform two-stream fusion classification, the RGB and DEM features extracted by the S3 stage dual-branch convolutional network and the semantic enhancement features extracted by the S4 stage visual language model are first input into the processing framework. Each stream captures the global contextual information of the single-modal features through a self-attention module (SA), and then a cross-attention module (CA) calculates the complementary weights between the heterogeneous features, achieving preliminary interactive fusion of multi-source data. In CSAM, the outputs of SA and CA are processed in parallel and then weighted and integrated to generate a unified fusion feature.

[0123] Specifically, the SA path enhances the semantic consistency of single-modal features through normalization, score calculation, softmax, and Add&MLP operations; (c) the CA path also employs normalization, score calculation, softmax, and Add&MLP operations to focus on the complementarity of heterogeneous features. The features from both paths are iteratively optimized through multiple additions and MLP layers. Finally, the fused features are input into a global average pooling layer (GAP) and a linear classifier, outputting landslide classification results. This design significantly improves the classification accuracy and model robustness of landslide features in complex terrain through dual-stream parallelism and cross-attention mechanisms.

[0124] In one embodiment of the present invention, step S6 includes the following:

[0125] Step S61: Verify the model's ability to identify landslide samples, including the following:

[0126] The accuracy, recall, and F1 score were used to verify the model's ability to identify landslide samples.

[0127] Step S62: Deploy the trained landslide classification model to a real-world application scenario to build an intelligent monitoring system.

[0128] Furthermore, in one embodiment of the present invention, step S6 further includes the following:

[0129] In stage S6, a comprehensive performance evaluation of the landslide classification model was conducted using multi-dimensional indicators. First, the model's ability to identify landslide samples was verified through core indicators such as accuracy, recall, and F1 score. Experiments show that the model of this invention significantly outperforms traditional CNN models in classification accuracy under complex geological conditions, with a particularly significant improvement in recall for landslide samples, effectively reducing the risk of missed detections.

[0130] Ablation experiments with different data combinations show that the dual-input configuration of RGB images and DEM data significantly optimizes the F1 score compared to a single data type, demonstrating the crucial role of multi-type remote sensing dataset fusion in improving model performance. Further visualization analysis using Gradient Weighted Class Activation Mapping (Grad-CAM) reveals that the model's feature responses are more significant in landslide boundaries and topographically undulating areas, indicating that the cross-self-attention mechanism effectively integrates complementary information from multiple data sources. Furthermore, comparative experiments validate the necessity of the cross-self-attention module. Removing this module decreases the model's efficiency in utilizing heterogeneous features, resulting in a significant decline in classification performance, further confirming the core role of this mechanism in deep semantic fusion. In generalization tests across different geological scenarios, the model exhibits stable classification performance. Finally, model evaluation results demonstrate that this invention, through multi-type remote sensing dataset fusion, the cross-self-attention mechanism, and robust architecture design, significantly improves the accuracy and reliability of landslide classification, providing a more practical technical solution for geological disaster monitoring.

[0131] In Phase S6, the trained landslide classification model is deployed to real-world application scenarios to build an intelligent monitoring system. First, a FastAPI interface and a web visualization platform are developed to support online uploading of multiple types of remote sensing datasets (RGB images, DEM data) and real-time return of classification results. The API interface adopts a lightweight design, optimizing the model inference process to meet the response requirements under high-concurrency scenarios. The web platform integrates interactive map and heatmap visualization functions, intuitively displaying landslide classification results and risk level distribution.

[0132] To ensure the long-term effectiveness of the model, an automated update mechanism was constructed. New remote sensing data is collected periodically, and an incremental learning strategy is used to update the model parameters. Simultaneously, transfer learning techniques are combined to quickly adapt to the geological characteristics of new regions. An automated hyperparameter tuning module is introduced, dynamically adjusting key parameters such as the weights of the cross-self-attention module and the configuration of dual-branch network channels based on intelligent algorithms to ensure optimal model performance under different geological conditions.

[0133] This deployment scheme provides a convenient and efficient technical means for geological disaster monitoring through intelligent interface design and automated model maintenance. In practical applications, the system can seamlessly integrate with existing remote sensing monitoring networks, assisting geological personnel in quickly locating landslide risk areas, providing data support for emergency decision-making, and significantly improving the timeliness of disaster early warning and prevention.

[0134] like Figure 2As shown, in one embodiment of the present invention, the present invention proposes a landslide classification system based on a visual language model and a cross-attention mechanism, used to execute a landslide classification method based on a visual language model and a cross-attention mechanism as described in any one of the present invention, characterized in that it includes a landslide classification model, wherein the landslide classification model includes the following:

[0135] Data preprocessing module: performs edge detection on RGB images and calculates terrain attributes on DEM data;

[0136] Feature extraction module: A dual-branch convolutional network is used to process RGB and DEM data separately;

[0137] Cross-self-attention module: Enables deep fusion of heterogeneous features;

[0138] Visual language model module: performs interpretive feature extraction based on text and images;

[0139] Classification and evaluation module: Outputs classification results and performs performance evaluation.

[0140] According to a second aspect of the present invention, a landslide classification system based on a visual language model and a cross-attention mechanism includes an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements a landslide classification method based on a visual language model and a cross-attention mechanism as described in any one of the present invention.

[0141] According to a third aspect of the present invention, a landslide classification system based on a visual language model and a cross-attention mechanism includes a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements any of the landslide classification methods based on a visual language model and a cross-attention mechanism of the present invention.

[0142] In addition to the above, the present invention also has related embodiments, including the following:

[0143] like Figure 3 As shown, in one embodiment of the present invention, the dual-branch convolutional network includes: a reflection-filled convolutional layer, which uses a mirror-filling strategy to preserve boundary information; multi-scale feature fusion, which fuses feature maps of different levels through skip connections; and channel normalization, which maps the dual-branch features uniformly to a 16-channel dimension.

[0144] The dual-branch convolutional network employs a 3-layer residual structure with a channel configuration of [64, 112, 160], maintaining spatial dimensionality through reflection padding. The optimal classification performance is achieved by combining the multi-type remote sensing datasets (RGB+DEM).

[0145] like Figure 4 As shown, in one embodiment of the present invention, the cross-self-attention mechanism includes: a self-attention module, which generates attention weights using the Softmax function by calculating the similarity matrix of the query-key-value matrix; a cross-attention module, which uses a negative similarity calculation strategy to focus on complementary information between heterogeneous data; and residual connections and layer normalization, which preserve the original features through residual connections and stabilize the training process through layer normalization. The cross-self-attention module includes: a self-attention unit, which handles long-distance dependencies of single-modal features; a cross-attention unit, which calculates complementary weights between heterogeneous features; and a feature fusion unit, which achieves multimodal feature fusion through element-wise addition.

[0146] As shown in Table 1, the data comparison clearly demonstrates that in the landslide classification task, the model of this invention exhibits significant superiority over other models such as Swin, ResNet, ConvNeXt, Eva, and Focal in key metrics including accuracy, recall, and F1 score. In terms of classification accuracy, the model of this invention far surpasses other comparative models in its accuracy in classifying landslides, more accurately identifying landslide samples from non-landslide samples and reducing false positives and false negatives. Regarding recall, its recall capability for actual landslide samples is more prominent, meaning that in real landslide scenarios, it can capture all landslide instances to the maximum extent, avoiding unidentified landslide samples due to model oversights. As for the F1 score, a core indicator that comprehensively considers precision and recall, the model of this invention achieves a higher F1 score due to its superior accuracy and recall performance, fully demonstrating its excellent ability to balance accurate identification and comprehensive recall in the landslide classification task.

[0147] Model accuracy Recall rate F1 This invention 95.19 93.43 0.9380 Swin 92.31 89.45 0.9003 ResNet 94.23 86.04 0.8884 ConvNeXt 91.23 87.71 0.8854 Eva 94.35 92.57 0.9257 Focal 92.07 88.00 0.8946

[0148] Table 1 Comparison Data

[0149] This fully demonstrates that the present invention, through innovative design of a multi-type remote sensing dataset fusion strategy, a dual-branch convolutional network feature extraction architecture, and a cross-self-attention mechanism, can more efficiently integrate multi-source information such as RGB images and DEM data, deeply mine key features related to landslides such as texture and terrain, optimize feature extraction and fusion effects, and thus improve the classification model's ability to distinguish landslide samples. Compared with other comparative models, the method of the present invention has significant advantages in multi-source data utilization efficiency, landslide feature capture accuracy, and comprehensive performance optimization of the classification model, exhibiting stronger classification effectiveness and stability. It provides a more reliable and efficient technical solution for landslide classification tasks, effectively meeting the practical needs of accurate classification in the field of landslide identification.

[0150] Therefore, the landslide classification method and system proposed in this invention, based on a visual language model and cross-attention mechanism, can provide accurate landslide classification results for geological disaster research and prevention personnel, assist them in deeply analyzing the characteristics and evolution patterns of landslide data, thereby optimizing geological disaster monitoring programs and emergency management strategies, and improving the efficiency of landslide identification and the safety of the geological disaster prevention system.

[0151] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A landslide classification method based on a visual language model and a cross-attention mechanism, characterized in that, Includes the following: Step S1: Construct multi-type remote sensing datasets, including the acquisition of RGB images and DEM data; Step S2: Process the RGB image using the Canny edge detection algorithm and generate an edge feature map to enhance the landslide boundary and texture features; The terrain attributes of DEM data are calculated and the terrain structure representation is refined; then, data cleaning is performed on multiple types of remote sensing datasets to remove outliers and missing values, and standardization is performed to eliminate the dimensional differences between the multiple types of remote sensing data. Step S3: Use a dual-branch convolutional network to process RGB images and DEM data respectively; including initial feature extraction through the Stem module, followed by the use of the encoder module, combined with reflection-filled convolutional layers and multi-scale residual connections, to capture local features and preserve boundary information, and generate standardized feature maps; Step S4: Introduce a visual language model, combining the visual features of RGB images with predefined text prompts, and enhance multimodal semantic understanding through a visual unit attention mechanism; the visual language model aligns feature maps with text embeddings, captures the semantic context of landslide features in the image, and provides richer multimodal representations for subsequent cross-self-attention fusion; Step S5: Capture global contextual information from RGB images, DEM data, and visual language model outputs using a self-attention module, and fuse heterogeneous features using a cross-self-attention module to achieve deep semantic integration of multiple types of remote sensing datasets; Step S6: Input the fused features into a global average pooling layer for spatial dimension compression, and then output the landslide classification results through a linear classifier; evaluate the model performance in multiple dimensions to verify its ability to identify landslide samples and its generalization.

2. The landslide classification method based on a visual language model and cross-attention mechanism according to claim 1, characterized in that, Step S1 includes the following: Step S11: The data sources for the various types of remote sensing datasets include satellite remote sensing platforms, UAV remote sensing equipment, and geographic information monitoring systems; RGB images are used to capture surface texture and color information, while DEM data is obtained through a geographic information system and derived terrain attributes such as slope, aspect, and curvature are calculated to deeply explore terrain structural features.

3. The landslide classification method based on a visual language model and cross-attention mechanism according to claim 1, characterized in that, Step S2 includes the following: Step S2, which calculates the terrain attributes of the DEM data and refines the terrain structure representation, includes the following: The slope α is calculated using the finite difference formula: Where f x f y The slope change rate in the x and y directions of the slope to be calculated in the DEM data is used, and then the surface inclination is calculated point by point to quantify the steepness of the terrain. Calculate slope aspect β: Use slope aspect to determine the orientation angle of terrain cells in DEM data; Calculate curvature r: in, and These are the second partial derivatives along the x-axis and y-axis, respectively. The mixed second partial derivative reflects the geometric characteristics of the Earth's surface.

4. The landslide classification method based on a visual language model and cross-attention mechanism according to claim 1, characterized in that, Step S3 includes the following: Step S31: Use a dual-branch convolutional network to process the RGB image and DEM data respectively; The dual-branch convolutional network includes the following: Reflection-filled convolutional layers: Employ a mirror-filling strategy to preserve boundary information; Multi-scale feature fusion: fusing feature maps from different levels through skip connections; Channel standardization: Maps bi-branch features uniformly to a 16-channel dimension; Step S32: Perform hierarchical feature learning on the encoder module with a 3-layer residual structure that has processed RGB image and DEM data respectively through a dual-branch convolutional network. Each encoder module contains a convolutional layer and a ReLU activation function, and gradually extracts multi-level features from edge texture to terrain structure; thus obtaining the dual features of RGB image and DEM data. Step S33: The dual features of the RGB image and DEM data are uniformly mapped to a 16-channel dimension through channel normalization.

5. The landslide classification method based on a visual language model and cross-attention mechanism according to claim 1, characterized in that, Step S4 includes the following: The visual language model in step S4 includes the following: Visual encoding unit: A Transformer-based visual encoder is used to process RGB images, and the initial feature dimensions are related to the input image resolution; Text encoding unit: Encodes predefined landslide-related text prompts using a pre-trained language model, generates semantic embeddings, aligns with visual features, and unifies the dimensions to preset values; Multimodal alignment module: Through the visual unit attention mechanism, it achieves cross-alignment of visual features and text embeddings, calculates the similarity of image-text pairs, enhances the semantic representation of landslide features, and outputs a fused multimodal feature map.

6. The landslide classification method based on a visual language model and cross-attention mechanism according to claim 1, characterized in that, Step S5 Includes the following: The fork self-attention module in step S5 includes the following: Self-attention units handle long-distance dependencies of single-modal features; Cross-attention units calculate complementary weights between heterogeneous features; The feature fusion unit achieves multimodal feature fusion through element-level addition; The deep semantic integration of multiple types of remote sensing datasets mentioned in step S5 includes RGB image + DEM data, RGB image + Canny edge detection, and RGB + terrain attribute combination.

7. The landslide classification method based on a visual language model and cross-attention mechanism according to claim 1, characterized in that, Step S6 includes the following: Step S61: Verify the model's ability to identify landslide samples, including the following: The accuracy, recall, and F1 score were used to verify the model's ability to identify landslide samples. Step S62: Deploy the trained landslide classification model to a real-world application scenario to build an intelligent monitoring system.

8. A landslide classification system based on a visual language model and a cross-attention mechanism, used to execute a landslide classification method based on a visual language model and a cross-attention mechanism as described in any one of claims 1 to 7, characterized in that, This includes a landslide classification model, wherein the landslide classification model comprises the following: Data preprocessing module: performs edge detection on RGB images and calculates terrain attributes on DEM data; Feature extraction module: A dual-branch convolutional network is used to process RGB and DEM data separately; Cross-self-attention module: Enables deep fusion of heterogeneous features; Visual language model module: performs interpretive feature extraction based on text and images; Classification and evaluation module: Outputs classification results and performs performance evaluation.

9. A landslide classification system based on a visual language model and a cross-attention mechanism, comprising an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a landslide classification method based on a visual language model and a cross-attention mechanism as described in any one of claims 1 to 7.

10. A landslide classification system based on a visual language model and a cross-attention mechanism, comprising a computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements any one of the landslide classification methods based on a visual language model and a cross-attention mechanism as described in claims 1 to 7.

Images