Landslide remote sensing image segmentation method based on multi-modal fusion
By employing a multimodal adaptive fusion and boundary uncertainty refinement mechanism, the problems of insufficient utilization of multi-scale features and inadequate modal data fusion in landslide detection are solved, achieving high-precision segmentation of landslide remote sensing images and improving the accuracy and robustness of detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-24
- Publication Date
- 2026-04-07
AI Technical Summary
Existing landslide detection methods suffer from insufficient utilization of multi-scale features, blurred landslide boundaries, and inadequate fusion of multi-modal data, resulting in insufficient detection accuracy and robustness, and failing to meet the needs of efficient disaster management under complex terrain conditions.
By employing multimodal adaptive fusion, multi-scale feature extraction from the visual basic model, cross-scale bidirectional attention fusion, and boundary uncertainty refinement mechanisms, high-precision segmentation of landslide remote sensing images is achieved through multimodal data mapping, low-rank compression, dynamic weighted fusion, feature extraction from the visual basic model, and generation of boundary uncertainty maps.
It improves the accuracy and reliability of landslide detection, enables high-precision landslide area segmentation under complex terrain conditions, enhances multi-scale perception capabilities and boundary positioning accuracy, and reduces errors.
Smart Images
Figure CN121811037A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image analysis and geological disaster monitoring technology, and in particular to a landslide remote sensing image segmentation method based on multimodal fusion. Background Technology
[0002] Landslides are among the most destructive geological hazards, characterized by their suddenness and immense destructive power. They are characterized by the sudden movement of soil, rock, and debris along slopes, and commonly occur in geologically unstable areas such as mountainous regions and mining areas. Landslides not only severely damage infrastructure and cause significant economic losses, but also directly threaten people's lives. Due to the sudden nature of landslides, timely detection and precise demarcation of landslide areas are crucial for effective disaster management and risk mitigation.
[0003] Existing landslide detection methods are mainly divided into two categories: traditional methods and deep learning-based methods. Traditional landslide detection methods primarily rely on field investigations and visual interpretation, predicting landslides by analyzing factors such as topography, geology, and rainfall. These methods are typically time-consuming, labor-intensive, and dependent on expert experience, making it difficult to achieve rapid, large-scale responses and failing to meet the timeliness requirements of modern disaster prevention and mitigation. Deep learning, with its fast response speed and significantly reduced human workload, has become the mainstream method for landslide detection.
[0004] However, existing deep learning methods mainly suffer from the following drawbacks:
[0005] First, the utilization of multi-scale features is insufficient: Landslides exhibit highly multi-scale characteristics in remote sensing imagery, meaning that small landslides of tens of meters or even landslides of several kilometers can exist within the same image. Traditional segmentation networks' single-scale feature extraction struggles to simultaneously capture the overall morphology of large-scale landslide areas and the local details of small-scale landslides. Therefore, insufficient multi-scale feature modeling is a significant factor limiting the accuracy of landslide detection.
[0006] Second, there is the issue of boundary ambiguity: landslide boundaries often transition seamlessly with natural terrain. Their edge areas are spectrally similar to bare land, colluvial deposits, and vegetation destruction zones, and are significantly affected by factors such as topographic shadows, clouds, and the angle of sunlight. This boundary ambiguity makes models prone to "expansion" or "erosion" artifacts, leading to overestimations or underestimations of the landslide extent, thus affecting the reliability of disaster assessments. Existing segmentation models are prone to breakage or missed detections in boundary areas, failing to accurately depict landslide edges and potentially causing serious decision-making errors in actual disaster assessments.
[0007] Third, multi-source data fusion is insufficient: For landslide detection, existing methods typically integrate multimodal data such as RGB optical imagery and DEM (Digital Elevation Model). However, due to the significant differences in physical properties between RGB and DEM, direct stitching can lead to feature space conflicts due to modal differences, and single-channel DEMs are easily overwhelmed by the spectral information of three-channel RGB. Traditional fusion methods often employ simple stitching or weighted averaging strategies, failing to fully consider the differentiated dependence of different regions on different modal information: in vegetated areas, spectral information may be more important; while in exposed rock and soil areas, topographic information may be more discriminative. This "one-size-fits-all" fusion strategy cannot adapt to the changes in landslide characteristics under different terrain and cover conditions, resulting in the failure to fully leverage the synergistic effect of multi-source data.
[0008] In summary, existing landslide detection methods generally suffer from problems such as insufficient utilization of multi-scale features, blurred landslide boundaries, and inadequate fusion of multimodal data under complex terrain conditions, resulting in insufficient detection accuracy and robustness. Summary of the Invention
[0009] To address the aforementioned technical problems in existing technologies, there is an urgent need for a landslide segmentation method capable of multimodal information fusion, multi-scale feature interaction, and refined boundary optimization. This would improve the accuracy and reliability of landslide detection and provide more precise technical support for disaster management and risk assessment. Therefore, this invention provides a landslide semantic segmentation method based on multimodal fusion and boundary uncertainty. Through multimodal adaptive fusion, multi-scale feature extraction from Visual Foundation Models (VFMs), cross-scale bidirectional attention fusion, and a boundary uncertainty refinement mechanism, high-precision landslide segmentation is achieved.
[0010] Specifically, this invention provides a landslide remote sensing image segmentation method based on multimodal fusion, the technical solution of which is as follows:
[0011] By mapping the multimodal data of landslide remote sensing images to the same dimension, corresponding token sequences are obtained.
[0012] Perform low-rank compression on each token sequence to obtain the feature matrix;
[0013] The feature matrix is decoupled and dynamically weighted and fused sequentially to obtain fused features; the fused features are then input into the visual base model.
[0014] Multiple intermediate layers of the visual base model are selected, and scale features are extracted from them respectively;
[0015] For the scale features corresponding to the adjacent intermediate layers after selection, sampling fusion features are generated through upsampling, downsampling and gating enhancement;
[0016] The sampled and fused features are input into BUR to obtain convolutional enhanced features, which are then convolved to generate an uncertainty map.
[0017] The uncertainty map is input into the decoder to segment the landslide remote sensing image, resulting in a segmented landslide remote sensing image.
[0018] Preferably, the multimodal data of the landslide remote sensing image are mapped to the same dimension to obtain their respective corresponding token sequences, specifically:
[0019] RGB data and DEM data Mapping to the same dimension yields the corresponding token sequences, as shown in the following formula:
[0020] ;
[0021] ;
[0022] In the formula, RGB data The token sequence, For DEM data The token sequence, This means converting a two-dimensional image into a one-dimensional token sequence. This represents the number of blocks after the image is cut. The dimension of the feature vector. Indicates dimension.
[0023] Preferably,
[0024] (1) The low-rank compression of each token sequence to obtain the feature matrix is as follows:
[0025] Low-rank compression is performed using matrix multiplication to obtain the RGB feature matrix. and DEM feature matrix The formula is as follows:
[0026] ;
[0027] ;
[0028] ;
[0029] In the formula, It is a low-rank compressed matrix. This is an intermediate implicit dimension;
[0030] (2) The decoupling and dynamic weighted fusion of the feature matrix to obtain the fused features are as follows:
[0031] Predicting RGB feature matrices using CNN and DEM feature matrix Their respective offset diagrams, the formulas are as follows:
[0032] ;
[0033] ;
[0034] In the formula, RGB feature matrix Offset map, DEM feature matrix Offset map, This indicates a reshaping operation. Represents a convolutional neural network;
[0035] Compare their respective offset maps with the reference grid. The corresponding sampling points are obtained by adding them together, as shown in the following formula:
[0036] ;
[0037] ;
[0038] In the formula, RGB data sampling points, For DEM data Sampling points;
[0039] For RGB feature matrix and DEM feature matrix In the sampling point set The above is subjected to bilinear interpolation sampling, and the formula is as follows:
[0040] ;
[0041] ;
[0042] Among them, the set of sampling points The set consisting of all sampling points. RGB feature matrix The feature matrix obtained after sampling DEM feature matrix The feature matrix obtained after sampling This represents the number of tokens after sampling. Indicates bilinear interpolation sampling;
[0043] Calculate the global query fusion matrix The formula is as follows:
[0044] ;
[0045] In the formula, This indicates element-wise addition;
[0046] Upsampling is performed to obtain the global query fusion matrix. Features The formula is as follows:
[0047] ;
[0048] In the formula, Indicates upsampling;
[0049] Modality weight prediction is performed to obtain the original modality weights, using the following formula:
[0050] ;
[0051] In the formula, For the predicted original mode weights, It is a multilayer perceptron;
[0052] The original modal weights are normalized using the following formula:
[0053] ;
[0054] The result of normalization Perform weighted fusion to obtain weighted fusion features. The formula is as follows:
[0055] ;
[0056] In the formula, This indicates element-wise multiplication;
[0057] Weighted fusion features Upsampling is performed on the input visual base model to obtain fused features. The formula is as follows:
[0058] ;
[0059] In the formula, This is the upsampled projection matrix.
[0060] Preferably, multiple intermediate layers of the visual base model are selected, and scale features are extracted from them respectively, specifically:
[0061] Select multiple intermediate Transformer layers from the basic visual model;
[0062] For each selected hidden state in the intermediate layer, a token-to-2D feature map transformation is performed to obtain the feature map. The formula is as follows:
[0063] ;
[0064] In the formula, For the first The hidden state of the layer , The selected layer number;
[0065] Using a lightweight convolutional neural network Extracting landslide remote sensing images Detail features The formula is as follows:
[0066] ;
[0067] feature map With detailed features Complementary fusion is performed to obtain scale features. The formula is as follows:
[0068] ;
[0069] In the formula, This represents the sigmoid activation function. This represents a one-dimensional convolution operation. This indicates a channel dimension splicing operation.
[0070] Preferably, for the scale features corresponding to adjacent intermediate layers after selection, sampling fusion features are generated through upsampling, downsampling, and gating enhancement, specifically including:
[0071] High-resolution features obtained by sampling and low-resolution features The formula is as follows:
[0072] ;
[0073] ;
[0074] In the formula, Indicates downsampling, ;
[0075] Calculate downsampling attention and upsampling attention The formula is as follows:
[0076] ;
[0077] ;
[0078] In the formula, Represents the attention function. Indicates query projection, Indicates key and value projection;
[0079] The gating enhancement formula is as follows:
[0080] ;
[0081] ;
[0082] In the formula, To enhance features through downsampling, To enhance features through upsampling, This indicates enhanced gating;
[0083] Upsampling is performed using the following formula:
[0084] ;
[0085] ;
[0086] In the formula, For downsampling features, For upsampling features;
[0087] The following formula is used to perform convolution and fusion:
[0088] ;
[0089] ;
[0090] In the formula, For downsampling fusion features, This refers to the upsampling fusion features.
[0091] Preferably, gating enhancement is performed, specifically including:
[0092] ;
[0093] ;
[0094] ;
[0095] ;
[0096] In the formula, This indicates the channel attention module. This indicates the shuffling attention module. Represents the Gaussian error linear unit activation function. This indicates the average pooling operation.
[0097] Preferably, the sampled fusion features are input into BUR to obtain convolutional enhanced features, which are then convolved to generate an uncertainty map, specifically including:
[0098] Sample fusion features As input to BUR, the convolutional enhancement feature F is obtained, as shown in the following formula:
[0099] ;
[0100] ;
[0101] In the formula, Represents a 3D convolution operation;
[0102] Generate uncertainty graph The formula is as follows:
[0103] ;
[0104] Among them, uncertainty diagram It consists of the degree of uncertainty at all locations. Uncertainty diagram The size.
[0105] Furthermore, it also includes generating boundary guidance maps and monitoring uncertainty maps. Specifically:
[0106] Generate boundary guidance map The formula is as follows:
[0107] ;
[0108] In the formula, For a realistic mask, This represents the max pooling operation;
[0109] During training, the auxiliary loss is calculated. The formula is as follows:
[0110] ;
[0111] In the formula, Represents the binary cross-entropy loss function;
[0112] Dynamically adjust the scaling factor Minimize the total loss function The formula is as follows:
[0113] ;
[0114] In the formula, The predicted segmentation map output by the visual base model. This is a real label image. This represents the cross-entropy loss.
[0115] Furthermore, it also includes correcting the uncertainty graph. Specifically:
[0116] Calculate boundary correction terms The formula is as follows:
[0117] ;
[0118] In the formula, This represents the 1×1 convolutional prediction head used to generate the boundary correction term;
[0119] Calculate the boundary refinement map The formula is as follows:
[0120] ;
[0121] In the formula, To represent the uncertainty diagram The generated initial prediction map.
[0122] Preferably, the selected number of layers is 4, specifically the 5th, 8th, 11th, and 22nd layers.
[0123] Furthermore, it also includes:
[0124] Obtain feature map Next, multi-scale feature alignment is performed to obtain aligned feature maps. The formula is as follows:
[0125] ;
[0126] In the formula, This indicates bilinear interpolation upsampling. Indicates the target scale. The target scaling factor.
[0127] Compared to existing technologies, the technical solution provided by this invention effectively solves the heterogeneity problem of multimodal data by constructing fusion features. It employs a visual base model to extract scale features from multiple intermediate layers, i.e., multi-scale features, and constructs sampled fusion features, achieving bidirectional interaction between high-level semantics and low-level details, thus enhancing multi-scale perception capabilities. Specifically, the visual base model uses the semantic information of high-level features to guide the optimization of low-level features, feeding back low-level details to the high-level model. That is, by generating sampled fusion features, it fully integrates multimodal data, enhances the spatial accuracy of semantic features, and achieves accurate segmentation of landslide remote sensing images. Attached Figure Description
[0128] Figure 1 This is a flowchart illustrating a landslide remote sensing image segmentation method based on multimodal fusion in one embodiment of the present invention.
[0129] Figure 2 This is a schematic diagram of the structure of a multimodal adaptive fusion module in one embodiment of the present invention.
[0130] Figure 3 This is a schematic diagram of the process for multi-scale feature extraction in the intermediate layer in one embodiment of the present invention.
[0131] Figure 4 This is a schematic diagram of the cross-scale bidirectional attention fusion module in one embodiment of the present invention.
[0132] Figure 5 This is a flowchart illustrating the boundary uncertainty refinement mechanism in one embodiment of the present invention. Detailed Implementation
[0133] The technical solutions provided by the present invention will be further described in detail below with reference to the embodiments and accompanying drawings.
[0134] Example 1
[0135] like Figure 1 As shown, the framework enhances the performance of landslide remote sensing image segmentation through multimodal bidirectional attention and boundary uncertainty refinement, including steps S1 to S4.
[0136] Step S1: As Figure 2 As shown, preprocessing and decoupling of features, along with dynamic weighted fusion, are performed on multimodal input data to ensure that the multimodal data maintains spatial consistency.
[0137] Step S1 specifically includes:
[0138] Step S11: Channel mapping and low-rank compression stage, mapping the 3-channel RGB data and 1-channel DEM data of the landslide remote sensing image to the same dimension. , , and These represent RGB optical imagery and DEM digital elevation model data, respectively. The dimensions of the original input image. , Let represent the height and width of the input image, respectively. 3 and 1 indicate that the image has 3 channels and 1 channel, respectively. Then, they are projected onto a sequence of tokens of the same dimension through independent 1×1 convolutions, as shown in the following formula:
[0139] ;
[0140] ;
[0141] In the formula, RGB data The token sequence, For DEM data The token sequence, This means converting a two-dimensional image into a one-dimensional token sequence. This refers to the number of patches, i.e., the number of blocks after the image is cut. The dimension of the feature vector. Indicates dimension.
[0142] To reduce computational complexity, matrix multiplication is performed on both token sequences to achieve low-rank compression while preserving key feature information. The formula is as follows:
[0143] ;
[0144] ;
[0145] ;
[0146] In the formula, It is a low-rank compressed matrix. The intermediate latent dimension (rank) for low-rank compression is used to reduce the dimensionality of multimodal features before fusion and improve computational efficiency. and These are the RGB feature moments and DEM feature matrix after dimensionality reduction, respectively.
[0147] Step S12: Modal dynamic migration stage, generating reference mesh. The formula is as follows:
[0148] ;
[0149] In the formula, , "2" indicates the coordinate dimension of each point. , This represents the size of the image after it has been cut into blocks.
[0150] Next, the offset map is predicted using a CNN (Convolutional Neural Network), as shown in the following formula:
[0151] ;
[0152] ;
[0153] In the formula, RGB feature matrix Offset map, DEM feature matrix Offset map; This represents the reshaping operation, which transforms the feature matrix from a two-dimensional matrix into a three-dimensional tensor. This represents a convolutional neural network used to predict offsets, which accepts a vector array of shape... The input tensor outputs a shape of Offset map.
[0154] Compare their respective offset maps with the reference grid. The corresponding sampling points are obtained by adding them together, as shown in the following formula:
[0155] ;
[0156] ;
[0157] In the formula, RGB data sampling points, For DEM data The sampling points.
[0158] For RGB feature matrix and DEM feature matrix In the sampling point set The above is subjected to bilinear interpolation sampling, and the formula is as follows:
[0159] ;
[0160] ;
[0161] Among them, the set of sampling points The set consisting of all sampling points. RGB feature matrix The feature matrix obtained after sampling DEM feature matrix The feature matrix obtained after sampling This represents the number of tokens after sampling. This indicates bilinear interpolation sampling.
[0162] Next, the query generation and weight evaluation are combined to generate a global query fusion matrix as the global query signal, as shown in the following formula:
[0163] ;
[0164] In the formula, This indicates element-wise addition;
[0165] Upsampling is performed, features are evaluated, and a global query fusion matrix is obtained. Features The formula is as follows:
[0166] ;
[0167] In the formula, This indicates upsampling.
[0168] Modality weight prediction is performed to obtain the original modality weights, using the following formula:
[0169] ;
[0170] In the formula, The original mode weights predicted by the MLP. It is a multilayer perceptron;
[0171] The original modal weights are normalized using the following formula:
[0172] ;
[0173] The result of normalization Perform weighted fusion to obtain weighted fusion features. The formula is as follows:
[0174] ;
[0175] In the formula, This indicates element-wise multiplication.
[0176] Weighted fusion features Upsampling is performed on the input visual base model to obtain fused features. The formula is as follows:
[0177] ;
[0178] In the formula, This is the upsampled projection matrix.
[0179] Finally, the fusion features The features are fed into the Transformer layer of the visual base model.
[0180] Step S2: As Figure 3 As shown, a visual basic model is used as a feature extractor, and features are selected from multiple layers and fused with detailed features.
[0181] Step S2 specifically includes:
[0182] Multi-scale features are extracted from multiple intermediate layers of the visual base model. Optionally, layers 5, 8, 11, and 22 are selected as feature extraction points, corresponding to feature representations at different levels of abstraction.
[0183] First, the hidden states of each selected intermediate layer are transformed into 2D feature maps to obtain the feature maps. The formula is as follows:
[0184] ;
[0185] In the formula, For the first The hidden state of the layer , The selected number of layers is 4 in this embodiment.
[0186] Then, features at different scales are obtained through bilinear interpolation. The one-dimensional token sequence is rearranged into a two-dimensional feature map. Since the tokens are cut from the image in sequence, this operation can restore their spatial structure.
[0187] After obtaining the feature map, the feature map is then processed. Perform multi-scale feature alignment to obtain aligned feature maps. The formula is as follows:
[0188] ;
[0189] In the formula, This indicates bilinear (Biliear) interpolation upsampling. Indicates the target scale. The target scaling factor.
[0190] Meanwhile, to compensate for DINOv3's shortcomings in fine-grained details, an ultra-lightweight convolutional neural network was designed. The N extractor extracts detailed features from the original input image. The formula is as follows:
[0191] ;
[0192] In the formula, The original input image is the landslide remote sensing image.
[0193] feature map With detailed features Complementary fusion is performed to obtain scale features. The formula is as follows:
[0194] ;
[0195] In the formula, This represents the sigmoid activation function. This represents a one-dimensional convolution operation. This indicates a channel dimension splicing operation.
[0196] Step S3: As Figure 4 As shown, a bidirectional attention mechanism is used to fully model the relationship between corresponding scale features of different intermediate layers, thereby realizing bidirectional information flow.
[0197] Step S3 specifically includes (wherein, information flow is) and , and , and ):
[0198] Step S31: Bidirectional information flow. First, information is transmitted back from high scale to low scale and from low scale to high scale, i.e., downsampling high-resolution features and upsampling low-resolution features, as shown in the following formula:
[0199] ;
[0200] ;
[0201] In the formula, Indicates downsampling, Specifically, in this embodiment, the objects downsampled are respectively , and The corresponding upsampled objects are respectively , and .
[0202] Calculate downsampling attention and upsampling attention The formula is as follows:
[0203] ;
[0204] ;
[0205] In the formula, This represents the attention function, which calculates the similarity between the query and the key, and then performs a weighted sum of the values. This represents the query projection, which linearly transforms the input features into a query vector. This represents the key-value projection, which linearly transforms the input features into a key-value vector.
[0206] Next, gating enhancement is performed, using the following formula:
[0207] ;
[0208] ;
[0209] In the formula, To enhance features through downsampling, To enhance features through upsampling, This indicates enhanced gating.
[0210] Sampling is performed using the following formula:
[0211] ;
[0212] ;
[0213] In the formula, For downsampling features, For upsampling features;
[0214] The following formula is used to perform convolution and fusion:
[0215] ;
[0216] ;
[0217] In the formula, For downsampling fusion features, This refers to the upsampling fusion features.
[0218] The gating mechanism includes:
[0219] ;
[0220] ;
[0221] ;
[0222] ;
[0223] In the formula, This indicates the channel attention module. This indicates the shuffling attention module. Represents the Gaussian error linear unit activation function. This indicates the average pooling operation.
[0224] Step S4: As Figure 5 As shown, the Boundary Uncertainty Refinement (BUR) mechanism, through an uncertainty-guided feature enhancement strategy, significantly improves the localization accuracy of landslide boundary areas. In the figure, C represents convolution, and S represents the activation function.
[0225] Step S4 specifically includes:
[0226] During training, the visual base model first generates boundary guidance maps. This serves as a monitoring signal to supervise the uncertainty map and clarify the boundary location of the landslide area. For the real mask... The expansion is achieved through max pooling, as shown in the following formula:
[0227] ;
[0228] In the formula, This represents a 3×3 max pooling operation, which dilates the mask, extending the slippage region outward by one pixel. The boundary region is then obtained by subtracting it from the original mask. (Boundary guidance map) Highlighting the transition zone between the landslide and its surroundings can serve as key guidance information during training.
[0229] The core of the BUR mechanism is the boundary uncertainty refinement head (BURHead), whose structure is as follows: Figure 5 As shown, the BUR mechanism receives sampled fused features after Cross-Scale Bidirectional Attention Fusion (CSAF). As input to BUR, the convolutional enhancement feature F is obtained, as shown in the following formula:
[0230] ;
[0231] ;
[0232] In the formula, Represents a 3D convolution operation;
[0233] Generate an uncertainty graph for the uncertainty branch of the BUR mechanism. The formula is as follows:
[0234] ;
[0235] Among them, uncertainty diagram It consists of the degree of uncertainty at all locations. Uncertainty diagram The size. Indicates position The degree of uncertainty at a location. A high uncertainty value (close to 1) indicates that the boundary of the area is unclear or difficult to identify.
[0236] Finally, the uncertainty diagram The input decoder is used to segment the landslide remote sensing image, resulting in a segmented landslide remote sensing image.
[0237] During training, the auxiliary loss is calculated. The formula is as follows:
[0238] ;
[0239] In the formula, Represents the binary cross-entropy loss function;
[0240] Dynamically adjust the scaling factor Minimize the total loss function The formula is as follows:
[0241] ;
[0242] In the formula, The predicted segmentation map output by the visual base model. This is a real label image. This represents the cross-entropy loss.
[0243] This also includes a modified uncertainty diagram. Specifically:
[0244] Calculate boundary correction terms This is used to refine the initial segmentation results, and the formula is as follows:
[0245] ;
[0246] In the formula, This represents the 1×1 convolutional prediction head used to generate the boundary correction term;
[0247] The final boundary refinement result is obtained by adding the initial segmentation prediction to the correction term. (Calculate the boundary refinement map.) The formula is as follows:
[0248] ;
[0249] In the formula, To represent the uncertainty diagram The generated initial prediction map.
[0250] As can be seen from the above embodiments and figures, the landslide remote sensing image segmentation method provided by the present invention achieves high-precision automatic identification and segmentation of landslide areas by introducing multimodal adaptive fusion, cross-scale feature interaction and uncertainty-guided boundary optimization mechanisms.
[0251] The core advantages of this method are reflected in the following aspects: First, by using a multimodal adaptive fusion module to dynamically weight and fuse RGB optical images and DEM elevation data, the heterogeneity problem between multi-source data is effectively solved, enabling the model to adaptively select the optimal modal information according to terrain features, thereby improving the recognition accuracy under complex terrain conditions.
[0252] Secondly, by combining the multi-layer feature extraction and cross-scale bidirectional attention fusion mechanism of the visual basic model, bidirectional information interaction between high-level semantic features and low-level detail features is realized, which significantly enhances the model's multi-scale perception ability and structural expression ability.
[0253] Furthermore, the boundary uncertainty refinement module proposed in this invention effectively identifies and adaptively corrects ambiguous areas at the edge of landslides through an uncertainty-guided feature enhancement strategy, significantly improving the "expansion" or "erosion" artifact problems that traditional segmentation models are prone to in the boundary region.
[0254] As shown in Tables 1 and 2, experiments have demonstrated that the technical solution provided by this invention exhibits superior segmentation performance compared to existing technologies in various remote sensing scenarios and geological conditions, especially demonstrating higher accuracy and robustness in complex terrain and small-scale landslide detection tasks.
[0255] Table 1 Comparative Experimental Results
[0256]
[0257] In Table 1, Unet, Segformer, TransUNet, TransLandSeg, DCT-UNet++, and APSAM are classic methods for comparison.
[0258] Table 2 Ablation Experiment Results
[0259]
[0260] In summary, compared to existing technologies, the technical solution provided by this invention effectively solves the heterogeneity problem of multimodal data by constructing fusion features. It employs a visual base model to extract scale features from multiple intermediate layers, i.e., multi-scale features, and constructs sampled fusion features, achieving bidirectional interaction between high-level semantics and low-level details, thus enhancing multi-scale perception capabilities. Specifically, the visual base model uses the semantic information of high-level features to guide the optimization of low-level features, feeding back low-level details to the high-level model. That is, by generating sampled fusion features, it fully integrates multimodal data, enhances the spatial accuracy of semantic features, and achieves accurate segmentation of landslide remote sensing images.
Claims
1. A landslide remote sensing image segmentation method based on multimodal fusion, characterized in that, include: By mapping the multimodal data of landslide remote sensing images to the same dimension, corresponding token sequences are obtained. Perform low-rank compression on each token sequence to obtain the feature matrix; The feature matrix is decoupled and dynamically weighted and fused sequentially to obtain fused features; the fused features are then input into the visual base model. Multiple intermediate layers of the visual base model are selected, and scale features are extracted from them respectively; For the scale features corresponding to the adjacent intermediate layers after selection, sampling fusion features are generated through upsampling, downsampling and gating enhancement; The sampled and fused features are input into BUR to obtain convolutional enhanced features, which are then convolved to generate an uncertainty map. The uncertainty map is input into the decoder to segment the landslide remote sensing image, resulting in a segmented landslide remote sensing image.
2. The landslide remote sensing image segmentation method based on multimodal fusion as described in claim 1, characterized in that, The process of mapping multimodal data from landslide remote sensing images to the same dimension to obtain corresponding token sequences is as follows: RGB data and DEM data Mapping to the same dimension yields the corresponding token sequences, as shown in the following formula: ； ； In the formula, RGB data The token sequence, For DEM data The token sequence, This means converting a two-dimensional image into a one-dimensional token sequence. This represents the number of blocks after the image is cut. The dimension of the feature vector. Indicates dimension.
3. The landslide remote sensing image segmentation method based on multimodal fusion as described in claim 2, characterized in that, (1) The low-rank compression of each token sequence to obtain the feature matrix is as follows: Low-rank compression is performed using matrix multiplication to obtain the RGB feature matrix. and DEM feature matrix The formula is as follows: ； ； ； In the formula, It is a low-rank compressed matrix. This is an intermediate implicit dimension; (2) The decoupling and dynamic weighted fusion of the feature matrix to obtain the fused features are as follows: Predicting RGB feature matrices using CNN and DEM feature matrix Their respective offset diagrams, the formulas are as follows: ； ； In the formula, RGB feature matrix Offset map, DEM feature matrix Offset map, This indicates a reshaping operation. Represents a convolutional neural network; Compare their respective offset maps with the reference grid. The corresponding sampling points are obtained by adding them together, as shown in the following formula: ； ； In the formula, RGB data sampling points, For DEM data Sampling points; For RGB feature matrix and DEM feature matrix In the sampling point set The above is subjected to bilinear interpolation sampling, and the formula is as follows: ； ； Among them, the set of sampling points The set consisting of all sampling points. RGB feature matrix The feature matrix obtained after sampling DEM feature matrix The feature matrix obtained after sampling This represents the number of tokens after sampling. Indicates bilinear interpolation sampling; Calculate the global query fusion matrix The formula is as follows: ； In the formula, This indicates element-wise addition; Upsampling is performed to obtain the global query fusion matrix. Features The formula is as follows: ； In the formula, Indicates upsampling; Modality weight prediction is performed to obtain the original modality weights, using the following formula: ； In the formula, For the predicted original mode weights, It is a multilayer perceptron; The original modal weights are normalized using the following formula: ； The result of normalization Perform weighted fusion to obtain weighted fusion features. The formula is as follows: ； In the formula, This indicates element-wise multiplication; Weighted fusion features Upsampling is performed on the input visual base model to obtain fused features. The formula is as follows: ； In the formula, This is the upsampled projection matrix.
4. The landslide remote sensing image segmentation method based on multimodal fusion as described in claim 3, characterized in that, The process involves selecting multiple intermediate layers of the visual base model and extracting scale features from them, specifically: Select multiple intermediate Transformer layers from the basic visual model; For each selected hidden state in the intermediate layer, a token-to-2D feature map transformation is performed to obtain the feature map. The formula is as follows: ； In the formula, For the first The hidden state of the layer , The selected layer number; Using lightweight convolutional neural networks Extracting detailed features from the original input image The formula is as follows: ； feature map With detailed features Complementary fusion is performed to obtain scale features. The formula is as follows: ； In the formula, This represents the sigmoid activation function. This represents a one-dimensional convolution operation. This indicates a channel dimension splicing operation.
5. The landslide remote sensing image segmentation method based on multimodal fusion as described in claim 4, characterized in that, The process of generating sampled fusion features from the scale features corresponding to adjacent intermediate layers after selection through upsampling, downsampling, and gating enhancement specifically includes: High-resolution features obtained by sampling and low-resolution features The formula is as follows: ； ； In the formula, Indicates downsampling, ; Calculate downsampling attention and upsampling attention The formula is as follows: ； ； In the formula, Represents the attention function. Indicates query projection, Indicates key and value projection; The gating enhancement formula is as follows: ； ； In the formula, To enhance features through downsampling, To enhance features through upsampling, This indicates enhanced gating; Upsampling is performed using the following formula: ； ； In the formula, For downsampling features, For upsampling features; The following formula is used to perform convolution and fusion: ； ； In the formula, For downsampling fusion features, This refers to the upsampling fusion features.
6. The landslide remote sensing image segmentation method based on multimodal fusion as described in claim 5, characterized in that, The aforementioned gating enhancement specifically includes: ； ； ； ； In the formula, This indicates the channel attention module. This indicates the shuffling attention module. Represents the Gaussian error linear unit activation function. This indicates the average pooling operation.
7. The landslide remote sensing image segmentation method based on multimodal fusion as described in claim 5, characterized in that, The process of inputting the sampled and fused features into BUR to obtain convolutional enhanced features, followed by convolution to generate an uncertainty map, specifically includes: Sample fusion features As input to BUR, the convolutional enhancement feature F is obtained, as shown in the following formula: ； ； In the formula, Represents a 3D convolution operation; Generate uncertainty graph The formula is as follows: ； Among them, uncertainty diagram It consists of the degree of uncertainty at all locations. Uncertainty diagram The size.
8. The landslide remote sensing image segmentation method based on multimodal fusion as described in claim 7, characterized in that, It also includes generating boundary guidance graphs and monitoring uncertainty graphs. Specifically: Generate boundary guidance map The formula is as follows: ； In the formula, For a realistic mask, This represents the max pooling operation; During training, the auxiliary loss is calculated. The formula is as follows: ； In the formula, Represents the binary cross-entropy loss function; Dynamically adjust the scaling factor Minimize the total loss function The formula is as follows: ； In the formula, The predicted segmentation map output by the visual base model. This is a real label image. This represents the cross-entropy loss.
9. The landslide remote sensing image segmentation method based on multimodal fusion as described in claim 7, characterized in that, It also includes corrected uncertainty graphs Specifically: Calculate boundary correction terms The formula is as follows: ； In the formula, This represents the 1×1 convolutional prediction head used to generate the boundary correction term; Calculate the boundary refinement map The formula is as follows: ； In the formula, To represent the uncertainty diagram The generated initial prediction map.
10. A landslide remote sensing image segmentation method based on multimodal fusion as described in any one of claims 4 to 9, characterized in that, The selected layers are 4, namely the 5th, 8th, 11th and 22nd layers.
11. A landslide remote sensing image segmentation method based on multimodal fusion as described in any one of claims 4 to 9, characterized in that, Also includes: Obtain feature map Next, multi-scale feature alignment is performed to obtain aligned feature maps. The formula is as follows: ； In the formula, This indicates bilinear interpolation upsampling. Indicates the target scale. The target scaling factor.