Landslide identification method and device based on SEEM-SAFPN model, medium and product

Abstract

The invention discloses a landslide identification method and device based on an SEEM-SAFPN model, a medium and a product, and relates to the field of artificial intelligence. Firstly, a remote sensing landslide data set is obtained and preprocessed, and a multi-modal data set is constructed; each sample in the multi-modal data set comprises a remote sensing image, a text query, a non-text query and a corresponding mask; the SEEM model and the SAFPN module are combined to construct an SEEM-SAFPN model; the SEEM-SAFPN model comprises an encoder, a decoder and a prediction head; the encoder comprises a text encoder, an image encoder, an SAFPN module and a visual sampler; the multi-modal data set is divided into a training set, a verification set and a test set which are respectively used for training, verifying and testing the SEEM-SAFPN model, and the trained SEEM-SAFPN model is used as a landslide extraction model; the landslide extraction model is adopted to identify the landslide area in the remote sensing image, so that the landslide identification efficiency and accuracy can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a landslide identification method, equipment, medium and product based on the SEEM-SAFPN model. Background Art

[0002] With global climate change and intensified human activities, the frequent occurrence of landslides is increasingly impacting human society and the natural environment. Traditional landslide identification methods rely on manual experience or simple statistical models, resulting in low efficiency and accuracy. Summary of the Invention

[0003] The purpose of this application is to provide a landslide identification method, equipment, medium and product based on the SEEM-SAFPN model to improve the efficiency and accuracy of landslide identification.

[0004] To achieve the above objectives, this application provides the following solutions.

[0005] In the first aspect, the present application provides a landslide identification method based on the SEEM-SAFPN model, including: obtaining a remote sensing landslide dataset and preprocessing it to construct a multimodal dataset; each sample in the multimodal dataset includes a remote sensing image, a text query, a non-text query and a corresponding mask; combining the SEEM model and the SAFPN module to construct a SEEM-SAFPN model; the SAFPN module refers to an attention-enhanced progressive feature fusion strategy module; the SEEM-SAFPN model includes an encoder, a decoder and a prediction head; the encoder includes a text encoder, an image encoder, a SAFPN module and a visual sampler; dividing the multimodal dataset into a training set, a validation set and a test set, which are used for training, validating and testing the SEEM-SAFPN model respectively, and using the trained SEEM-SAFPN model as a landslide extraction model; and using the landslide extraction model to identify landslide areas in remote sensing images.

[0006] Optionally, the remote sensing landslide dataset is obtained and preprocessed to construct a multimodal dataset, specifically comprising: obtaining a remote sensing landslide dataset, wherein the remote sensing landslide dataset includes multiple landslide images and multiple non-landslide images and corresponding label information; the landslide images and non-landslide images are all remote sensing images; using landslide images as positive samples and non-landslide images as negative samples, and cleaning the remote sensing landslide dataset in accordance with a requirement that the positive-negative sample ratio is 1:1 to obtain a cleaned dataset; saving the label information of each sample in the cleaned dataset in a structured text form and converting it into a corresponding mask; wherein the mask corresponding to the landslide area is called a positive mask, and the mask corresponding to the background area is called a negative mask; randomly generating text queries and non-text queries corresponding to each sample by simulating a user in an automated manner; the text query includes categories and sentences; the non-text query includes points, boxes, graffiti, polygons, and example images; and storing the remote sensing images, text queries, non-text queries, and masks of each sample in a one-to-one correspondence to construct a multimodal dataset.

[0007] Optionally, the text encoder in the encoder is used to convert the text query into a text prompt P t The image encoder generates multi-scale feature maps F with different resolutions and semantic information levels by performing layer-by-layer convolution and downsampling operations on the input remote sensing image. v ; The multi-scale feature map F v It includes shallow feature maps with higher resolution but less semantic information and deep feature maps with lower resolution but stronger semantic information

[0008] Optionally, the SAFPN module includes an AFPN module and a SKAttention mechanism; the AFPN module refers to an adaptive feature pyramid network module; the SKAttention mechanism refers to a selective kernel attention mechanism; the SKAttention mechanism is used to Perform adaptive adjustment of attention weights to obtain the adjusted feature map The adjusted feature map With shallow feature maps After being processed by the AFPN module, the enhanced feature map F' is obtained. v .

[0009] Optionally, the processing of the SKAttention mechanism includes: the deep feature map Convolution is performed with convolution kernels of sizes 3×3, 5×5 and 7×7 respectively to obtain three feature maps U1, U2 and U3; based on the formula U=U1+U2+U3, the three feature maps U1, U2 and U3 are added element by element to generate a feature map U that integrates multiple receptive field information; the feature map U is embedded in global information through global average pooling, and its average value is calculated along the H and W dimensions to obtain a one-dimensional vector S of C×1×1 dimension representing the importance of each channel information; where H, W and C represent the height, width and number of channels of the feature map respectively; the one-dimensional vector S is compressed into an L-dimensional vector Z after passing through the fully connected layer; the L-dimensional vector Z is normalized using the Softmax function, and the attention weight of each channel is adjusted between each vector to obtain the attention weight vectors a, b and d, which are applied to the weighted calculation of the feature maps U1, U2 and U3 respectively to obtain the feature maps A1, A2 and A3; after feature selection, the feature maps A1, A2 and A3 are added element by element to obtain the adjusted feature maps

[0010] Optionally, the AFPN module performs horizontal connection, downsampling, upsampling and adaptive spatial fusion operations on shallow feature maps. and adjusted feature maps The multi-scale features are fused to obtain the enhanced feature map F' v .

[0011] Optionally, the visual sampler in the encoder is based on formula P v =VisualSampler(s,F' v ) All types of non-text queries and enhanced feature graph F' v Convert to visual cues P v ; Wherein VisualSampler() is the operation performed by the visual sampler; s is a mask of a point, box, scribble, polygon and / or sampling area from an example image; the enhanced feature map F' v With text prompt P t , visual prompts P v and memory prompt P m Map them together into the image-text joint semantic space, and pass them to the decoder after scale alignment operation;

[0012] The decoder uses a masked cross attention mechanism MaskedCrossAtt() based on the formula Generate memory reminders for the current stage in It is the memory reminder of the previous stage; M p is the mask of the previous stage; F' vis the enhanced feature map of the current stage;

[0013] The decoder, on the other hand, uses a masked self-attention mechanism MaskedSelfAtt() based on the formula Output mask embedding and category embeddings where q h represents a learnable query;

[0014] The prediction head is based on the formula Infer the mask M; the mask M represents the extracted landslide area; MaskPredictor() represents the prediction operation for the mask;

[0015] The prediction head is based on the formula Infer the semantic concept S; the semantic concept S represents the predicted category or sentence; ClassPredictor() represents the prediction operation for semantics.

[0016] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the landslide identification method based on the SEEM-SAFPN model.

[0017] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the landslide identification method based on the SEEM-SAFPN model.

[0018] In a fourth aspect, the present application provides a computer program product, including a computer program, which implements the landslide identification method based on the SEEM-SAFPN model when executed by a processor.

[0019] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0020] This application provides a landslide identification method, device, medium, and product based on the SEEM-SAFPN model. By combining the SEEM model with the SAFPN module, a multi-level feature pyramid is constructed while effectively utilizing attention to enhance features captured at different levels. This solves the problem of feature information loss or degradation caused by indirect interactions between non-adjacent layers in traditional feature pyramid networks, thereby enabling efficient prediction at multiple scales simultaneously. The constructed SEEM-SAFPN model not only avoids information loss caused by semantic gaps, but also further enhances the ability to capture landslide area boundary details and spatial distribution characteristics, optimizes landslide area boundary identification, effectively reduces missed detections and false detections, and significantly improves the efficiency and accuracy of landslide identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flowchart of a landslide identification method based on the SEEM-SAFPN model in this application;

[0023] Figure 2 Schematic diagram of positive and negative samples and their corresponding label information;

[0024] Figure 3 Schematic diagram of the overall framework of the SEEM-SAFPN model constructed for this application;

[0025] Figure 4 It is a structural diagram of the SAFPN module;

[0026] Figure 5 Schematic diagram of the processing process of the AFPN module;

[0027] Figure 6 Schematic diagram of the adaptive spatial fusion operation of three different levels of image features;

[0028] Figure 7 This is a schematic diagram of the processing of the SKAttention mechanism;

[0029] Figure 8 Schematic diagram of the interaction mode of SEEM model;

[0030] Figure 9 This is a schematic diagram of the Bijie landslide dataset of Wuhan University;

[0031] Figure 10 Schematic diagram comparing the loss values ​​of the SEEM-SAFPN model and the SEEM model during training;

[0032] Figure 11 This is a visual comparison of the landslide identification results of the SEEM-SAFPN model and the SEEM model;

[0033] Figure 12 Schematic diagram of the segmentation capability of the SEEM-SAFPN model in different task scenarios. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] The purpose of this application is to propose a landslide identification method, equipment, medium and product based on the SEEM-SAFPN model, aiming to improve the efficiency and accuracy of landslide identification.

[0036] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0037] In recent years, deep learning technology has shown significant advantages in the field of image processing. In addition, with the increasing computing resources, deep learning algorithms are also being optimized and improved. They can now mine the deep features of targets in remote sensing images (also called images), including morphology, texture, spectrum and spatial relationships, and can be effectively applied to the accurate identification and effective prediction of landslide areas. On this basis, this application provides a landslide identification method based on the SEEM-SAFPN model. In an exemplary embodiment, as Figure 1 As shown, the landslide identification method based on the SEEM-SAFPN model includes the following steps 1 to 4.

[0038] Step 1: Obtain a remote sensing landslide dataset and preprocess it to construct a multimodal dataset; each sample in the multimodal dataset includes a remote sensing image, a text query, a non-text query, and a corresponding mask.

[0039] The step 1 specifically includes the following steps 1.1 to 1.5.

[0040] Step 1.1: Obtain a remote sensing landslide dataset, which includes multiple landslide images and multiple non-landslide images and corresponding label information.

[0041] Among them, landslide images and non-landslide images are all remote sensing images. The landslide images are called positive samples, and the non-landslide images are called negative samples. Some examples of positive and negative sample images and their corresponding label information are as follows: Figure 2 The label information refers to the binary mask pixel value (0 / 255) at each pixel position in the image. The landslide area has a pixel value of 255, corresponding to the white area; the background area has a pixel value of 0, corresponding to the black area. The acquired remote sensing landslide dataset can be an existing dataset, such as the open remote sensing landslide dataset created by Wuhan University called the Bijie Landslide Dataset, or a user-created dataset.

[0042] Step 1.2: Use landslide images as positive samples and non-landslide images as negative samples. Clean the remote sensing landslide dataset to obtain the cleaned dataset, ensuring a 1:1 ratio of positive and negative samples. This means that the cleaned dataset must have a 1:1 ratio of positive and negative samples.

[0043] Step 1.3: Save the label information of each sample in the cleaned dataset in the form of structured text and convert it into the corresponding mask; the mask corresponding to the landslide area is called the positive mask (pm), and the mask corresponding to the background area is called the negative mask (nm).

[0044] like Figure 2 As shown in the figure, the label information of each sample in the cleaned dataset represents visual content. Therefore, the visual label information in the image needs to be stored as structured text. For example, a Python script can be used to convert the binary mask pixel values ​​(0 / 255) of the label information itself into the 0 / 1 values ​​required for model training, and use these as masks for the sample images. This process saves the label information in the image as structured text, facilitating subsequent training, inference, or evaluation.

[0045] Step 1.4: Generate text queries and non-text queries corresponding to each sample in an automated manner, imitating the user; the text queries include categories and sentences; the non-text queries include points, boxes, graffiti, polygons, and sample images.

[0046] The SEEM-SAFPN model constructed in this application is an improved SEEM model obtained by combining the SEEM model (Segment Everything Everywhere All at Once Model) with the proposed SAFPN module. During the training process, the improved new model allows high-level features (rich in semantic information but with lower resolution) to be directly fused with low-level features (high resolution but with less semantic information). This more effectively preserves and enhances the information of each layer of features, thereby improving the accuracy of landslide identification.

[0047] Because the SEEM model inputs textual and non-textual queries alongside remote sensing images, including points, boxes, scribbles, polygons, and masks, the cleaned dataset needs to be converted to the COCO dataset format. Specifically, Python code can be used to convert the pixel values ​​of 0 and 255 in the image mask to the pixel values ​​of 0 and 1 required for training. The JSON file required by the COCO dataset is then created, containing the image name, its corresponding ID, the input textual query, and the textual mask.

[0048] On the other hand, it is also necessary to simulate the user in an automated way to randomly generate text queries and non-text queries corresponding to each sample. The text query includes categories and sentences. For example, categories: "slope", "landslide"; sentences: "Identify landslide bodies that may have geological hazards", "landslide areas affecting roads", etc. The non-text queries include points, boxes, graffiti, polygons, and example images (also called reference images). During the training process, non-text queries are randomly generated by users in an automated way. During the use process after the model training is completed, non-text queries can be given manually by the user or generated randomly.

[0049] Step 1.5: Store the remote sensing image, text query, non-text query, and mask of each sample in a one-to-one correspondence to construct a multimodal dataset.

[0050] In the multimodal dataset constructed in this application, the remote sensing image, text query and non-text query corresponding to each sample are used as the input of the SEEM-SAFPN model, and the mask and semantic concept corresponding to each sample are used as the output of the SEEM-SAFPN model to train, verify and test the SEEM-SAFPN model.

[0051] Step 2: Combine the SEEM model and SAFPN module to build the SEEM-SAFPN model.

[0052] In order to achieve more efficient and accurate landslide information extraction, this application combines the attention-enhanced progressive feature fusion strategy with the SEEM model to construct a new landslide extraction model - SEEM-SAFPN model. Its overall framework is as follows Figure 3 As shown. The SEEM-SAFPN model combines the advantages of the SEEM model and the SAFPN module. The SEEM model is a multi-task, multi-modal unified segmentation model that aims to support general segmentation (including semantic segmentation, instance segmentation, and panoramic segmentation) and interactive segmentation through a unified network architecture. The SEEM model is based on a simple Transformer encoder-decoder architecture, and integrates a text encoder on this basis. Its decoding process is similar to the generative large language model (LLM), but supports multimodal input and multimodal output. In the SEEM model, all queries are passed to the decoder as prompts, while the image encoder, text encoder, and visual sampler serve as prompt encoders to encode various types of queries. Specifically, the SEEM-SAFPN model of this application uniformly encodes remote sensing images and non-text queries into visual prompts, and uses a text encoder to convert text queries into text prompts, thereby achieving alignment of visual prompts and text prompts. In addition, different types of prompts can achieve information complementarity and collaborative optimization through a cross-attention mechanism. Ultimately, the SEEM-SAFPN model is able to generate high-quality segmentation results using multiple cues, demonstrating excellent performance in complex task scenarios.

[0053] like Figure 3 As shown, the overall framework of the SEEM-SAFPN model adopts an encoder-decoder architecture, specifically including: 2.1) encoder; 2.2) decoder; and 2.3) prediction head, which are introduced as follows.

[0054] 2.1) Encoder

[0055] like Figure 3 As shown, the encoder of the SEEM-SAFPN model specifically includes: 2.1.1) text encoder; 2.1.2) image encoder; 2.1.3) SAFPN module; and 2.1.4) visual sampler.

[0056] 2.1.1) Text Encoder

[0057] like Figure 3 As shown in the figure, in the process of landslide identification, the text query such as category and sentence is converted into text prompt P by text encoder. t .

[0058] 2.1.2) Image Encoder

[0059] The image encoder generates multi-scale feature maps F with different resolutions and semantic information levels by performing layer-by-layer convolution and downsampling operations on the input remote sensing image I. v Among them, the multi-scale feature map F v A feature map with higher medium resolution but less semantic information is called a shallow feature map. The multi-scale feature map F v The feature map with lower medium resolution but stronger semantic information is called deep feature map.

[0060] For example, given a remote sensing image I of size H×W×3, where H and W are the height and width of the remote sensing image, respectively. In a specific embodiment, H=W=512. Figure 3 As shown in the figure, a remote sensing image I with a size of 512×512×3 is input into the image encoder. The image encoder is usually based on a convolutional neural network (CNN) or a transformer architecture. It gradually extracts the global features of the remote sensing image through multi-layer convolution or transformer modules and generates a multi-scale feature map F. v ={F v [2],F v [3],F v [4],F v [5]}={Res2,Res3,Res4,Res5}. v [2],F v [3],F v [4],F v [5] represent the image features of the 2nd, 3rd, 4th and 5th scales (also called feature maps), Figure 3 In the example, they are represented as Res2, Res3, Res4, and Res5.

[0061] For example, a ResNet-based encoder progressively extracts semantic information from an image through multiple convolutional layers and residual blocks. During this extraction process, the image encoder gradually reduces the spatial resolution of feature maps through downsampling operations (such as pooling or convolutions with a stride greater than 1), while increasing the number of channels to extract deeper, global semantic information. Feature maps Res2, Res3, Res4, and Res5 correspond to the outputs of different layers in the image encoder. Feature maps at each layer are progressively generated through convolutional layers and downsampling operations. Res2 has the highest spatial resolution and the lowest number of channels, making it suitable for capturing detailed image information such as edges and textures. Res3, with its lower spatial resolution and downsampling, begins to capture more abstract semantic information while retaining some detail. Res4 further reduces spatial resolution and increases the number of channels, enabling it to capture higher-level semantic information, such as object shape and partial structure. Res5, with the lowest spatial resolution and the highest number of channels, is suitable for capturing global semantic information, such as the overall layout and category information of a scene. In the SEEM-SAFPN model architecture of this application, these multi-scale feature maps no longer enter the image-text joint semantic space directly, but are enhanced by the SAFPN module before subsequent processing. In this embodiment, Res2 and Res3 are used as shallow feature maps. Use Res4 and Res5 as deep feature maps

[0062] 2.1.3) SAFPN module

[0063] like Figure 4 As shown, the SAFPN module constructed in this application includes an AFPN (Asymptotic Feature Pyramid Network) module and an SK (Selective Kernel) Attention mechanism (selective kernel attention mechanism), which are described in detail as follows.

[0064] 2.1.3.1) AFPN module

[0065] For the processing of multi-scale feature maps, the traditional FPN (Feature Pyramid Network) uses the high resolution of low-level features and the high semantic information of high-level features to fuse the features of these different layers and make predictions on each fused feature layer separately. However, this method will lead to problems such as feature information loss or degradation due to indirect interactions between non-adjacent layers. To solve this problem, researchers proposed an efficient feature fusion network structure designed for target detection tasks - AFPN, which has a structure like Figure 5 shown.

[0066] Like many target detection methods based on feature pyramid networks, AFPN does not change the structure of the backbone network before feature fusion, but instead extracts feature maps of different scales from the backbone in an orderly manner. However, after obtaining feature maps of different scales, compared with traditional FPN, AFPN supports direct interaction between non-adjacent layers. This design allows deep features (rich in semantic information but low resolution) to be directly fused with shallow features (high resolution but less semantic information), thereby more effectively retaining and utilizing the information of each layer of features. However, considering that the semantic gap between non-adjacent layer features is greater than the semantic gap between adjacent layer features, especially the bottom and top features, this directly leads to poor fusion effect of non-adjacent layer features. AFPN adopts a progressive strategy to fuse multi-level features and cleverly avoids this effect.

[0067] First, AFPN extracts the last layer of features from each feature layer of Backbone, generating a set of feature maps of different scales, represented as {C1, C2···C K}, Figure 5 Then, the shallow feature map C1 is fused with C2 through downsampling operation to obtain the new feature map P 12 , perform upsampling operation on C2 and fuse it with C1 to obtain a new feature map P 21 . In the process of multi-level fusion, AFPN uses the idea of ​​ASFF (Adaptive Spatial Fusion of Feature) to design a fusion step called "Adaptive Spatial Fusion (ASF)". On the one hand, in order to align the dimensions and prepare for feature fusion, the features are upsampled using 1×1 convolution and bilinear interpolation methods. On the other hand, downsampling is performed using different convolution kernels and strides. For example, a 2×2 convolution with a stride of 2 can be applied to achieve a 2x downsampling, a 4×4 convolution with a stride of 4 can be used to achieve a 4x downsampling, and an 8×8 convolution with a stride of 8 can be used to achieve an 8x downsampling. Taking the fusion of feature maps of three scales as an example, Figure 6 shown.

[0068] During the fusion process, Represents the feature vector from level k to level l at position (i, j) in the feature map x. is the result feature vector at level l obtained by adaptive spatial fusion of multi-level features, which is composed of feature vector and The linear combination definition is as follows:

[0069]

[0070] in and Represents the spatial weights of the three scale features at level l, affected by This mechanism effectively resolves the information conflict that can arise when fusing multi-scale features by dynamically adjusting the weights of features at different levels during the fusion process. Specifically, the ASF operation evaluates the spatial contribution of each feature layer to the object detection task and automatically adjusts its weight based on this contribution, prioritizing the fusion of more important or relevant feature information. This not only enhances the representational power of the fused features but also improves the model's detection accuracy for objects of different scales.

[0071] Finally, {C3, C4···, C K These higher-level features are incorporated into the fusion process. This effectively avoids large semantic gaps during feature fusion, thereby minimizing the loss or degradation of high-level semantic information during propagation and interaction. For example, C2 and C3 are fused together before being fused with C4, reducing the semantic gap between C2 and C4. Each fused feature map is then output by its own prediction module (Predict).

[0072] 2.1.3.2) SKAttention Mechanism

[0073] For the newly constructed SAFPN module in this application, before feature fusion, in order to enable the model to more accurately capture the multi-scale texture features and spatial context associations of the landslide area, the SKAttention mechanism is introduced to enhance the key information focusing ability of the feature map. Its structure is as follows: Figure 7 shown.

[0074] See also Figure 7 The processing process of the SKAttention mechanism specifically includes the following steps 2.1.3.2.1) to 2.1.3.2.8).

[0075] 2.1.3.2.1) Deep feature map After convolution with kernels of sizes 3×3, 5×5, and 7×7, three feature maps U1, U2, and U3 are obtained.

[0076] like Figure 3 As shown, in the SEEM-SAFPN model of this application, the deep feature map is input into the SKAttention mechanism for processing. Substitute Figure 7 Features in Figure X .like Figure 7 As shown, the features of the original input Figure X After convolution with kernels of sizes 3×3, 5×5, and 7×7, three feature maps U1, U2, and U3 are obtained.

[0077] 2.1.3.2.2) Based on the formula U=U1+U2+U3, the three feature maps U1, U2 and U3 are added element by element to generate a feature map U that integrates information from multiple receptive fields.

[0078] 2.1.3.2.3) The feature map U is embedded with global information through global average pooling (gap). Its average value is calculated along the H and W dimensions to obtain a one-dimensional vector S of dimension C×1×1 representing the importance of each channel information; where H, W, and C represent the height, width, and number of channels of the feature map, respectively.

[0079] Specifically, the c-th element Sc of the one-dimensional vector S is obtained by compressing the feature map U in the spatial dimension H×W, and the formula is as follows:

[0080]

[0081] Among them, U c (i, j) represents the cth element of the feature map U; (i, j) represents the position of the cth element; F gap () represents the global average pooling operation.

[0082] 2.1.3.2.4) The one-dimensional vector S is compressed into an L-dimensional vector Z after passing through the fully connected layer.

[0083] like Figure 7 As shown in Figure 2, the one-dimensional vector S is compressed into an L-dimensional vector Z through a fully connected layer (Fully Connected Layer, abbreviated as fcl) to provide guidance for accurate and adaptive selection. The formula is as follows:

[0084] Z=F fc (S) = δ(β(W s )) (3)

[0085] Among them F fc () represents the fully connected operation, δ is the ReLU activation function, β represents batch normalization, W s Is a weight matrix of shape R × C. In order to study the effect of parameter R on model efficiency, a reduction ratio r is introduced to control its value, as follows:

[0086] R=max(C / r,R min ) (4)

[0087] where R min Indicates the minimum value of R. In the experiment, R is set min is 32.

[0088] 2.1.3.2.5) Use the Softmax function to normalize the L-dimensional vector Z, and adjust the attention weight of each channel between each vector to obtain the attention weight vectors a, b, and d, which are applied to the weighted calculation of feature maps U1, U2, and U3, respectively, to obtain feature maps A1, A2, and A3.

[0089] This application uses multiple linear transformations to extract information from the channel dimension, converting the original L-dimensional vector Z into a C-dimensional vector. Softmax is then used for normalization, and the attention weights of each channel are adjusted between each vector to obtain the attention weight vectors a, b, and d. The formula is as follows:

[0090]

[0091] Wherein, the L-dimensional vector Z is the global compressed feature vector used for guidance. Ac, Bc, and Dc represent the vector of shape 1×R in the c-th row of the learnable parameter matrix A, B, and D of size C×R, respectively. c 、b c and d c The cth row vector in the attention weight vectors a, b, and d, respectively; c = 1, 2, ... C. a, b, and d represent the soft attention vectors for feature maps U1, U2, and U3. The final feature map A is obtained by applying the attention weights on various kernels.

[0092] 2.1.3.2.6) After feature selection, perform element-by-element addition on feature maps A1, A2, and A3 to obtain the adjusted feature map

[0093] Specifically, the attention weight vectors a, b, and d are applied to the weighted calculation of the feature maps U1, U2, and U3, respectively, and after feature selection, the feature maps A1, A2, and A3 are added element by element to obtain the adjusted feature map A. The formula is as follows:

[0094] A c =a c U1 c +b c U2 c +d c U3 c , a c +b c +d c =1 (8)

[0095] but,

[0096]

[0097] Among them, U1 c 、U2 c 、U3 c 、A c Represent the c-th row vector of feature maps U1, U2, U3 and feature map A respectively. The feature map A is adaptively adjusted by the attention weight of this mechanism, compared with the original feature Figure X , which can more effectively highlight the information of important channels while suppressing irrelevant or redundant features.

[0098] like Figure 4 and 5 As shown in the figure, since the shallow feature maps C1 and C2 usually contain a large amount of low-level information (such as edges, textures, etc.), when using the SKAttention mechanism for weighted fusion, some irrelevant or redundant features may be introduced into subsequent calculations, affecting the fusion effect. In contrast, the deep feature maps C3 and C4 have more abstract semantic features, and low-level redundant information has been gradually filtered out in the previous layers. Applying the SKAttention mechanism to these deep feature maps to obtain adjusted feature maps C3' and C4' can not only reduce the computational burden, but also make better use of these abstract, high-level semantic information while filtering irrelevant information, so that the fused features can be more effectively used for tasks such as target detection or classification, ensuring the effectiveness and accuracy of feature fusion. Therefore, the structural diagram of the new module SAFPN proposed in this application combining the SKAttention mechanism and the AFPN module is as follows. Figure 4 shown.

[0099] Reference Figure 3 and Figure 4 In the SEEM-SAFPN model of this application, the image encoder is used as the backbone network in the SAFPN module and is connected to the SAFPN module. Specifically, the shallow feature map Substitute them into feature maps C1 and C2 respectively, and transform the deep feature maps Substitute them into feature maps C3 and C4 respectively. After feature maps C3 and C4 are processed by the SKAttention mechanism, the adjusted feature maps are obtained. The adjusted feature map With shallow feature maps After being processed by the AFPN module, the enhanced feature map F' is obtained. v ={F' v [2],F' v[3],F' v [4],F' v [5]}. Specifically, Figure 5 and Figure 6 As shown, the AFPN module performs horizontal connection, downsampling, upsampling and adaptive spatial fusion operations on shallow feature maps. and adjusted feature maps The multi-scale features are fused to obtain the enhanced feature map F' v , and output through the prediction module (Predict) in the SAFPN module.

[0100] 2.1.4) Visual Sampler

[0101] like Figure 3 As shown, the remote sensing image I is first extracted by the image encoder to obtain a multi-scale feature map F v , and then the enhanced feature map F' is obtained through the SAFPN module v The visual sampler enhances the feature map F' v and all types of non-text queries (e.g., points, boxes, scribbles, polygons, and sampled regions from another example image) are converted into visual cues P v , the formula is as follows:

[0102] P v =VisualSampler(s,F' v ) (10)

[0103] Where VisualSampler() is the operation performed by the visual sampler. s is a point, box, scribble, polygon and / or a sampling region from an example image (also called a reference region). The visual sampler first pools the features of the corresponding region from the image features using a point sampling method. For all visual cues (such as points, boxes, scribbles, polygons, etc.), a maximum of 512 point feature vectors are extracted from the sampling region specified by the cues in a uniform interpolation manner. When relying solely on the textual cues P t These non-text queries help disambiguate user intent when the correct segment cannot be identified.

[0104] 2.2) Decoder

[0105] like Figure 3 As shown, the enhanced feature map F' v As image features, and text hint P t , visual prompts P v and memory prompt P mThe visual cues P are mapped to the joint semantic space of image and text together, and after further scale alignment, they are passed to the decoder in a unified form. Through the joint training of panoptic segmentation and reference segmentation, the visual cue P is realized in the joint semantic space of image and text. v With text prompt P t The natural alignment of the image and the surrounding image greatly improves the segmentation accuracy and the intuitiveness of user interaction. It is also flexible and scalable, making it suitable for fine-grained segmentation tasks in complex scenes.

[0106] On the one hand, the enhanced feature map F' obtained by the encoder v As well as mapping various prompts to the image-text joint semantic space, the decoder can combine the memory prompts generated during the previous segmentation through the masked cross attention mechanism MaskedCrossAtt() Get the memory hints needed for this segmentation The formula is as follows:

[0107]

[0108] in, It is a memory reminder for the current stage; is the memory cue of the previous stage; the memory cue encodes historical information by using a mask-guided cross-attention layer. p is the mask of the previous stage; F' v is the enhanced feature map of the current stage. In this way, the cross attention is only effective in the area specified by the previous mask. Interacts with other prompts in the decoder to convey historical information about the current round.

[0109] On the other hand, the decoder uses a masked self-attention mechanism MaskedSelfAtt() to learn the query Q h With text, visual and memory cues <P t 、P v 、P m > Collaborative interaction, output mask embedding and category embeddings The formula is as follows:

[0110]

[0111] In MaskedCrossAtt() and MaskedSelfAtt(), the symbols ";" and "|" are separators used to separate different input parameters. For example, in MaskedCrossAtt(), the symbol ";" is used to separate the main input parameters, i.e., the memory prompts of the previous stage. and the mask M of the previous stage p The symbol "|" is used to separate the main input parameters from the additional input parameters, such as separating the enhanced feature map F' of the current stage v . <P t 、P v 、P m > represents a collection of textual cues, visual cues, and memory cues.

[0112] Although the SEEM model architecture is simple, it also uses complex interactions between queries and prompts, such as Figure 8 As shown, Figure 8 Part (a) shows how queries and prompts interact. Figure 8 Part (b) shows the self-attention masking approach. The query Q can be learned h It is a set of trainable vectors in the decoder, which is usually used to guide the model's attention mechanism and help the model extract relevant information from the input features. These query vectors are the parameters of the model and will be optimized through back propagation during the training process. During the training process, the decoder can learn the query Q h Copied as object query Q o , text query Q t and visual query Q v , each task has the same weight for different segmentation tasks such as general, reference and interactive segmentation, and the corresponding prompts <P t 、P v 、P m >Freely interact with their queries via masked self-attention.

[0113] The SEEM model on which this application is based is a multimodal segmentation architecture. It realizes the multi-task unification of general segmentation (semantic / instance / panoramic) and interactive segmentation through the Transformer encoder-decoder framework. It constructs a cross-modal prompt encoding system: the image encoder extracts the image features of the remote sensing image and outputs the feature map; the visual sampler converts points, boxes, graffiti, polygons and example images into visual prompts; the text encoder maps the semantic query into text prompts; the three are fused into the image-text joint semantic space through a parameter-sharing prompt embedding layer. The decoder based on the cross-attention mechanism and the self-attention mechanism dynamically integrates four types of interactive prompts (feature maps, text prompts, memory prompts and visual prompts) to achieve collaborative optimization of visual-text features, and finally outputs the segmentation results and text information through the prediction head. This design breaks through the traditional segmentation model's dependence on a single input modality.

[0114] 2.3) Prediction Head

[0115] The prediction head is embedded based on the output mask and category embeddings The mask M and semantic concept S are inferred as follows:

[0116]

[0117] MaskPredictor() performs mask-based predictions, while ClassPredictor() performs semantic-based predictions. Mask M represents the extracted landslide region, indicating the landslide location in the remote sensing image. The semantic concept S represents the predicted class or statement, such as "landslide," "landslide area affecting roads," and so on.

[0118] The SEEM-SAFPN model constructed in this application combines the SEEM model with the proposed SAFPN module. The new model allows high-level features (rich in semantic information but low in resolution) to be directly fused with low-level features (high in resolution but less in semantic information) during training, more effectively retaining and enhancing the information of each layer of features, thereby improving the accuracy of landslide identification.

[0119] Step 3: Divide the multimodal dataset into training set, validation set and test set, which are used to train, validate and test the SEEM-SAFPN model respectively. The trained SEEM-SAFPN model is used as the landslide extraction model.

[0120] Wuhan University created an open remote sensing landslide dataset called the Bijie landslide dataset for developing automatic landslide detection methods. The Bijie landslide dataset consists of satellite optical remote sensing images, shape files of landslide boundaries, and digital elevation models. Figure 9 As shown in the figure, the remote sensing images in the Bijie landslide dataset consist of 770 landslide images and 2003 non-landslide images. All images are from TripleSat satellite images taken between May and August 2018 and have been cropped. Figure 9 In the figure, the red dot indicates the landslide location, and the remote sensing image collected at the red dot location is the landslide image.

[0121] In one specific embodiment, to train, validate, and test the SEEM-SAFPN model, a dataset of 2,773 remote sensing images was cleaned into 770 landslide images and 770 non-landslide images, with a positive-to-negative sample ratio of 1:1, for a total of 1,540 images. The cleaned landslide and non-landslide images were then divided into training, validation, and test sets at a ratio of 8:1:1. The total number of images in the training set was 1,232, the validation set was 154, and the test set was 154. The positive-to-negative sample ratio in each of the training, validation, and test sets was 1:1.

[0122] exist Figure 3 In the new model architecture shown, during landslide training and inference, the SEEM-SAFPN model first converts text queries (e.g., category: "slope," sentence: "Identify landslide bodies that may pose geological hazards") into text prompts through a text encoder. Simultaneously, a 512×512×3 remote sensing image is fed into the image encoder, which extracts multi-scale feature maps of the landslide image: Res2 and Res3 (high resolution, rich in detail), and Res4 and Res5 (low resolution, strong in semantic information).

[0123] In the new model architecture, the multi-scale feature map no longer directly enters the image-text joint semantic space, but is first input into the SAFPN module, which uses an attention-enhanced progressive feature fusion strategy to gradually fuse feature information from non-adjacent layers for subsequent processing. Specifically, the deep feature maps Res4 and Res5 have stronger semantic information because they filter out redundant information. These feature maps are first enhanced through the SK Attention mechanism to improve the model's global perception of the landslide area. Subsequently, the enhanced deep feature map and the shallow feature maps Res2 and Res3 are jointly input into the AFPN module, and after aligning the scale through upsampling or downsampling, as shown in the figure. Figure 4 As shown in the figure, the ASF mechanism is used at each layer to dynamically adjust the fusion weights of feature layers at different scales, enabling semantically rich high-level features and high-resolution low-level features to work efficiently together to form a multi-scale semantically enhanced feature map. As a result, SAFPN avoids information loss caused by semantic gaps while further enhancing its ability to capture the boundary details and spatial distribution characteristics of landslide areas. Through this design, SAFPN avoids information loss caused by semantic gaps while further enhancing its ability to capture the boundary details and spatial distribution characteristics of landslide areas.

[0124] Then, the fused enhanced feature map F' output by the SAFPN module v On the one hand, F' vAll types of non-text queries are converted into visual cues. Furthermore, textual, visual, and memory cues are mapped into a joint image-text semantic space. After further scale alignment, they are passed to the decoder in a unified form. Memory cues are obtained in the decoder through cross-attention for the current stage and are used in the next training pass (no memory cues are present in the first computation, as the previous stage is not present). Finally, the decoder uses self-attention and cross-attention mechanisms to collaboratively interact with the learnable query and various cues, outputting mask embeddings and category embeddings to generate the landslide segmentation result (mask M) and semantic concept S. The proposed training method is summarized in PyTorch-style pseudocode in Table 1. Note that the parameters / functions in the pseudocode cannot be italicized, so the parameter / function format will differ from that in the previous text. The comments following each code sentence will govern.

[0125] Table 1 Pseudocode of SEEM-SAFPN model

[0126]

[0127]

[0128] The positive mask pm refers to the landslide area mask, and the negative mask nm refers to the background area mask. Qh is a trainable parameter that is randomly initialized during model construction and automatically learns the most appropriate representation through a data-driven approach throughout the training process. msa refers to the self-attention mask between the query Q and the prompt P. It is converted into a weight matrix by calculating the similarity between Q and P. Matches below a certain threshold are weakened or even blocked, forming a binary or continuous value mask matrix, thereby suppressing irrelevant or invalid information interaction.

[0129] To verify the accuracy of the model, extensive experiments were conducted on the Bijie landslide dataset from Wuhan University to validate the effectiveness of the landslide extraction method. The dataset was divided into training, validation, and test sets in an 8:1:1 ratio. The proposed SEEM-SAFPN model was trained using Python 3.8.15 as the programming language, and image data preprocessing libraries such as Numpy, Pillow, and OpenCV were used. The SEEM-SAFPN model was constructed and trained using the PyTorch 1.8.1 framework, with GPU acceleration using CUDA 10.2.0. All comparative experiments were trained for 50 epochs, with a batch size of 4 and an initial learning rate of 6e-05. A polynomial decay strategy was used to adjust the learning rate, and a learning rate warmup mechanism was also introduced. At the beginning of training, the learning rate was gradually increased from a low value to the preset initial value. In the early stages of training, the learning rate was gradually increased from a low value to the set initial learning rate, and the learning rate growth during the warmup process was linear. The minimum value of the learning rate decay will not be lower than 1.0000000000000002e-06. The specific parameters during the training process are shown in Table 2 below.

[0130] Table 2 Experimental parameters and training settings

[0131]

[0132]

[0133] In order to better monitor the model training process, Wandb is also used to visualize the Loss value curve, such as Figure 10 shown. Figure 10 In the figure, the horizontal axis represents the training rounds, and the vertical axis represents the change of the loss value (loss value) of different loss functions in each task; Grounding, Spatial and Mask represent semantic alignment, spatial perception and mask extraction tasks respectively; Dice, Bce and Ce represent Dice loss, Bce loss and Ce loss respectively. Figure 10 As can be seen from the results, the SEEM-SAFPN model outperforms the SEEM model in terms of optimization speed, training stability, and final loss value, whether in semantic alignment (Grounding), spatial perception (Spatial), or mask extraction (Mask). This fully demonstrates the excellent performance of the SEEM-SAFPN model in feature fusion and semantic information extraction. After integrating the SAFPN module, the model can more efficiently handle the interaction and fusion between features at different levels, thereby significantly improving the overall learning effect.

[0134] In landslide identification tasks, landslide areas often exhibit varying spatial scales and morphological variations. The SEEM-SAFPN model, with its unique structure, effectively overcomes this problem, better capturing multi-scale information and significantly improving identification accuracy. To validate the effectiveness of the proposed SEEM-SAFPN model, experiments were conducted on the Bijie landslide dataset from Wuhan University. The dataset was divided into training, validation, and test sets in an 8:1:1 ratio, with a 1:1 ratio of positive and negative samples. The SEEM-SAFPN model was trained and validated using the training and validation sets, and the performance of the improved SEEM-SAFPN model was evaluated on the test set. The parameters of the trained SEEM-SAFPN model were encapsulated and used as a landslide extraction model for landslide identification.

[0135] The trained SEEM-SAFPN model is tested using the test set, and the visualization results on the test set are compared with the original SEEM model. Figure 11 As shown in the figure, landslide represents the identified landslide area. Figure 11 The results show that the SEEM-SAFPN model constructed in this application outperforms the original SEEM model in landslide identification. The newly proposed SEEM-SAFPN model not only generates more realistic landslide edges, but also accurately identifies landslides in complex environments with unclear texture features.

[0136] Furthermore, we used metrics such as global average accuracy (aAcc), mean F-score (mFscore), mean precision (mPrecision), mean recall (mRecall), mean intersection over union (mIoU), mean Dice coefficient (mDice), and mean accuracy (mAcc) to compare with traditional FCN, DeepLab v3+, SegFormer, and the original SEEM model. aAcc reflects the model's ability to correctly classify pixels across the entire image, and is the ratio of correctly classified pixels to the total number of pixels. The formula is as follows:

[0137]

[0138] Where N is the total number of categories, TP n Indicates the number of the nth category correctly identified as positive, TN n Indicates the number of the nth category correctly identified as negative class, P n Refers to the total number of positive classes, N n is the total number of actual negative classes. aAcc is the calculated global average accuracy (aAcc) value.

[0139] mFscore, mPrecision, and mRecall are a comprehensive measure of the accuracy and completeness of the model's ability to identify a specific class. mPrecision is the ratio of the number of correctly predicted positive classes (TP) to all predicted positive classes (TP + FP), as shown in Equation (16). mRecall is the ratio of the number of correctly predicted positive classes to the number of actual positive classes, as shown in Equation (17). mFscore is the harmonic mean of precision and recall, as shown in Equation (18).

[0140]

[0141] Among them, TP n Indicates the number of the nth category correctly identified as positive, TN n Indicates the number of the nth category correctly identified as negative class, FP n Indicates the number of other categories mistakenly identified as the nth category, FN n Indicates the number of times the nth category is mistakenly identified as other categories. mPrecision, mRecall, and mFscore are the calculated average precision (mPrecision), average recall, and average F score (mFscore), respectively.

[0142] mIoU and mDice evaluate the overlap between the predicted and true regions and are key metrics in the field of semantic segmentation. mIoU is the ratio of the intersection of the predicted positive class and the actual positive class to their union, as shown in formula (19). The mDice coefficient is the ratio of the intersection of twice the predicted positive class and the actual positive class to the sum of their respective numbers, similar to the F score, as shown in formula (20).

[0143]

[0144] Among them, mIOU and mDice are the calculated mean intersection-over-union (mIoU) value and the average Dice coefficient value, respectively.

[0145] These metrics were used to evaluate the performance of the trained SEEM-SAFPN model (a landslide extraction model) and compared with selected visualization results from the test set. This demonstrates the effectiveness and stability of the SEEM-SAFPN model in high-resolution urban landslide extraction. Table 3 shows a comparison of the quantitative evaluation results of different models on the test set.

[0146] Table 3 Quantitative evaluation results of different models on the test set

[0147] Model aAcc mFscore mPrecision mRecall mIoU mDice FCN 87.58 86.37 85.52 87.26 79.24 86.39 DeepLabv3+ 89.82 88.65 89.23 88.07 81.54 88.68 SegFormer 93.57 92.91 92.63 93.18 87.92 92.94 SEEM 93.89 93.32 93.71 92.94 88.56 93.36 SEEM-SAFPN 94.28 93.78 93.83 93.65 89.13 93.82

[0148] The results shown in Table 3 show that the SEEM-SAFPN model of this application outperforms the existing models in terms of indicators such as aAcc, mFscore, mPrecision, mRecall, mIoU, and mDice compared with the traditional FCN, DeepLab v3+, SegFormer, and the original SEEM model, indicating the effectiveness and stability of the SEEM-SAFPN model of this application in the landslide extraction task.

[0149] In order to comprehensively evaluate the performance of the landslide extraction model of this application, in addition to the semantic segmentation indicators, key evaluation indicators such as panoptic quality (PQ), segmentation quality (SQ) and recognition quality (RQ) are also used to evaluate the recognition ability of the SEEM-SAFPN model and the SEEM model. For interactive segmentation, the number of clicks (NoC) metric is used to evaluate the interactive segmentation performance. It measures the number of clicks required to achieve a certain IoU, namely 50%, 85% and 90%, denoted as NoC50, NoC85 and NoC90 respectively. These indicators not only reflect the accuracy and robustness of the model in the image segmentation task, but also can fully measure its comprehensive ability in target recognition and region segmentation. Among them, SQ focuses on the segmentation accuracy of the target area; RQ calculates the recognition accuracy of the target, that is, the indicator to judge whether each target is correctly recognized; and PQ combines the segmentation accuracy SQ and the target recognition quality RQ, as shown in the following formula.

[0150]

[0151] Where TP represents a correctly matched predicted instance, FP represents an incorrectly predicted instance, and FN represents a missed true instance. IoU(p, g) represents the intersection over union (IoU) between the predicted instance p and the true instance g. The comparison results are shown in Table 4 below. RQ, SQ, and PQ represent the calculated RQ, SQ, and PQ values, respectively.

[0152] Table 4 Comparison of quantitative evaluation results of SEEM-SAFPN model and SEEM model on the test set

[0153] Model RQ SQ PQ NoC50 NoC85 NoC90 SEEM 94.039 86.611 81.448 13.72 19.27 19.75 SEEM-SAFPN 96.346 94.462 91.010 3.07 12.259 18.28

[0154] In Table 4, the number of clicks (NoC) metric is considered better. Taking NOC50 as an example, the SEEM model requires an average of 13.72 clicks to achieve 50% accuracy, while the SEEM-SAFPN model requires only 3.07 clicks. In interactive segmentation tasks, the SEEM-SAFPN model not only rapidly generates landslide area masks and corresponding labels through simple clicks or scribbles, but also supports more efficient user interactions, such as clicking or stroking on a reference image, to intelligently identify and segment regions with similar semantics in the target image. This capability is primarily due to the SEEM-SAFPN model's visual sampler and dynamic alignment technology in the joint visual-semantic space introduced during training. These techniques enable the model to unify different types of spatial queries (such as points, boxes, scribbles, polygons, and masks), thereby improving the flexibility and interactivity of segmentation tasks. A novel memory cue mechanism introduced by the SEEM-SAFPN model also allows for gradual refinement of segmentation results over multiple interactions, effectively transferring knowledge of previously generated masks to guide the optimization of the current training batch.

[0155] Figure 12 The segmentation capability of the SEEM-SAFPN model in different task scenarios is demonstrated. Figure 12 Part (a) shows that in the semantic segmentation scenario, after learning the training set, the SEEM-SAFPN model can accurately identify and segment the landslide area in a complex environment. Figure 12 Part (b) shows that in both reference segmentation and interactive segmentation scenarios, when specifying a target by clicking or scribbling, the SEEM-SAFPN model can accurately generate corresponding segmentation masks and semantic labels. In addition, the SEEM-SAFPN model also supports cross-image segmentation: after specifying a target by clicking or scribbling on another reference image (example image), SEEM can use this input to segment objects with the same semantics in other images.

[0156] The SEEM-SAFPN model of the present application not only avoids the information loss caused by semantic gaps, but also further enhances the ability to capture the boundary details and spatial distribution characteristics of the landslide area, optimizes the boundary identification of the landslide area, and effectively reduces missed detections and false detections. The SEEM-SAFPN model of the present application combines the SEEM model with the attention-enhanced progressive feature fusion strategy module (SAFPN), effectively utilizing attention to enhance the features captured at different levels while constructing a multi-level feature pyramid to solve the problem of feature information loss or degradation caused by indirect interactions between non-adjacent layers in the traditional feature pyramid network (FPN), thereby performing efficient predictions at multiple scales simultaneously. Experimental results show that the SEEM-SAFPN model of the present application has achieved significant performance improvements in landslide prediction tasks, especially showing stronger robustness when dealing with large-scale landslide structures and complex background areas.

[0157] Step 4: Use the landslide extraction model to identify the landslide area in the remote sensing image.

[0158] When identifying landslide areas, we only need to input the current remote sensing image to be identified and the corresponding text and non-text queries into the landslide extraction model, and the corresponding mask and semantic concept can be output. The mask indicates the landslide area in the remote sensing image, and the semantic concept indicates the category or statement corresponding to the predicted mask.

[0159] The landslide extraction model trained by the present invention not only inherits the advantages of the SEEM model in panoramic segmentation and multi-scale feature extraction, but also provides the model with more detailed feature information support through the attention-enhanced progressive feature fusion strategy, significantly enhancing the interaction ability between feature maps of different scales, thereby improving the accuracy of the model in identifying landslide areas in high-resolution remote sensing images.

[0160] This application compares the SEEM-SAFPN model with the original SEEM, DeepLab v3+ and FCN models in many aspects. The experimental results show that the SEEM-SAFPN model proposed in this application is effective. Specifically, the experimental results can be summarized as follows: 1) In terms of quantitative evaluation results, the experimental results show that this application, with the support of the SEEM model with the introduction of the SAFPN module, has significant improvements in key performance indicators such as aAcc, mIoU, and mDice compared to the native SEEM model and other advanced semantic segmentation models (such as FCN, DeepLab v3+, etc.). This result fully verifies the effectiveness of the landslide extraction model proposed in this application in improving model accuracy and efficiency. 2) In terms of visualization results, the SEEM-SAFPN model can effectively identify and extract landslide features in complex environments, especially when dealing with narrow or small landslide features. The model exhibits strong feature extraction capabilities and excellent edge detail restoration capabilities.

[0161] The reason for this advantage is that SAFPN shows extremely strong adaptability in this task. Due to its attention-enhanced progressive feature fusion strategy, SAFPN can effectively highlight important high-level features through SKAttention. At the same time, it integrates feature maps from different scales of SEEM through AFPN, from high-level semantic information to low-level detail information, ensuring that the characteristics of the landslide can be captured at different scales. The adaptive spatial fusion operation of ASF can dynamically adjust the weights between different feature levels and optimize the contribution of features in the fusion process. Therefore, the SEEM-SAFPN model proposed by combining the SAFPN module with the efficient panoramic segmentation network SEEM effectively solves the problem of feature information loss or degradation caused by indirect interactions between non-adjacent layers. Therefore, this application effectively improves the accuracy and efficiency of landslide identification through deep learning technology, and provides scientific support for geological disaster risk management.

[0162] In an exemplary embodiment, the present application further provides a computer device, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, memory, and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements the landslide identification method based on the SEEM-SAFPN model.

[0163] In an exemplary embodiment, the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the landslide identification method based on the SEEM-SAFPN model is implemented.

[0164] In an exemplary embodiment, the present application further provides a computer program product, including a computer program, which implements the landslide identification method based on the SEEM-SAFPN model when executed by a processor.

[0165] It will be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment method can be completed by hardware related to computer program instructions, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the process of the embodiment of the above-mentioned method. Among them, any reference to memory or other media in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0166] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0167] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A landslide identification method based on SEEM-SAFPN model, characterized in that: include: Acquire remote sensing landslide datasets and preprocess them to construct multimodal datasets; Each sample in the multimodal dataset includes a remote sensing image, a text query, a non-text query, and a corresponding mask; The SEEM model and the SAFPN module are combined to construct a SEEM-SAFPN model; the SAFPN module refers to an attention-enhanced progressive feature fusion strategy module; the SEEM-SAFPN model includes an encoder, a decoder, and a prediction head; the encoder includes a text encoder, an image encoder, a SAFPN module, and a visual sampler; The multimodal dataset is divided into training set, validation set and test set, which are used to train, validate and test the SEEM-SAFPN model respectively. The trained SEEM-SAFPN model is used as the landslide extraction model. Landslide extraction model is used to identify landslide areas in remote sensing images.

2. The landslide identification method based on the SEEM-SAFPN model according to claim 1 is characterized in that: The acquisition of remote sensing landslide dataset and preprocessing to construct a multimodal dataset specifically includes: Acquire a remote sensing landslide dataset, wherein the remote sensing landslide dataset includes a plurality of landslide images and a plurality of non-landslide images and corresponding label information; the landslide images and the non-landslide images are all remote sensing images; The landslide images are used as positive samples and the non-landslide images are used as negative samples. The remote sensing landslide dataset is cleaned according to the requirement that the positive-negative sample ratio is 1:1 to obtain the cleaned dataset. The label information of each sample in the cleaned dataset is saved in the form of structured text and converted into corresponding masks; the mask corresponding to the landslide area is called a positive mask, and the mask corresponding to the background area is called a negative mask; Imitating users in an automated manner to randomly generate text queries and non-text queries corresponding to each sample; the text queries include categories and sentences; the non-text queries include points, boxes, graffiti, polygons, and sample images; The remote sensing images, text queries, non-text queries and masks of each sample are stored one by one to construct a multimodal dataset.

3. The landslide identification method based on the SEEM-SAFPN model according to claim 2 is characterized in that: The text encoder in the encoder is used to convert the text query into a text prompt P t ; The image encoder generates multi-scale feature maps F with different resolutions and semantic information levels by performing layer-by-layer convolution and downsampling operations on the input remote sensing image. v ; The multi-scale feature map F v It includes shallow feature maps with higher resolution but less semantic information and deep feature maps with lower resolution but stronger semantic information 4. The landslide identification method based on the SEEM-SAFPN model according to claim 3 is characterized in that: The SAFPN module includes an AFPN module and an SKAttention mechanism; the AFPN module refers to an adaptive feature pyramid network module; the SKAttention mechanism refers to a selective kernel attention mechanism; The SKAttention mechanism is used to Perform adaptive adjustment of attention weights to obtain the adjusted feature map The adjusted feature map With shallow feature maps After being processed by the AFPN module, the enhanced feature map F' is obtained. v .

5. The landslide identification method based on the SEEM-SAFPN model according to claim 4 is characterized in that: The processing of the SKAttention mechanism includes: The deep feature map After convolution with kernels of sizes 3×3, 5×5, and 7×7, three feature maps U1, U2, and U3 are obtained; Based on the formula U=U1+U2+U3, the three feature maps U1, U2 and U3 are added element by element to generate a feature map U that integrates multiple receptive field information; The feature map U is embedded with global information through global average pooling. The average value is calculated along the H and W dimensions to obtain a one-dimensional vector S of dimension C×1×1 representing the importance of each channel information. H, W, and C represent the height, width, and number of channels of the feature map, respectively. The one-dimensional vector S is compressed into an L-dimensional vector Z after passing through the fully connected layer; Use the Softmax function to normalize the L-dimensional vector Z, and adjust the attention weight of each channel between each vector to obtain the attention weight vectors a, b, and d, which are applied to the weighted calculation of the feature maps U1, U2, and U3, respectively, to obtain the feature maps A1, A2, and A3; After feature selection, the feature maps A1, A2 and A3 are added element by element to obtain the adjusted feature map 6. The landslide identification method based on the SEEM-SAFPN model according to claim 5 is characterized in that: The AFPN module performs horizontal connection, downsampling, upsampling and adaptive spatial fusion operations on shallow feature maps. and adjusted feature maps The multi-scale features are fused to obtain the enhanced feature map F' v .

7. The landslide identification method based on the SEEM-SAFPN model according to claim 6 is characterized in that: The visual sampler in the encoder is based on the formula P v =VisualSampler(s,F' v ) All types of non-text queries and enhanced feature graph F' v Convert to visual cues P v ; where VisualSampler() is the operation performed by the visual sampler; s is a point, box, scribble, polygon and / or mask of the sampled area from the example image; The enhanced feature map F' v With text prompt P t , visual prompts P v and memory prompt P m Map them together into the image-text joint semantic space, and pass them to the decoder after scale alignment operation; The decoder uses a masked cross attention mechanism MaskedCrossAtt() based on the formula Generate memory reminders for the current stage in It is the memory reminder of the previous stage; M p is the mask of the previous stage; F' v is the enhanced feature map of the current stage; The decoder, on the other hand, uses a masked self-attention mechanism MaskedSelfAtt() based on the formula Output mask embedding and category embeddings where Q h represents a learnable query; The prediction head is based on the formula Infer the mask M; the mask M represents the extracted landslide area; MaskPredictor() represents the prediction operation for the mask; The prediction head is based on the formula Infer the semantic concept S; the semantic concept S represents the predicted category or sentence; ClassPredictor() represents the prediction operation for semantics.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the landslide identification method based on the SEEM-SAFPN model according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the landslide identification method based on the SEEM-SAFPN model described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the landslide identification method based on the SEEM-SAFPN model described in any one of claims 1 to 7 is implemented.

Images