Landslide disaster early warning methods, equipment, media and products based on deep learning

Abstract

This application discloses a landslide disaster warning method, equipment, medium, and product based on deep learning, which relates to the field of geological disaster monitoring. First, a remote sensing landslide dataset is obtained and preprocessed to construct a multimodal dataset; each sample in the multimodal dataset includes a remote sensing image, a text query, a non-text query, and a corresponding mask; a deep learning model is constructed, including an encoder, a DenseASPP module, a decoder, and a prediction head; the encoder includes an image encoder, a text encoder, and a visual sampler; the multimodal dataset is divided into a training set, a validation set, and a test set, which are used for training, validating, and testing the deep learning model, respectively, and the trained deep learning model is used as a landslide extraction model; the landslide extraction model is used to identify landslide areas in remote sensing images, and landslide disaster warnings are issued based on changes in the landslide areas, which can significantly improve the efficiency and accuracy of landslide area identification and landslide disaster warnings.

Description

Technical Field

[0001] The present application relates to the field of geological disaster monitoring technology, and in particular to a landslide disaster early warning method, equipment, medium and product based on deep learning. Background Art

[0002] Landslides are one of the world's major geological hazards. Due to the complexity of terrain background and morphology, automatic and rapid landslide identification and disaster warning are major challenges in disaster prevention and mitigation. Currently, traditional manual visual interpretation requires extensive experience and suffers from low automation, time-consuming and labor-intensive tasks, and limited scalability. Although pixel-based automated methods are widely used for landslide identification, they rely solely on spectral information and ignore the spatial characteristics of the target, significantly limiting their effectiveness.

[0003] With the development of remote sensing and geographic information system technologies, spatial information collection and analysis methods have become increasingly diverse and efficient, providing favorable conditions for remote sensing landslide identification. Remote sensing technology, with its wide coverage, diverse dimensionality, and strong real-time performance, offers irreplaceable advantages in landslide information extraction, providing strong technical support for the monitoring, assessment, and management of geological hazards. Against this backdrop, with the release of numerous meticulously annotated remote sensing landslide datasets and the rapid advancement of artificial intelligence technology from theoretical knowledge to practical applications, deep learning technology has also experienced rapid growth. Given the strong feature extraction capabilities of deep learning methods in image processing, many researchers in the remote sensing field have applied them to the task of extracting landslides from remote sensing imagery. Currently, in the task of extracting urban landslides at high spatial resolution, the spectral information of man-made objects such as landslides, roads, parking lots, and exposed spaces in high-resolution imagery is highly similar. This low inter-class variance and high intra-class variance makes accurate landslide extraction extremely difficult. This similarity makes it difficult for traditional extraction algorithms to distinguish landslides from other objects with close or overlapping edges, resulting in reduced extraction accuracy and increased false positives. Summary of the Invention

[0004] The purpose of this application is to provide a landslide disaster warning method, equipment, medium and product based on deep learning to improve the efficiency and accuracy of landslide area identification and landslide disaster warning.

[0005] To achieve the above objectives, this application provides the following solutions.

[0006] In the first aspect, the present application provides a landslide disaster early warning method based on deep learning, comprising:

[0007] Acquire a remote sensing landslide dataset and perform preprocessing to construct a multimodal dataset; each sample in the multimodal dataset includes a remote sensing image, a text query, a non-text query, and a corresponding mask;

[0008] Construct a deep learning model, which includes an encoder, a DenseASPP module, a decoder, and a prediction head; the encoder includes an image encoder, a text encoder, and a visual sampler; the DenseASPP module refers to a densely connected atrous spatial pyramid pooling module;

[0009] The multimodal dataset is divided into a training set, a validation set, and a test set, which are used to train, validate, and test the deep learning model respectively. The trained deep learning model is used as the landslide extraction model.

[0010] The landslide extraction model is used to identify landslide areas in remote sensing images, and landslide disaster warnings are issued based on changes in landslide areas.

[0011] Optionally, the acquiring of a remote sensing landslide dataset and preprocessing thereof to construct a multimodal dataset specifically includes:

[0012] Acquire a remote sensing landslide dataset, wherein the remote sensing landslide dataset includes a plurality of landslide images and a plurality of non-landslide images and corresponding label information; the landslide images and the non-landslide images are all remote sensing images;

[0013] The landslide images are used as positive samples and the non-landslide images are used as negative samples. The remote sensing landslide dataset is cleaned according to the requirement that the positive-negative sample ratio is 1:1 to obtain the cleaned dataset.

[0014] The label information of each sample in the cleaned dataset is saved in the form of structured text and converted into corresponding masks; the mask corresponding to the landslide area is called a positive mask, and the mask corresponding to the background area is called a negative mask;

[0015] Imitating users in an automated manner to randomly generate text queries and non-text queries corresponding to each sample; the text queries include categories and sentences; the non-text queries include points, boxes, graffiti, polygons, and sample images;

[0016] The remote sensing images, text queries, non-text queries and masks of each sample are stored one by one to construct a multimodal dataset.

[0017] Optionally, the image encoder is used to perform layer-by-layer convolution and downsampling operations on the input remote sensing image to generate multi-scale feature maps with different resolutions and semantic information levels. ; Multi-scale feature map The feature map with higher resolution but less semantic information is divided into the original feature map ; Multi-scale feature map The feature map with lower medium resolution but stronger semantic information is divided into the feature map to be enhanced ;

[0018] The visual sampler is used to All types of non-text queries and multi-scale feature maps Convert to visual cues ;in The operations performed for the visual sampler; masks for points, boxes, scribbles, polygons, and / or sampled regions from example images;

[0019] The text encoder is used to convert text queries into text prompts .

[0020] Optionally, the DenseASPP module is used to The feature map to be enhanced in Perform feature enhancement and fusion to obtain the enhanced feature map ; Enhance the feature map With the original feature map Together as the feature map to be mapped .

[0021] Optionally, the DenseASPP module includes a multi-scale dilated convolutional layer and a densely connected channel;

[0022] The multi-scale dilated convolution layer contains multiple 3×3 dilated convolution layers, each of which uses a different dilation rate; a 1×1 convolution layer is added before each dilated convolution layer; each dilated convolution layer is densely connected through dense connection channels; the output of each dilated convolution layer is compared with the feature map to be enhanced of the original input. After cascading, it passes through a 1×1 convolution layer to obtain the enhanced feature map .

[0023] Optionally, the feature map to be mapped With text prompt , visual cues and memory cues Map them together into the image-text joint semantic space, undergo scale alignment, and pass them to the decoder in a unified form;

[0024] The decoder uses a cross-attention mechanism with a mask , based on the formula Generate memory reminders for the current stage ;in It is a memory reminder for the previous stage; is the mask of the previous stage; is the feature map to be mapped at the current stage; ;

[0025] The decoder, on the other hand, uses a self-attention mechanism with a mask , based on the formula Output mask embedding and category embeddings ;in Indicates a learnable query.

[0026] Optionally, the prediction head is based on the formula Inferring the mask ; Mask represents the extracted landslide area; Represents the prediction operation for the mask; the prediction head is based on the formula Inferring semantic concepts ;Semantic concept Represents the predicted category or statement; Represents a semantic prediction operation.

[0027] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the landslide disaster warning method based on deep learning.

[0028] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the landslide disaster warning method based on deep learning.

[0029] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the landslide disaster warning method based on deep learning.

[0030] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0031] The present application provides a landslide disaster warning method, device, medium and product based on deep learning, in which a deep learning model is constructed including an encoder, a DenseASPP module, a decoder and a prediction head; wherein the encoder includes an image encoder, a text encoder and a visual sampler, which can realize multimodal data fusion of remote sensing images, text queries and non-text queries; the deep learning model is based on multimodal data fusion and dense multi-scale perception, which can not only perform fine-grained segmentation of global and local features at the same time, but also extract multi-scale information of images at different scales through void convolution, and enhance the feature expression ability of the model by densely connecting feature maps of different scales, thereby effectively improving the efficiency and accuracy of landslide area identification and landslide disaster warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flowchart of a landslide disaster early warning method based on deep learning is provided for this application;

[0034] Figure 2 Schematic diagram of positive and negative samples and their corresponding label information;

[0035] Figure 3 Schematic diagram of the overall framework of the deep learning model constructed for this application;

[0036] Figure 4 It is the structural diagram of ASPP module;

[0037] Figure 5 Schematic diagram of the multi-scale receptive field of the DenseASPP module;

[0038] Figure 6 This is a structural diagram of the DenseASPP module;

[0039] Figure 7 A schematic diagram of the interaction mode of query and prompt;

[0040] Figure 8 This is a visual comparison of the landslide identification results of the deep learning model and the SEEM model in this application;

[0041] Figure 9 Schematic diagram of the segmentation capability of the deep learning model in this application in different task scenarios. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] The purpose of this application is to propose a landslide disaster early warning method, equipment, medium and product based on deep learning, aiming to improve the efficiency and accuracy of landslide area identification and landslide disaster early warning.

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0045] In recent years, deep learning technology has shown significant advantages in the field of image processing. Especially in the context of the gradual improvement of computing resources, deep learning algorithms have been continuously optimized and can deeply explore the deep features in remote sensing images (also called images), such as morphology, texture, spectrum and spatial relationships. This progress enables deep learning to be effectively applied to the precise identification of landslide areas and landslide disaster warning, improving the accuracy and timeliness of landslide monitoring. On this basis, this application provides a landslide disaster warning method based on deep learning. In an exemplary embodiment, Figure 1 As shown, the landslide disaster warning method based on deep learning includes the following steps 1 to 4.

[0046] Step 1: Obtain a remote sensing landslide dataset and preprocess it to construct a multimodal dataset; each sample in the multimodal dataset includes a remote sensing image, a text query, a non-text query, and a corresponding mask.

[0047] The step 1 specifically includes the following steps 1.1 to 1.5.

[0048] Step 1.1: Obtain a remote sensing landslide dataset, which includes multiple landslide images and multiple non-landslide images and corresponding label information.

[0049] Among them, landslide images and non-landslide images are all remote sensing images. The landslide images are called positive samples, and the non-landslide images are called negative samples. Some examples of positive and negative sample images and their corresponding label information are as follows: Figure 2 The label information refers to the binary mask pixel value (0 / 255) at each pixel position in the image. The landslide area has a pixel value of 255, corresponding to the white area; the background area has a pixel value of 0, corresponding to the black area. The acquired remote sensing landslide dataset can be an existing dataset, such as the open remote sensing landslide dataset created by Wuhan University called the Bijie Landslide Dataset, or a user-created dataset.

[0050] Step 1.2: Use landslide images as positive samples and non-landslide images as negative samples. Clean the remote sensing landslide dataset to obtain the cleaned dataset, ensuring a 1:1 ratio of positive and negative samples. This means that the cleaned dataset must have a 1:1 ratio of positive and negative samples.

[0051] Step 1.3: Save the label information of each sample in the cleaned dataset in structured text format and convert it into the corresponding mask; the mask corresponding to the landslide area is called the positive mask (pm), and the mask corresponding to the background area is called the negative mask (nm).

[0052] like Figure 2 As shown, the label information for each sample in the cleaned dataset represents visual content. Therefore, the visual label information in the image needs to be stored as structured text. For example, a Python script can be used to convert the binary mask pixel values (0 / 255) of the label information into the 0 / 1 values required for model training, and use these as masks for the sample images. This process allows the label information in the image to be stored as structured text, facilitating subsequent training, inference, and evaluation.

[0053] Step 1.4: Generate text queries and non-text queries corresponding to each sample in an automated manner, imitating the user; the text queries include categories and sentences; the non-text queries include points, boxes, graffiti, polygons, and sample images.

[0054] The overall framework of the deep learning model constructed in this application is as follows Figure 3 As shown, it includes an encoder, a DenseASPP module, a decoder, and a prediction head, wherein the encoder includes an image encoder, a text encoder, and a visual sampler. Since the deep learning model inputs text queries and non-text queries at the same time as remote sensing images, including points, boxes, scribbles, polygons, and their masks / masks, etc., on the one hand, the cleaned dataset needs to be converted into the format of the COCO dataset. Specifically, the Python code can be used to convert the pixel values of 0 and 255 of the image mask into the values of 0 and 1 required for training, and create the JSON file required by the COCO dataset, which contains the name of the remote sensing image, the corresponding ID, the input text query, and the mask in text form.

[0055] That is to say, the encoder of the deep learning model of this application needs to combine text queries (such as "landslides affecting roads") and multimodal inputs such as points, boxes, graffiti, polygons, and masks / masks. Therefore, on the one hand, it is necessary to use a Python script to convert the binary mask pixel values (0 / 255) of the image into the 0 / 1 values required for training, and construct a COCO format JSON file - this file includes the name and ID of the remote sensing image, text queries, and text-structured mask annotation information (i.e., masks in text form), thereby realizing the standardized mapping of label data from pixel space to semantic space, and providing a unified multimodal data interface for model training, reasoning, and evaluation.

[0056] On the other hand, it is necessary to automatically simulate the user's random generation of text queries and non-text queries for each sample. Text queries include categories and statements. For example, categories include "slope" and "landslide"; statements include "identify landslide bodies that may pose geological hazards" and "landslide areas affecting roads." Non-text queries include points, boxes, graffiti, polygons, and example images (also called reference images). During training, non-text queries are automatically simulated and randomly generated by the user. After model training is completed, non-text queries can be manually provided by the user or randomly generated.

[0057] Step 1.5: Store the remote sensing image, text query, non-text query, and mask of each sample in a one-to-one correspondence to construct a multimodal dataset.

[0058] In the multimodal dataset constructed in this application, the remote sensing image, text query and non-text query corresponding to each sample are used as the input of the deep learning model, and the mask and semantic concept corresponding to each sample are used as the output of the deep learning model to train, verify and test the deep learning model.

[0059] Step 2: Build a deep learning model, which includes an encoder, a DenseASPP module, a decoder, and a prediction head; the encoder includes an image encoder, a text encoder, and a visual sampler.

[0060] In order to achieve more efficient and accurate landslide information extraction, this application proposes a new deep learning model, such as Figure 3 As shown in Figure 2, the deep learning model combines the advantages of multimodal data fusion and the DenseASPP module. Building on its strengths in panoptic segmentation and multiscale feature extraction, the DenseASPP module enhances the receptive field and densely connects multiscale features, enabling efficient interaction and information transfer between features at different scales, thus providing more refined feature support for the model. This design significantly improves the model's accuracy in identifying landslide areas in high-resolution remote sensing imagery.

[0061] like Figure 3 As shown in FIG, the overall framework of the deep learning model includes: 2.1) encoder; 2.2) DenseASPP module; 2.3) decoder; and 2.4) prediction head, which are described in detail below.

[0062] 2.1) Encoder.

[0063] like Figure 3As shown, the deep learning model of this application adopts an encoder-decoder architecture. The encoder of the deep learning model specifically includes: 2.1.1) image encoder; 2.1.2) text encoder; and 2.1.3) visual sampler.

[0064] 2.1.1) Image encoder.

[0065] See also Figure 3 , the input of the image encoder is a remote sensing image , the output is a multi-scale feature map The image encoder is based on the input remote sensing image Perform layer-by-layer convolution and downsampling operations to generate multi-scale feature maps with different resolutions and semantic information levels Among them, the multi-scale feature map The feature map with higher resolution but less semantic information is called the original feature map ; Multi-scale feature map The feature map with lower medium resolution but stronger semantic information is called the feature map to be enhanced. .

[0066] For example, given a size of H × W ×3 remote sensing images ,in H and W are the height and width of the remote sensing image respectively. In a specific embodiment, H = W =512. Figure 3 As shown, the remote sensing image with a size of 512×512×3 Input image encoder, which is usually based on Convolutional Neural Networks (CNN) or Transformer architecture, gradually extracts the global features of remote sensing images through multi-layer convolution or Transformer modules and generates multi-scale feature maps .in Represent the image features of the 2nd, 3rd, 4th, and 5th scales respectively (also called feature maps), Figure 3 It is expressed as The first scale usually refers to the original image features.

[0067] For example, a ResNet-based encoder progressively extracts semantic information from an image through multiple convolutional layers and residual blocks. During this extraction process, the image encoder gradually reduces the spatial resolution of feature maps through downsampling operations (such as pooling or convolutions with a stride greater than 1), while increasing the number of channels to extract deeper, global semantic information. Feature maps Res2, Res3, Res4, and Res5 correspond to the outputs of different layers in the image encoder. Feature maps at each layer are progressively generated through convolutional layers and downsampling operations. Res2 has the highest spatial resolution and the lowest number of channels, making it suitable for capturing image details such as edges and textures. Res3, with its lower spatial resolution and downsampling, captures more abstract semantic information while retaining some detail. Res4 further reduces spatial resolution and increases the number of channels, enabling it to capture higher-level semantic information, such as object shape and partial structure. Res5, with its lowest spatial resolution and the highest number of channels, is suitable for capturing global semantic information, such as the overall layout and category information of a scene. Not all of these multi-scale feature maps are directly entered into the image-text joint semantic space. Instead, some of them are enhanced by the DenseASPP module before subsequent processing.

[0068] Specifically, the shallow feature maps Res2 and Res3 contain a lot of redundant information, which is not directly helpful for the final semantic segmentation task. Blindly expanding the receptive field will only increase the model calculation pressure. Res4 and Res5 are rich in a lot of high-level semantic information and global context. Using the DenseASPP module here can further enhance the representation ability of the feature map, thereby improving the model accuracy. Therefore, the feature maps Res4 and Res5 are used as the feature maps to be enhanced. , after the DenseASPP module, the model's ability to identify landslide areas is improved by expanding the receptive field and dense connections. The feature maps Res2 and Res3 are used as the original feature maps Directly map into the joint image-text semantic space.

[0069] 2.1.2) Text encoder.

[0070] In the process of landslide identification, text queries such as categories and sentences are converted into text prompts through text encoders. .

[0071] 2.1.3) Visual Sampler.

[0072] like Figure 3 As shown, remote sensing images First, the image encoder extracts a multi-scale feature map , and then the multi-scale feature map is transformed into and all types of non-text queries (e.g., points, boxes, scribbles, polygons, and sampled regions from another example image) into visual cues , the formula is as follows:

[0073] (1).

[0074] in The action performed for the Vision Sampler. The mask corresponding to the point, box, scribble, polygon and / or sampling area from the example image (also called reference area). The visual sampler first pools the features of the corresponding area from the image features by point sampling. For all visual cues (such as points, boxes, scribbles, polygons, etc.), a maximum of 512 point feature vectors are extracted from the sampling area specified by the cues in a uniform interpolation manner. When relying only on text cues, the visual sampler can extract the features of the corresponding area from the image features. These non-text queries help disambiguate user intent when the correct segment cannot be identified.

[0075] This application implements visual cues in a unified image-text joint semantic space by jointly training panoptic segmentation and reference segmentation. With text prompt The natural alignment of the image and the surrounding text greatly improves the segmentation accuracy and the intuitiveness of user interaction. It is also flexible and scalable, making it suitable for fine-grained segmentation tasks in complex scenes.

[0076] 2.2) DenseASPP module.

[0077] Before introducing the DenseASPP module used in this application, let's first explain the basic ASPP module. ASPP (Atrous Spatial Pyramid Pooling) is a technique widely used in deep learning and computer vision, particularly in semantic segmentation tasks. It combines the concepts of dilated convolution and spatial pyramid pooling to capture multi-scale contextual information, thereby improving the model's ability to recognize objects of various sizes in an image.

[0078] like Figure 4As shown in the figure, the ASPP module mainly consists of two parts, namely the dilated convolution layer and the spatial pyramid pooling layer. Among them, dilated convolution is a special type of convolution operation that aims to expand the network receptive field without increasing the number of parameters or the amount of computation. The traditional convolution operation extracts spatial features by sliding the convolution kernel on the input feature map, where each element of the convolution kernel is simply multiplied by the corresponding element of the input feature map. In contrast, dilated convolution introduces an additional parameter - the dilation rate, which represents the spacing between adjacent elements in the convolution kernel. When the dilation rate is 1, the dilated convolution is equivalent to the standard convolution; when the dilation rate is greater than 1, the convolution kernel is "expanded" and the convolution operation covers a wider input area, but the number of parameters actually involved in the calculation remains unchanged.

[0079] Assume the input feature map is , the convolution kernel is , the void rate is , then the output of the hole convolution layer It can be calculated by the following formula:

[0080] (2);

[0081] in, Represents the position on the output feature map; Represents the relative position in the convolution kernel.

[0082] In traditional CNN models, the input image size must be fixed due to the presence of fully connected layers. This limitation requires that all images be scaled to the same size before being input to the network, which can lead to image distortion or information loss. The spatial pyramid pooling layer is designed to address this issue. It allows the network to accept input images of arbitrary size. By adding a spatial pyramid pooling layer after the convolutional layer to generate a fixed-length output vector, the image size is no longer restricted. The key to the spatial pyramid pooling layer is that it divides the output feature map of the last convolutional layer into several regions and performs a pooling operation (such as max pooling or average pooling) within each region. These regions are divided according to predefined pyramid levels, with each level having a different number of regions, forming a pyramid structure. In this way, regardless of the size of the input image, the spatial pyramid pooling layer can output a fixed-length feature vector, providing input for subsequent fully connected layers or other processing layers.

[0083] After combining the two, the ASPP module achieves the effect of pyramid pooling by applying different dilation rates on multiple parallel dilation convolution branches. Figure 4As shown, each of the multiple dilated convolution branches uses a different dilation rate, typically ranging from 1 (normal convolution), 6, 12, and 18. This aims to cover different receptive fields and capture more local information. The global average pooling layer performs global average pooling on the input feature map, then passes it through a 1×1 convolution layer and finally upsamples it to the same size as the original input feature map. This step captures global contextual information and helps improve performance. The fusion layer is responsible for fusing the outputs of all dilated convolution branches and the global average pooling results in the channel dimension. An additional 1×1 convolution layer adjusts the number of channels to produce a comprehensive feature representation. With this design, ASPP cleverly combines dilated convolution with spatial pyramid pooling, improving the model's understanding of image details and global structure. By effectively integrating multi-scale contextual information, it significantly improves segmentation accuracy and boundary clarity.

[0084] Building on ASPP, DenseASPP effectively overcomes the limitations of traditional semantic segmentation networks when dealing with objects of varying scales. While traditional convolutional networks struggle to simultaneously capture fine-grained local details and global context, DenseASPP achieves full fusion of multi-scale features by densely stacking dilated convolutional layers, thereby constructing a richer feature pyramid and significantly expanding the receptive field. Specifically, while maintaining feature map resolution, DenseASPP continuously stacks dilated convolutional layers and utilizes a dense connection mechanism, effectively overcoming the lack of responsiveness of single-layer dilated convolutions to multi-scale objects.

[0085] In traditional ASPP, dilated convolutional layers operate in parallel, and the four sub-branches do not share any information during the forward propagation. In contrast, the dilated convolutional layers in DenseASPP share information through dense connections. Layers with smaller and larger dilation rates (i.e., dilation rates) rely on each other, and the forward propagation process not only forms a denser feature pyramid but also proposes larger filters to perceive a wider context. The dilated convolutional layers are organized in a cascade structure, with the dilation rate increasing with each layer: lower dilation rates are used in lower layers, while deeper layers use higher dilation rates. The output of each dilated layer is not only connected to the input feature map of the current layer, but also concatenated with the outputs of all lower layers to form a richer feature map, which is then fed into the next layer. The final output feature map of the DenseASPP module is generated by the combination of dilated convolutions of multiple rates and scales. The DenseASPP module constructs a denser and more powerful feature pyramid with only a few dilated convolutional layers. Compared to the original ASPP, this change brings two main benefits: a denser feature pyramid and a larger receptive field.

[0086] The “denser” here not only means that the feature pyramid has better scale diversity, but also means that more pixels are involved in the convolution than ASPP. The DenseASPP module can not only sample the input at different scales, but also use dense connections to achieve a diversified integration of layers with different expansion rates. Each integration is equivalent to a kernel of a different scale, that is, a different receptive field. Therefore, a feature map with more scales than ASPP is obtained. For a layer with expansion rate and kernel size The equivalent receptive field size of the hole convolution layer is:

[0087] (3);

[0088] For example, for a 3×3 convolutional layer, when =3, the corresponding receptive field size is 7.

[0089] Stacking two convolutional layers together can give a larger receptive field. Suppose there are two convolutional layers with receptive field sizes of and , the new receptive field is:

[0090] (4).

[0091] For example, stacking a convolutional layer with a receptive field of 7 and a convolutional layer with a receptive field of 13 will result in a convolutional layer with a receptive field of 19. Taking densely stacked dilated convolutions with expansion rates (3, 6, 12, 18) as an example, the simplified structure is as follows Figure 5 As shown. Among them, the right side of each strip Represents the receptive field size of the corresponding combination. The feature pyramid generated by DenseASPP has greater scale diversity (i.e., high resolution on the scale axis) and a larger receptive field. Represents the maximum receptive field of the feature pyramid, function Represents a kernel size and void ratio The hole convolution layer of ={6, 12, 18, 24}, the maximum receptive field is:

[0092] (5).

[0093] In DenseASPP, ={6, 12, 18, 24}, the maximum receptive field is:

[0094] (6).

[0095] According to formula (4), the sum of the receptive fields of every two receptive fields stacked together is -1, so the receptive fields in formula (6) are 4, which is -3.

[0096] The DenseASPP module structure diagram is as follows Figure 6 As shown, it mainly includes: 2.2.1) multi-scale hole convolution layer and 2.2.2) dense connection channel.

[0097] 2.2.1) Multi-scale dilated convolutional layer: The DenseASPP module contains multiple 3×3 dilated convolutional layers, each with a different dilation rate. = {3, 6, 12, 18, 24}, which is used to extract multi-scale long-distance spatial information and enhance feature fusion capabilities, thereby effectively expanding the receptive field. To avoid the network structure being too wide, a 1×1 convolution layer is added before each dilated convolution layer to perform channel compression and reduce computational complexity. Essentially, dilated convolution is achieved by inserting a dilated convolution filter between standard convolution filters. The pixel values are upsampled to zero, so that the hole rate The larger the value, the wider the receptive field, thus improving the perception of large-scale targets and complex objects.

[0098] 2.2.2) Densely Connected Channels: Dense connections are introduced between different feature layers to promote feature reuse and improve information flow. This connection method can enhance the ability to express edge details and improve the accuracy and clarity of object boundaries in segmentation tasks.

[0099] Specifically for Figure 3 The deep learning model shown in Figure 2, where the input of the DenseASPP module is the feature map to be enhanced , by enhancing the feature map in the multi-scale feature map Perform feature enhancement and fusion to obtain the enhanced feature map . And the enhanced feature map With the original feature map Together as the feature map to be mapped .

[0100] So, if Figure 6 As shown, when the feature map to be enhanced After entering the DenseASPP module, it first enters a 1×1 convolution layer to achieve preliminary channel compression and feature extraction. Then, the feature map is sent to the first =3, this dilated convolution layer inserts two zero values between adjacent values of the convolution kernel to achieve sparse sampling of the dilated convolution, thereby significantly expanding the receptive field without increasing the number of parameters and effectively capturing multi-scale spatial information. The output of this layer will then be cascaded with the original input feature map to form a richer feature representation and provide a more sufficient information basis for subsequent layers. Subsequently, the cascaded feature map will sequentially enter the layers with a larger dilation rate ( =6, 12, 18, 24) of the dilated convolution layers. In each layer, the convolution kernel further expands the receptive field by inserting a corresponding number of zeros, capturing a wider range of contextual information. To prevent the network from being too wide, a 1×1 convolution layer is added before each dilated convolution layer to reduce the feature map depth to half of the original. The output of each layer is densely connected with the feature maps of all previous layers, which not only promotes feature reuse, but also enables information of different scales to complement and fuse at a deeper level. The output of each dilated convolution layer is connected to the feature map to be enhanced of the original input. After cascading, it passes through a 1×1 convolution layer and finally outputs the enhanced feature map .

[0101] 2.3) Decoder.

[0102] like Figure 3 As shown, the feature map to be mapped With text prompt , visual cues and memory cues They are mapped together into the image-text joint semantic space, and after the scale alignment operation, they are passed to the decoder in a unified form.

[0103] On the one hand, the feature map to be mapped obtained by the encoder And various tips Mapping to the image-text joint semantic space, the decoder can use the cross-attention mechanism with mask , combined with the memory hint generated during the previous segmentation , get the memory hints needed for this segmentation , the formula is as follows:

[0104] (7).

[0105] in, It is a memory reminder for the current stage; is the memory cue of the previous stage; the memory cue encodes historical information by using a mask-guided cross-attention layer. is the mask of the previous stage; is the feature map to be mapped at the current stage. In this way, the cross attention is only effective in the area specified by the previous mask. Interacts with other prompts in the decoder to convey historical information about the current round.

[0106] On the other hand, the decoder uses a self-attention mechanism with a mask , you will be able to learn to query With text, visual and memory cues Collaborative interaction, output mask embedding and category embeddings , the formula is as follows:

[0107] (8).

[0108] exist and In the " is a separator used to separate different input parameters. For example In the example, the symbol “;” is used to separate the main input parameters, i.e. the memory hints of the previous stage. and the mask of the previous stage .symbol" " is used to separate the main input parameters from the additional input parameters, such as separating the feature maps to be mapped . Represents a collection of textual cues, visual cues, and memory cues.

[0109] Although the architecture of the deep learning model in this application is simple, it also uses complex interactions between queries and prompts, such as Figure 7 As shown. You can learn to query It is a set of trainable vectors in the decoder, which is usually used to guide the model's attention mechanism to help the model extract relevant information from the input features. These query vectors are the parameters of the model and will be optimized through back propagation during the training process. During the training process, the decoder can learn query vectors. Copied as object query , text query and visual query , each task has the same weight for different segmentation tasks such as general, reference and interactive segmentation, and the corresponding prompts Freely interact with their queries via masked self-attention.

[0110] The deep learning model underlying this application serves as a unified multimodal segmentation architecture. It unifies the multi-tasks of general segmentation (semantic / instance / panoramic) and interactive segmentation through an encoder-decoder framework. It constructs a cross-modal prompt encoding system: The image encoder extracts image features from remote sensing images and outputs feature maps; the visual sampler converts points, boxes, graffiti, polygons, and example images into visual cues; and the text encoder maps semantic queries to text cues. The three are fused into an image-text joint semantic space through a parameter-sharing prompt embedding layer. The decoder, based on the cross-attention mechanism and self-attention mechanism, dynamically integrates four types of interactive cues (feature maps, text cues, memory cues, and visual cues) to achieve collaborative optimization of visual-text features. Ultimately, the unified prediction head outputs segmentation results and text information. This design breaks through the traditional segmentation model's reliance on a single input modality.

[0111] 2.4) Prediction head.

[0112] The prediction head is based on mask embedding and category embeddings Inferring the mask and semantic concepts , the formula is as follows:

[0113] (9);

[0114] (10);

[0115] in, Represents the prediction operation for the mask; Represents a semantic prediction operation. Mask Represents the extracted landslide area, that is, indicates the area where the landslide is located in the remote sensing image. Represents the predicted category or sentence, such as "landslide", "landslide area affecting roads", etc.

[0116] The deep learning model constructed in this application integrates the DenseASPP module into a multimodal segmentation architecture based on the attention mechanism, leveraging the advantages of both. The deep learning model not only simultaneously performs fine-grained segmentation of global and local features, but also extracts multi-scale information from images at different scales through dilated convolutions. By densely connecting feature maps at different scales, the network's feature representation capabilities are enhanced, thereby improving the accuracy of landslide area extraction and identification.

[0117] Step 3: Divide the multimodal dataset into training set, validation set and test set, which are used to train, validate and test the deep learning model respectively. The trained deep learning model is used as the landslide extraction model.

[0118] Wuhan University has created an open remote sensing landslide dataset, the Bijie Landslide Dataset, to support the development of automated landslide detection methods. The dataset includes satellite optical remote sensing imagery, shape files of landslide boundaries, and a digital elevation model. The remote sensing images in the Bijie Landslide Dataset consist of 770 landslide images and 2,003 non-landslide images. All images were obtained from TripleSat satellite imagery acquired between May and August 2018 and have been cropped.

[0119] In one specific embodiment, to train, validate, and test a deep learning model, a dataset of 2,773 remote sensing images was cleaned into 770 landslide images and 770 non-landslide images, with a positive-to-negative sample ratio of 1:1, for a total of 1,540 images. The cleaned landslide and non-landslide images were then divided into training, validation, and test sets at a ratio of 8:1:1. The total number of images in the training set was 1,232, the validation set was 154, and the test set was 154. The positive-to-negative sample ratio in each of the training, validation, and test sets was 1:1.

[0120] based on Figure 3 The new deep learning model architecture shown in the figure above demonstrates that during landslide training and inference, the deep learning model first converts text queries into text prompts using a text encoder. Simultaneously, a 512×512×3 remote sensing image is fed into the image encoder, extracting multi-scale feature maps of the landslide image: Res2 (high resolution, rich in detail), and Res3, Res4, and Res5 (low resolution, strong in semantic information). Feature maps Res4 and Res5 are no longer directly entered into the joint image-text semantic space; instead, they are enhanced by the DenseASPP module before subsequent processing.

[0121] Then, the shallow feature maps Res2 and Res3, along with the enhanced feature maps Res4 and Res5, are converted into visual cues through the visual sampler and all types of non-text queries. They are then mapped into the joint image-text semantic space along with textual cues, visual cues, and memory cues. After undergoing scale alignment, they are passed to the decoder in a unified form. The memory cues are derived from the decoder through cross-attention for the current stage and are used for the next training step. During the first computation, since the previous stage is not present, no memory cues are present.

[0122] Finally, the decoder interacts collaboratively with various cues based on learnable queries through self-attention and cross-attention mechanisms, outputting mask embeddings and category embeddings, thereby generating landslide segmentation results (output in the form of masks) and semantic concepts.

[0123] This application summarizes the proposed training method in the following PyTorch-style pseudocode. It should be noted that the parameters / functions in the pseudocode cannot be expressed in italics, etc., so they will be different from the parameters / function forms in the previous text. The comments after each sentence of code shall prevail, and the comments of the corresponding code shall start with the symbol "#".

[0124] The pseudo code of the deep learning model training process is as follows:

[0125] Input: remote sensing image img: [B, 3, H, W]; positive mask pm and negative mask nm: [B, 1, H, W]; text query: txt [a, b, c......]; #B represents the batch size, H and W represent the height and width, and the positive and negative masks can come from remote sensing image labels and non-text queries

[0126] Variables: learnable query Qh; self-attention mask msa between query Q and prompt P;

[0127] Function: Image encoder: Img_Encoder(), Text encoder: Text_Encoder(), Visual sampler: Visual_Sampler(), DenseASPP module: DenseASPP(), Cross attention mechanism: feature_attn(), Self attention mechanism: prompt_attn(), Output: output();

[0128] def init(): #initialization operation

[0129] Qo, Qt, Qv = Qh.copy(); #Initialize object query Qo, text query Qt, and visual query Qv

[0130] Fv, Pt = img_Encoder(img), Text_Encoder(txt); #Fv and Pt represent image features (feature maps) and text prompts respectively

[0131] Fv[4],Fv[5]= DenseASPP(Fv[4]), DenseASPP(Fv[5]); #Fv[4],Fv[5] represent the image features of the 4th and 5th scales output by the image encoder

[0132] Pv = Visual_Sampler(Fv, pm, nm); # Sample visual cues Pv from image features, positive and negative masks

[0133] def Deep_Learning_Model_Decoder(Fv, Qo, Qt, Qv, Pv, Pt, Pm): #Define the decoder of the deep learning model

[0134] Qo, Qt, Qv = feature_attn(Fv, Qo, Qt, Qv,); #Cross-attention calculation of query and image features

[0135] Qo, Qt, Qv = prompt_attn(msa, Qo, Qt, Qv, Pv, Pt, Pm); #Calculate self-attention for query and prompt, Pm represents memory prompt

[0136] Om, Oc, Pm = output(Fv, Qo, Qt, Qv, Pv, Pt, Pm) #Calculate mask Om and category (semantic concept) output Oc

[0137] def forward(img, pm, nm, txt): #training forward propagation

[0138] Fv, Qo, Qt, Qv, Pv, Pt = init(); Pm = None; # Initialize variables

[0139] for i in range(max_iter): #Start iteration

[0140] Om, Oc, Pm = Deep_Learning_Model_Decoder(Fv, Qo, Qt, Qv, Pv, Pt, Pm); #Decoder operation.

[0141] The positive mask pm refers to the landslide area mask, and the negative mask nm refers to the background area mask. The learnable query Qh is a trainable parameter that is randomly initialized during model construction and automatically learns the most appropriate representation through a data-driven approach throughout the training process. msa refers to the self-attention mask between the query Q (e.g., Qo, Qt, Qv) and the prompt P (e.g., Pv, Pt, Pm). It is converted into a weight matrix by calculating the similarity between Q and P. Matches below a certain threshold are weakened or even blocked, forming a binary or continuous-valued mask matrix, thereby suppressing irrelevant or invalid information interactions.

[0142] The deep learning model was constructed and trained using the PyTorch 1.8.1 framework, using CUDA 10.2.0 for GPU acceleration. All comparative experiments were trained for 50 epochs, with a batch size of 4 and an initial learning rate of 6e-05. A polynomial decay strategy was used to adjust the learning rate, and a learning rate warmup mechanism was introduced. At the beginning of training, the learning rate was gradually increased from a low value to the preset initial value. In the early stages of training, the learning rate was gradually increased from a low value to the set initial learning rate, with the warmup period showing a linear increase in the learning rate. The minimum value of the learning rate decay was never less than 1.00000000000000002e-06. The specific training parameters are shown in Table 1 below.

[0143] Table 1 Experimental parameters and training settings

[0144]

[0145] In the landslide identification task, the landslide area often has different spatial scales and morphological changes. The deep learning model can effectively overcome this problem through its unique structure, thereby better capturing multi-scale information and significantly improving the recognition accuracy. In order to verify the effectiveness of the deep learning model proposed in this application, experiments were conducted on the Bijie landslide dataset of Wuhan University. The dataset was divided into training set, validation set and test set in a ratio of 8:1:1, and the positive and negative sample ratio was guaranteed to be 1:1. The deep learning model was trained and verified using the training set and validation set, and the performance of the improved deep learning model was evaluated on the test set. The parameters of the trained deep learning model were encapsulated and used as a landslide extraction model when performing landslide identification.

[0146] The trained Deep Learning Model (DLM) is tested using the test set. The visualization results obtained on the test set are compared with the large visual model SEEM (Segment Everything Everywhere All at Once). The results are as follows: Figure 8 As shown. Figure 8 The results show that the deep learning model constructed in this application outperforms the SEEM model in landslide identification. The newly proposed deep learning model not only generates more realistic landslide edges, but also significantly improves the recognition accuracy and stability in complex environments.

[0147] Furthermore, we used global average accuracy (aAcc), mean F-score (mFscore), mean precision (mPrecision), mean recall (mRecall), mean intersection over union (mIoU), and mean Dice coefficient (mDice) to compare with traditional FCN, DeepLab v3+, SegFormer, SEEM, and other models. aAcc reflects the model's ability to correctly classify pixels across the entire image, and is the ratio of correctly classified pixels to the total number of pixels. The formula is as follows:

[0148] (11);

[0149] in, is the total number of categories, Indicates the The number of categories correctly identified as positive classes, Indicates the The number of categories correctly identified as negative classes, is the total number of positive classes. is the total number of actual negative classes. is the calculated global average accuracy (aAcc) value.

[0150] mFscore, mPrecision, and mRecall are a comprehensive measure of the accuracy and completeness of the model's ability to identify specific categories. mPrecision is the ratio of correctly predicted positives (TP) to all predicted positives (TP + FP), as shown in Formula (12). mRecall is the ratio of correctly predicted positives to actual positives, as shown in Formula (13). mFscore is the harmonic mean of precision and recall, as shown in Formula (14).

[0151] (12);

[0152] (13);

[0153] (14).

[0154] in, Indicates the The number of categories correctly identified as positive classes, Indicates the The number of categories correctly identified as negative classes, Indicates that other categories are incorrectly identified as The number of categories, Indicates the The number of categories that were incorrectly identified as other categories. 、 and They are the calculated average precision (mPrecision) value, average recall (mRecall) value, and average F score (mFscore) value respectively.

[0155] mIoU and mDice evaluate the overlap between the predicted and true regions and are key metrics in the field of semantic segmentation. mIoU is the ratio of the intersection of the predicted positive class and the actual positive class to their union, as shown in formula (15). The mDice coefficient is the ratio of twice the intersection of the predicted positive class and the actual positive class to the sum of their respective numbers, similar to the F score, as shown in formula (16).

[0156] (15);

[0157] (16);

[0158] in, and They are the calculated mean intersection-over-union (mIoU) values and the average Dice coefficient values, respectively.

[0159] These metrics were used to evaluate the performance of the trained deep learning model (the landslide extraction model) and compared with selected visualization results from the test set. This demonstrated the effectiveness and stability of this deep learning model for high-resolution urban landslide extraction. Table 2 shows a comparison of the quantitative evaluation results of different models on the test set.

[0160] Table 2 Quantitative evaluation results of different models on the test set

[0161]

[0162] The results shown in Table 2 show that the deep learning model (DLM) of this application outperforms the existing models in terms of indicators such as aAcc, mFscore, mPrecision, mRecall, mIoU and mDice compared with the traditional FCN, DeepLab v3+, SegFormer and SEEM models, indicating the effectiveness and stability of the deep learning model of this application in the landslide extraction task.

[0163] To comprehensively evaluate the performance of the landslide extraction model proposed in this application, in addition to semantic segmentation metrics, key evaluation metrics such as panoptic quality (PQ), segmentation quality (SQ), and recognition quality (RQ) were used to assess the recognition capabilities of the deep learning model (DLM) and SEEM model. For interactive segmentation, the number of clicks (NoC) metric was used to evaluate interactive segmentation performance. This metric measures the number of clicks required to achieve a certain intersection over union (IoU) of 50%, 85%, and 90%, denoted as NoC50, NoC85, and NoC90, respectively. These metrics not only reflect the model's accuracy and robustness in image segmentation tasks but also comprehensively measure its overall capabilities in object recognition and region segmentation. SQ focuses on the segmentation accuracy of the target region; RQ calculates the recognition accuracy of the target, determining whether each target was correctly identified; and PQ combines SQ and RQ, as shown below.

[0164] (17);

[0165] (18);

[0166] (19);

[0167] in, represents the correctly matched prediction instance, represents an incorrect prediction instance, Indicates a real instance of missed detection. Represents a prediction instance With real examples The intersection and union ratio. 、 and The calculated RQ, SQ and PQ values are shown in Table 3 below.

[0168] Table 3 Comparison of quantitative evaluation results of deep learning model and SEEM model on the test set

[0169]

[0170] In Table 3, the smaller the number of clicks (NoC), the better. Taking NOC50 as an example, the SEEM model requires an average of 13.72 clicks to achieve 50% accuracy, while the deep learning model (DLM) proposed in this application only requires an average of 8.59 clicks. In interactive segmentation tasks, the deep learning model proposed in this application not only rapidly generates landslide area masks and corresponding labels through simple clicks or scribbles, but also supports more efficient user interaction, such as clicking or stroking on a reference image to intelligently identify and segment regions with similar semantics in the target image. This capability is primarily due to the visual sampler and dynamic alignment technology of the joint visual-semantic space introduced during the deep learning model's training process. These techniques enable the model to unify different types of spatial queries (such as points, boxes, scribbles, polygons, and masks), thereby improving the flexibility and interactivity of the segmentation task. A novel memory cue mechanism proposed in the deep learning model also allows for gradual refinement of segmentation results over multiple interactions, effectively transferring previously generated mask knowledge to guide the optimization of the current training batch.

[0171] Figure 9 The segmentation capability of the deep learning model in this application is demonstrated in different task scenarios. Figure 9 Part (a) shows that after learning the training set, the deep learning model can automatically identify the landslide body (landslide area) and accurately segment the mask boundary without additional prompts when inputting remote sensing images. Figure 9 Part (b) demonstrates the intent-driven interactive segmentation capability of the deep learning model of this application. When provided with click or graffiti prompts, the deep learning model can accurately extract the landslide area based on the intention of the user input. Figure 9 Part (c) demonstrates the cross-scenario reference segmentation capability of this application's deep learning model. Experts can annotate typical features on historical landslide remote sensing images (reference images). The deep learning model, based on a joint spatial matching method, identifies regions with similar visual semantics in the target image, thereby enabling zero-shot landslide analogy recognition. This capability significantly reduces the annotation cost of new scenes and improves the model's generalization and applicability.

[0172] Step 4: Use the landslide extraction model to identify the landslide area in the remote sensing image, and issue landslide disaster warning based on the changes in the landslide area.

[0173] To identify landslide areas, the landslide extraction model simply inputs the remote sensing image to be identified, along with the corresponding text and non-text queries. This model then outputs a corresponding mask and semantic concepts. The mask precisely outlines the landslide area in the remote sensing image, while the semantic concepts indicate the category or sentence corresponding to the predicted mask. By analyzing time-series images, the spatial distribution and morphological changes (e.g., location, shape, and area) of landslides within the same area over time can be monitored. Combined with a dual-indicator early warning method based on deformation velocity and deformation area, this approach enables real-time, rapid, and accurate early warning of landslide hazards.

[0174] The landslide extraction model trained by the present invention, based on multimodal data fusion and dense connection of multi-scale features, can not only simultaneously perform fine-grained segmentation of global and local features, but also extract multi-scale information of images at different scales through void convolution. By densely connecting feature maps of different scales, the network's feature expression capability is enhanced, thereby improving the accuracy and efficiency of landslide identification. In addition, to verify the accuracy of the model, a large number of effectiveness verification experiments were conducted on a public landslide dataset from Wuhan University and compared with various advanced extraction models. The experimental results show that the deep learning model proposed in this application is effective. Specifically, the experimental results can be summarized as follows: 1) In terms of quantitative evaluation, the deep learning model proposed in this application has significant improvements in multiple key performance indicators (such as aAcc, mIoU, and mDice) compared with mainstream semantic segmentation models (such as SEEM, FCN, DeepLab v3+, etc.). 2) In terms of visualization results, the deep learning model proposed in this application can effectively identify and extract landslide features in complex environments. Especially when dealing with small-scale landslide features, the deep learning model of this application demonstrated powerful feature extraction capabilities and excellent edge detail recovery performance, further verifying its superior performance in practical applications.

[0175] This advantage stems from the introduction of the ASPP module, which cleverly combines dilated convolution with spatial pyramid pooling, enhancing the deep learning model's ability to perceive image details and global structure. The use of dilated convolution allows the network to expand its receptive field without increasing computational effort, capturing a wider range of contextual information. Spatial pyramid pooling, through multi-scale processing, further enhances the model's feature extraction capabilities at different scales. As a result, the ASPP module not only enhances sensitivity to detail but also effectively incorporates multi-scale contextual information, significantly improving the segmentation accuracy and boundary clarity of landslide areas. DenseASPP, building on this, incorporates the design principles of DenseNet and fully leverages its efficient information flow and gradient propagation properties. Dense connections improve feature utilization, enabling each layer to receive information from all previous layers, thus avoiding information loss and vanishing gradients. The resulting deep learning model, combining the DenseASPP module with multimodal data fusion, demonstrates superior performance in landslide detection tasks. Therefore, this application effectively improves the accuracy and efficiency of landslide area identification and landslide disaster warning through deep learning technology, providing scientific support for geological disaster risk management.

[0176] In an exemplary embodiment, the present application also provides a computer device, which can be a server or a terminal. The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the landslide disaster early warning method based on deep learning is implemented.

[0177] In an exemplary embodiment, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the landslide disaster early warning method based on deep learning.

[0178] In an exemplary embodiment, the present application also provides a computer program product, including a computer program, which implements the landslide disaster early warning method based on deep learning when executed by a processor.

[0179] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by hardware associated with computer program instructions. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory or other media in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, and the like. Volatile memory may include random access memory (RAM) or external cache memory, and the like. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0180] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0181] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A landslide disaster early warning method based on deep learning, characterized in that: include: Acquire remote sensing landslide datasets and preprocess them to construct multimodal datasets; Each sample in the multimodal dataset includes a remote sensing image, a text query, a non-text query, and a corresponding mask; Construct a deep learning model, which includes an encoder, a DenseASPP module, a decoder, and a prediction head; the encoder includes an image encoder, a text encoder, and a visual sampler; the DenseASPP module refers to a densely connected atrous spatial pyramid pooling module; The input of the image encoder is the remote sensing image , the output is a multi-scale feature map ; Multi-scale feature map The feature map with higher resolution but less semantic information is called the original feature map ; Multi-scale feature map The feature map with lower medium resolution but stronger semantic information is called the feature map to be enhanced. ; Convert text queries into text prompts via a text encoder ; Remote sensing images First, the image encoder extracts a multi-scale feature map , and then the multi-scale feature map is transformed into and all types of non-text queries are converted to visual cues ; Enabling visual cues in a unified joint image-text semantic space via joint training of panoptic and reference segmentation With text prompt natural alignment of The input of the DenseASPP module is the feature map to be enhanced , by enhancing the feature map in the multi-scale feature map Perform feature enhancement and fusion to obtain the enhanced feature map ; And the enhanced feature map With the original feature map Together as the feature map to be mapped ; Feature map to be mapped With text prompt , visual cues and memory cues Map them together into the image-text joint semantic space, undergo scale alignment, and pass them to the decoder in a unified form; The decoder uses a self-attention mechanism with a mask , you will be able to learn to query With text, visual and memory cues Collaborative interaction, output mask embedding and category embeddings ; The prediction head is based on mask embedding and category embeddings Inferring the mask and semantic concepts ; Mask Represents the extracted landslide area, semantic concept Represents the predicted category or statement; The multimodal dataset is divided into a training set, a validation set, and a test set, which are used to train, validate, and test the deep learning model respectively. The trained deep learning model is used as the landslide extraction model. The landslide extraction model is used to identify landslide areas in remote sensing images, and landslide disaster warnings are issued based on changes in landslide areas.

2. The landslide disaster early warning method based on deep learning according to claim 1 is characterized in that: The acquisition of remote sensing landslide dataset and preprocessing to construct a multimodal dataset specifically includes: Acquire a remote sensing landslide dataset, wherein the remote sensing landslide dataset includes a plurality of landslide images and a plurality of non-landslide images and corresponding label information; the landslide images and the non-landslide images are all remote sensing images; The landslide images are used as positive samples and the non-landslide images are used as negative samples. The remote sensing landslide dataset is cleaned according to the requirement that the positive and negative sample ratio is 1:1 to obtain the cleaned dataset. The label information of each sample in the cleaned dataset is saved in the form of structured text and converted into corresponding masks; the mask corresponding to the landslide area is called a positive mask, and the mask corresponding to the background area is called a negative mask; Imitating users in an automated manner to randomly generate text queries and non-text queries corresponding to each sample; the text queries include categories and sentences; the non-text queries include points, boxes, graffiti, polygons, and sample images; The remote sensing images, text queries, non-text queries and masks of each sample are stored one by one to construct a multimodal dataset.

3. The landslide disaster early warning method based on deep learning according to claim 2 is characterized in that: The image encoder is used to perform layer-by-layer convolution and downsampling operations on the input remote sensing image to generate multi-scale feature maps with different resolutions and semantic information levels. ; Multi-scale feature map Divided into original feature maps and the feature map to be enhanced ; The visual sampler is used to All types of non-text queries and multi-scale feature maps Convert to visual cues ;in The operations performed for the visual sampler; Masks for points, boxes, scribbles, polygons, and / or sampled regions from example images.

4. The landslide disaster early warning method based on deep learning according to claim 3 is characterized in that: The DenseASPP module includes multi-scale dilated convolutional layers and densely connected channels; The multi-scale dilated convolution layer contains multiple 3×3 dilated convolution layers, each of which uses a different dilation rate; a 1×1 convolution layer is added before each dilated convolution layer; each dilated convolution layer is densely connected through dense connection channels; the output of each dilated convolution layer is compared with the feature map to be enhanced of the original input. After cascading, it passes through a 1×1 convolution layer to obtain the enhanced feature map .

5. The landslide disaster early warning method based on deep learning according to claim 3 is characterized in that: The decoder uses a cross-attention mechanism with a mask , based on the formula Generate memory reminders for the current stage ;in It is a memory reminder for the previous stage; is the mask of the previous stage; is the feature map to be mapped at the current stage; ; The decoder, on the other hand, uses a self-attention mechanism with a mask , based on the formula Output mask embedding and category embeddings ;in Indicates a learnable query.

6. The landslide disaster early warning method based on deep learning according to claim 5 is characterized in that: The prediction head is based on the formula Inferring the mask ; Mask represents the extracted landslide area; Represents the prediction operation for the mask; the prediction head is based on the formula Inferring semantic concepts ;Semantic concept Represents the predicted category or statement; Represents a semantic prediction operation.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the landslide disaster warning method based on deep learning according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the landslide disaster early warning method based on deep learning described in any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the landslide disaster early warning method based on deep learning described in any one of claims 1 to 6 is implemented.

Images