Intelligent landslide identification method based on comprehensive multi-modal remote sensing image

By adopting a semantic segmentation model with a dual encoder structure in landslide recognition, combining RGB images and landslide terrain images, the problems of low accuracy and poor generalization of existing models are solved, and higher accuracy and better generalization of landslides are achieved. Identify effects.

CN118941961APending Publication Date: 2024-11-12POWERCHINA HUADONG ENG CORP LTD +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411148092.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing general deep learning semantic segmentation model only uses RGB images when recognizing landslides, ignoring the topographic properties of landslides, resulting in low recognition results accuracy and difficult for generalized models to generalize in different landslide research areas.

Method used

A landslide intelligent recognition method of integrated multimodal remote sensing images is designed, using a semantic segmentation model with a dual encoder structure, combining optical feature extraction encoder and terrain feature extraction encoder, fusing RGB images and landslide terrain images to output landslide recognition As a result, MobileNetV2 is used as the backbone network and feature fusion and prediction are performed through the fusion and decoder.

Benefits of technology

It improves the accuracy and generalization of landslide recognition, can more accurately identify the optical and topographic semantic features of landslides, and is suitable for special landslide recognition tasks, improving the generalization ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941961A_ABST
    Figure CN118941961A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of landslide identification, in particular to an intelligent landslide identification method based on a comprehensive multi-modal remote sensing image, which comprises the following steps: constructing a semantic segmentation model which comprises an optical feature extraction encoder, a topographic feature extraction encoder, a fusion device and a decoder; the optical feature extraction encoder and the topographic feature extraction encoder each comprise an image reading module. The fusion device is used for fusing the output characteristics of the two encoders; the decoder is used for converting the feature map output by the fusion device into a prediction result; and the model receives input of the landslide optical image and the landslide topographic image at the same time, and outputs a landslide identification result. By designing the structure of the double encoders and inputting the optical feature image and the topographic feature image, the landslide recognition method has the capability of reasoning based on landslide topographic information, and is high in landslide recognition precision and good in generalization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of landslide identification, and in particular to an intelligent landslide identification method integrating multi-modal remote sensing images. Background Art

[0002] Landslide identification is a very important topic in landslide research. For a new landslide study area, the first important task is often to obtain the distribution and scale information. At present, landslide identification work in landslide study areas is generally carried out by remote sensing image recognition.

[0003] The method of manual visual interpretation is highly dependent on the landslide experience of experts, and the time and energy costs are too high. Therefore, some remote sensing image landslide identification methods based on automatic thresholds have gradually emerged. For example, thresholds are automatically or semi-automatically set through reflectivity, normalized vegetation index (NDVI), slope, etc., and the landslide area and non-landslide area are binarized through the threshold. These methods are still widely used, but the problems they have are also obvious, that is, the thresholds consider a single angle and have poor generalization. They are often only applicable to a certain landslide research area and are difficult to be used universally.

[0004] With the continuous development of machine vision technology, deep learning semantic segmentation methods are often used for landslide identification. However, the existing general deep learning semantic segmentation model only uses RGB images as model input when performing landslide identification, which makes the model ignore the terrain properties of the landslide during recognition, resulting in low accuracy of landslide identification results. Summary of the invention

[0005] In order to solve the technical problem that the existing general semantic segmentation model has low accuracy in landslide identification results, the purpose of the present invention is to provide a landslide intelligent identification method based on comprehensive multimodal remote sensing images. The technical solution adopted is as follows: Constructing a semantic segmentation model, the model comprising an optical feature extraction encoder and a terrain feature extraction encoder, a fuser, and a decoder; The optical feature extraction encoder and the terrain feature extraction encoder both include an image reading module; the fusion device is used to fuse the output features of the two encoders; The decoder is used to convert the feature map output by the fusion device into a prediction result; the model simultaneously receives inputs of the landslide optical image and the landslide terrain image, and outputs a landslide identification result.

[0006] Preferably, the downsampling multiples of the optical feature extraction encoder and the terrain feature extraction encoder are equal, and both use MobileNetV2 as the backbone network.

[0007] Preferably, the image reading module included in the optical feature extraction encoder is used to read RGB images, and the image reading module included in the terrain feature extraction encoder is used to read landslide terrain images, which are three-band images composed of relative elevation, mountain shadow and slope.

[0008] Preferably, the image reading module is further used to perform normalization processing on the read RGB image or landslide terrain image, including:

[0009] Among them, T represents the normalized pixel value of the pixel point, Represents the pixel value before normalization. Represents the minimum value of all pixel values ​​before normalization in the image where the pixel is located. It represents the maximum value of all pixel values ​​before normalization in the image where the pixel is located. max() represents the maximum value function. It is the preset anti-0 parameter. The floor symbol.

[0010] Preferably, the number of channels of the features output by the two encoders is adjusted so that the sum of the number of channels of the two optical features output by the optical feature extraction encoder is greater than the number of channels of one terrain feature output by the terrain feature extraction encoder.

[0011] Preferably, the decoder consists of a convolution layer part for adjusting the number of channels, a bilinear interpolation part for adjusting the size, and a SoftMax processing module.

[0012] The embodiments of the present invention have at least the following beneficial effects: The present invention adopts a dual encoder structure, that is, a dual backbone network structure, to fuse optical features and terrain features, so that the optical semantic features and terrain semantic features of the landslide can be fully identified, and optical and terrain images are used simultaneously during input, and the terrain features are combined while considering the optical features. In addition, the landslide recognition model designed by the present invention can be specifically used for landslide recognition, and has high accuracy and good generalization for landslide recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1It is a flow chart of the steps of a landslide intelligent identification method based on integrated multi-modal remote sensing images provided by the present invention; Figure 2 It is a structural schematic diagram of the semantic segmentation model provided by the present invention; Figure 3 It is a schematic diagram of the design of the fusion device provided by the present invention; Figure 4 It is a structural schematic diagram of a computer device provided by the present invention. DETAILED DESCRIPTION

[0015] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a landslide intelligent identification method based on a comprehensive multi-modal remote sensing image proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0016] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0017] The specific scheme of the landslide intelligent identification method based on integrated multi-modal remote sensing images provided by the present invention is described in detail below with reference to the accompanying drawings.

[0018] See also Figure 1 , which shows a flowchart of a method for intelligent landslide identification based on integrated multi-modal remote sensing images provided by an embodiment of the present invention, the method comprising the following steps: Step 1: construct a semantic segmentation model, which includes an optical feature extraction encoder and a terrain feature extraction encoder, a fuser and a decoder.

[0019] This embodiment aims to design a semantic segmentation model for identifying landslides. In view of the low accuracy of the existing model that only uses RGB images as input, this embodiment adopts a dual-backbone network structure to design the architecture of the semantic segmentation model, that is, to design a dual-encoder semantic segmentation model architecture, one encoder is used to input the RGB image of the landslide, and the other encoder is used to input the terrain image related to the landslide. In this way, the semantic segmentation model constructed in this embodiment can have the ability to reason based on the landslide terrain information.

[0020] In addition, the backbone network of the model can be trained on the general VOC dataset, and then the entire model can be used to train on the landslide dataset, so that the model can quickly converge while grasping the characteristics of the landslide. Finally, a semantic segmentation model with strong generalization for landslides can be trained.

[0021] Based on this, in order to solve the problem that the general model performs poorly in landslide identification, this embodiment constructs a semantic segmentation model that integrates multimodal remote sensing images and has dual encoders, such as Figure 2 Schematic diagram of the structure of the model, including an optical feature extraction encoder and a terrain feature extraction encoder, a fusion unit and a decoder. The following embodiment specifically describes each component of the semantic segmentation model.

[0022] Step 2: The optical feature extraction encoder and the terrain feature extraction encoder both include an image reading module; the fusion device is used to fuse the output features of the two encoders.

[0023] First, in this embodiment, a common image classification network is selected, such as VGG, ShuffleNet, MobileNet. Remove its classifier module to make it a simple feature extraction backbone network. Write a dual encoder, that is, write an optical feature extraction encoder (Encoder1) and a terrain feature extraction encoder (Encoder2) based on the selected image classification network. The downsampling multiples of the two encoders are equal by adjusting the structure of the network, that is, the step size of the convolution layer and the pooling layer, or adding a new convolution layer or pooling layer.

[0024] Among them, according to the predicted difficulty of landslide identification in the data set, we can refer to the design ideas in U-Net and DeepLabV3+, and let an encoder output the shallow features and deep features extracted by the backbone network at the same time; we can add some optimizations for feature extraction to the backbone network according to some actual conditions, such as void convolution, Mamba architecture, attention mechanism, etc.

[0025] Specifically, in this embodiment, the downsampling multiples of the optical feature extraction encoder and the terrain feature extraction encoder are designed to be equal, and both use MobileNetV2 as the backbone network, such as Figure 2 As shown. As a lightweight backbone network, MobileNetV2 introduces optimizations such as dilated convolution and inverted residual, which can achieve better feature extraction effects under low parameter conditions. On the other hand, the optical feature extraction backbone network outputs low-level optical features and deep optical features at the same time. In other embodiments, ShuffleNetV2 can also be used as the backbone network.

[0026] The image reading module included in the optical feature extraction encoder is used to read RGB images, and the image reading module included in the terrain feature extraction encoder is used to read landslide terrain images. The landslide terrain image is a three-band image consisting of relative elevation, mountain shadow and slope image, where the relative elevation is band 1, the mountain shadow is band 2, and the slope image is band 3.

[0027] Furthermore, when one encoder is matched with one image reading module, the number of image channels finally read into the memory should be the same as the number of input channels required in the corresponding encoder. In this embodiment, the number of channels is 3.

[0028] At the same time, the floating point type should be normalized, and the images involved generally include: DSM images, slope images, and curvature images. Generally, for compatibility with RGB images, these floating point images should be converted to 8-bit unsigned integer images, that is, the value range is [0,255]. Here, it is necessary to establish a mapping relationship from floating point images to 8-bit unsigned integer images. The establishment of this mapping relationship depends on the data situation of the floating point type itself and the specific implementation scenario.

[0029] In this embodiment, the image reading module is also used to perform normalization processing on the read RGB image or landslide terrain image, including:

[0030] Among them, T represents the normalized pixel value of the pixel point, Represents the pixel value before normalization. Represents the minimum value of all pixel values ​​before normalization in the image where the pixel is located. It represents the maximum value of all pixel values ​​before normalization in the image where the pixel is located. max() represents the maximum value function. It is the preset anti-0 parameter. The floor symbol.

[0031] In this embodiment, the mixer is used to fuse the output features of the two encoders. The schematic diagram of its design idea is as follows: Figure 3 As shown, Figure 3 middle represents the convolution used for channel number adjustment, Represents the convolution used to achieve full feature fusion. First, a batch of convolutional layers are used to adjust the number of channels of the features output by the two backbone networks, so that the ratio of the number of channels of each part of the features becomes 48:256:256; then, two parts of optical features and one part of terrain features are stacked, and then another batch of convolutional layers are used to perform deep feature extraction. It is also worth mentioning that both batches of convolutional layers are followed by the BatchNorm-ReLU-Dropout structure to improve the generalization ability of the model and avoid overfitting.

[0032] It should be noted that in the design of the fusion device, the number of channels of the features output by the two encoders may need to be specially adjusted. The number of channels assists the neural network model in determining whether the optical semantics of the landslide or the topographic semantics of the landslide needs to be considered more. In general, the number of channels of the features output by the two encoders is adjusted so that the sum of the number of channels of the two optical features output by the optical feature extraction encoder is greater than the number of channels of one topographic feature output by the topographic feature extraction encoder, which means that the weight of the optical feature in the fusion should be higher to better capture the optical semantics; at the same time, the number of channels of the shallow optical feature is not greater than 1 / 2 of the number of channels of the deep optical feature, where the number of channels of the deep optical feature refers to the number of channels of the optical feature obtained from the deeper layers (e.g., the layers closer to the output) in the deep learning model. Usually, these layers have more channels because they need to extract more complex and abstract features. The number of channels of the shallow optical feature refers to the number of channels of the optical feature obtained from the shallower layers (e.g., the layers closer to the input) in the deep learning model.

[0033] Step three, the decoder is used to convert the feature map output by the fusion device into a prediction result; the model simultaneously receives the input of the landslide optical image and the landslide terrain image, and outputs the landslide identification result.

[0034] In this embodiment, the model also includes a decoder for fusion feature decoding, and the only work of the decoder is to convert the feature map output by the fuser into a prediction result. Therefore, the decoder consists of a convolution layer part for adjusting the number of channels, a bilinear interpolation part for size adjustment, and a SoftMax processing module.

[0035] By connecting the encoder, the fuser and the decoder in sequence, a semantic segmentation model is constructed. This model receives inputs of both the optical image of the landslide and the terrain image of the landslide, and has a strong landslide generalization ability. Furthermore, the semantic segmentation model constructed in this embodiment can be trained according to a common model training method. In this embodiment, as a specific example, the training method of the model can specifically include: using a stochastic gradient descent optimizer based on a momentum coefficient of 0.9 and a weight decay coefficient of 10-4; using a learning rate descent method with cosine annealing decay; using a cross entropy loss function. The above configuration can achieve a fast convergence training effect in most data cases.

[0036] The training dataset is a remote sensing image of the reservoir area collected by drones. After a series of processing, a total of 633 small window images of 1024×1024 size were obtained. The original dataset was randomly divided into a training set and a validation set in a ratio of 8:2, which were used for model training and reasoning validation respectively. The landslide terrain image of the terrain feature extraction backbone network is a three-band image consisting of relative elevation, mountain shadow and slope.

[0037] In addition to the semantic segmentation model of this embodiment, the landslide model and some other common SOTA semantic segmentation networks are trained at the same time, and the results of the comparative test are shown in Table 1.

[0038] Table 1 Comparison table of model training results

[0039] Among them, the intersection over union (IoU) is the ratio of the intersection and union of the predicted result and the true label for each category. And mIoU is the average of the IoU values ​​calculated for all categories. mPA represents the average pixel accuracy, which is obtained by calculating the ratio of the number of correctly classified pixels to the total number of pixels. It can be seen that when multimodal fusion is used for landslide identification, the mIoU of landslide identification using the semantic segmentation model with MobileNetV2 as the backbone network has an improvement of at least 2.2 percentage points, reflecting the important role of integrating terrain semantics for landslide identification.

[0040] The present invention also provides a landslide intelligent identification system based on integrated multimodal remote sensing images. The system and an embodiment of a landslide intelligent identification method based on integrated multimodal remote sensing images have the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0041] The present application also provides a computer device. Figure 4, which shows a schematic diagram of the structure of a computer device provided by an embodiment of the present invention, the computer device includes a memory 401, a processor 402, and a computer program 403 stored in the memory 401 and running on the processor 402, wherein when the processor 402 executes the computer program 403, the computer device can execute any one of the above-mentioned integrated multi-modal remote sensing image intelligent landslide identification methods.

[0042] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer device, the computer device can execute any one of the above-mentioned intelligent landslide identification methods based on integrated multi-modal remote sensing images.

[0043] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program code is stored. When the computer program code is executed on a computer device, the computer device can execute any one of the above-mentioned integrated multi-modal remote sensing image intelligent landslide identification methods.

[0044] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A landslide intelligent identification method based on multi-modal remote sensing images, characterized in that: The method comprises the following steps: Constructing a semantic segmentation model, the model comprising an optical feature extraction encoder and a terrain feature extraction encoder, a fuser, and a decoder; The optical feature extraction encoder and the terrain feature extraction encoder both include an image reading module; the fusion device is used to fuse the output features of the two encoders; The decoder is used to convert the feature map output by the fusion device into a prediction result; the model simultaneously receives inputs of the landslide optical image and the landslide terrain image, and outputs a landslide identification result.

2. The method for intelligent landslide identification based on integrated multimodal remote sensing images according to claim 1 is characterized in that: The optical feature extraction encoder and the terrain feature extraction encoder have the same downsampling multiples, and both use MobileNetV2 as the backbone network.

3. The method for intelligent landslide identification based on integrated multi-modal remote sensing images according to claim 1 is characterized in that: The image reading module included in the optical feature extraction encoder is used to read RGB images, and the image reading module included in the terrain feature extraction encoder is used to read landslide terrain images. The landslide terrain image is a three-band image composed of relative elevation, mountain shadow and slope.

4. The method for intelligent landslide identification based on integrated multi-modal remote sensing images according to claim 3 is characterized in that: The image reading module is also used to perform normalization processing on the read RGB image or landslide terrain image, including: Among them, T represents the normalized pixel value of the pixel point, Represents the pixel value before normalization. Represents the minimum value of all pixel values ​​before normalization in the image where the pixel is located. It represents the maximum value of all pixel values ​​before normalization in the image where the pixel is located. max() represents the maximum value function. It is the preset anti-0 parameter. The floor symbol.

5. The method for intelligent landslide identification based on integrated multi-modal remote sensing images according to claim 1 is characterized in that: The number of channels of the features output by the two encoders is adjusted so that the sum of the number of channels of the two optical features output by the optical feature extraction encoder is greater than the number of channels of one terrain feature output by the terrain feature extraction encoder.

6. The method for intelligent landslide identification based on integrated multi-modal remote sensing images according to claim 1 is characterized in that: The decoder consists of a convolution layer part for adjusting the number of channels, a bilinear interpolation part for adjusting the size, and a SoftMax processing module.

Citation Information

Cited By

  • Remote sensing image crop classification method based on Mama

    CN119418141A

  • Landslide extraction method based on multi-task learning and multi-source data feature fusion

    CN120014481A