Landslide Risk Assessment Method Based on Multimodal Learning

Abstract

The present invention discloses a landslide risk assessment method based on multimodal learning, which mainly solves the problem that the existing methods cannot process multimodal data, thus resulting in low accuracy of landslide risk assessment. The implementation scheme is as follows: extracting the digital elevation model corresponding to the high-resolution remote sensing sub-map; performing semantic segmentation on the high-resolution remote sensing sub-map to extract the edges of landslide risk points; calculating the slope map and aspect map corresponding to the extracted digital elevation model; extracting the respective corresponding feature sequences from the slope map and aspect map; performing residential area detection and road detection on the high-resolution remote sensing sub-map to be evaluated; performing image embedding on the high-resolution remote sensing sub-map, slope map, and aspect map; performing sequence embedding on the feature sequences; performing point embedding on the detection results of residential areas and roads; and inputting the embedded data into a visual self-attention model to obtain the landslide risk assessment level. The present invention improves the assessment accuracy of landslide risk points and can be used for early warning of geological disasters.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image interpretation, and particularly relates to a landslide risk assessment method, which can be used to evaluate the disaster risk level of landslide risk points and give geological disaster warnings. Background Art

[0002] For a long time, many related technologies have been applied to pre-disaster warnings and post-disaster detections of landslides. However, there are still many challenges. A large amount of work needs to be completed manually, and accurate and sufficient expert knowledge is required to ensure the accuracy and scientific nature of the evaluation results. In recent years, with the development of computer technology and the perfection of automation technology, more and more experts and scholars have incorporated automation knowledge into relevant research on landslides, liberating more technical experts from dangerous and cumbersome field investigations. Current landslide risk assessment models are more often based on a series of complex means such as field measurements, precise modeling, and simulation reasoning to obtain a relatively accurate assessment model. However, since these means require the participation of professionals, it is difficult to deploy and evaluate them on a large scale. Although some researchers have put the automated detection of landslides on the agenda, most of the research only qualitatively analyzes landslides through a single feature, and there is relatively little research on automated assessment models with multiple features, multi-modalities, and multiple manifestation forms.

[0003] In terms of specific landslide grade classification, in 2011, the Ministry of Natural Resources of the People's Republic of China issued a standard for classifying landslides by volume. The existing landslide grade classification divides landslides into small-scale landslides, medium-scale landslides, large-scale landslides, and extra-large-scale landslides according to the volume of the landslide, which provides guidance for the grade classification of landslides. However, for risk points where landslides have not occurred, it is impossible to directly extract the landslide volume information by measuring the volume of the landslide. Therefore, currently, area-volume mapping is mainly used to quantify the risk assessment of landslides. However, the effect of this assessment only based on area is not good. The main reason is that some parameters required in the area-volume mapping of landslides are based on the statistical results of landslides in a specific area, and there may be a large difference in statistical distribution from landslides in other areas due to differences in terrain and landform, resulting in a low assessment accuracy for landslide risk points in other areas. In addition, for the risk assessment of landslides, area is not the only evaluation criterion. In addition to area and volume, the grade classification of landslides will also involve many factors to a great extent, such as the area affected after the landslide occurs, the facilities affected, the slope direction of the landslide, the slope gradient of the landslide, etc.

[0004] Considering that single-modal data cannot well describe various characteristics of landslides, some experts and scholars have applied multi-modal learning to the field of remote sensing image interpretation. Related research based on multi-modal learning can model independent parts of information within multiple modalities, enabling the model to make greater use of information originally belonging to multiple independent modalities during the process of processing information and fuse it in a higher semantic space to improve problems such as insufficient feature and weak representation ability of traditional single-modal information. Chen et al. proposed a fusion method combining a variational model and a deep convolutional neural network in "Spatial-spectral fusion by combining deep learning and variational model" aiming to fuse the detailed information of high-resolution panchromatic data and the low-resolution hyperspectral information to obtain a hyperspectral image with more details and use it for the representation of multi-modal tasks. Liu et al. combined the information of two modalities, namely RGB high-resolution remote sensing images and digital elevation model (DEM), extracted texture information from high-resolution remote sensing images and topographic information of landslides from the digital elevation model respectively, and then fused these two types of features at a level with higher semantic features, finally achieving a good segmentation effect.

[0005] In the above-mentioned literature, in the processing of multi-modal information, the utilization of multi-modal information is achieved by processing the original data. This form of data processing can well complete the learning of implicit knowledge in the case of a large amount of data to improve the evaluation accuracy of landslide risk points. However, due to the lack of high-quality labeled data for landslide risk points, it is difficult for the autonomous learning ability of the model to learn deeper hidden features, including the slope, aspect, bearing point, etc. of landslide risk points, thus resulting in a relatively low evaluation accuracy of landslide risk levels. Summary of the Invention

[0006] The purpose of the present invention is to propose a landslide risk assessment method based on multi-modal learning to learn deeper hidden features and improve the evaluation accuracy of landslide risk levels in view of the problems existing in the above-mentioned multi-modal learning tasks.

[0007] The technical idea for achieving the object of the present invention is as follows: By using prior knowledge to extract the slope map and aspect map of the landslide, and at the same time extracting key targets such as buildings and roads around the landslide, the learning difficulty in the autonomous learning process of the model is reduced; by integrating the slope-related features of the landslide risk points obtained and the features of the surrounding key targets, the feature content in the evaluation process of the landslide risk points is enriched; and the final evaluation result is output through a visual self-attention model for result prediction.

[0008] According to the above idea, the implementation solution of the present invention includes the following steps:

[0009] (1) Extract the required high-resolution remote sensing sub-image IMG from the high-resolution remote sensing image, and extract the height data corresponding to all the high-resolution remote sensing sub-images IMG from the digital elevation model to form a regional digital elevation model DEM corresponding one-to-one with the high-resolution remote sensing sub-image IMG;

[0010] (2) Perform semantic segmentation on the high-resolution remote sensing sub-image IMG for landslide risk points to obtain the segmentation result R seg , and combine it with the geographic coordinate projection information in the high-resolution remote sensing image to obtain the edge E of the landslide risk points;

[0011] (3) Use the regional digital elevation model DEM to calculate the original slope map S′ and the original aspect map A corresponding to the high-resolution remote sensing sub-image IMG respectively ′ , and crop them according to the edge E of the landslide risk points to obtain the subsequent available slope map S and aspect map A;

[0012] (4) Perform 16×16 max pooling on the slope map S to obtain the feature sequence F corresponding to the slope map S S ;

[0013] (5) Perform 16×16 average pooling on the aspect map A to obtain the feature sequence F corresponding to the aspect map A A ;

[0014] (6) Detect settlements in the high-resolution remote sensing sub-image IMG to obtain the detection result R b , and mark the R with settlements b as 1, and the R without settlements b as 0;

[0015] (7) Detect roads in the high-resolution remote sensing sub-image IMG to obtain the detection result R r , and mark the R with roads r as 1, and the R without roads r as 0;

[0016] (8) For the semantic segmentation result R of the above-mentioned landslide risk points segPerform embedding operations on the slope map S and aspect map A of the image data to obtain the relevant features F of the image data img ;

[0017] (9) Perform sequence data embedding operations on the feature sequence F of the above slope map S and the feature sequence F of the aspect map A to obtain the relevant features F of the sequence data seq ;

[0018] (10) Embed the detection results R of the above residential areas b and the detection results R of the roads r to obtain the relevant features F of the point data poi ;

[0019] (11) Input the above three relevant features F img , F seq and F poi into the existing visual self-attention model for evaluation to obtain the landslide risk level of the landslide risk points

[0020] Compared with the existing methods, the present invention has the following advantages

[0021] First, by improving the usage form of the digital elevation model DEM, the present invention extracts the slope map, aspect map and their respective corresponding feature sequences in the regional digital elevation model DEM based on prior knowledge, reducing the difficulty of the model in autonomously learning the hidden knowledge in the digital elevation model DEM

[0022] Second, by extracting information such as residential areas and roads near the landslide risk points, the present invention increases the feature abundance of the available features in the landslide disaster risk assessment, enabling the model to utilize not only the features of the slope itself but also the information of the surrounding key targets

[0023] Third, the present invention proposes a feature embedding form for sequence data and point data, providing the possibility for the utilization of different forms of data such as image data, sequence data, and point data, and expanding the input format of the evaluation model Description of the Drawings

[0024] Figure 1 is the implementation flowchart of the present invention

[0025] Figure 2 is the schematic diagram of pixel distribution in the present invention

[0026] Figure 3 is the simulation diagram of the detection result of residential areas in the present invention

[0027] Figure 4 is the simulation diagram of the detection result of roads in the present invention Detailed implementation manners

[0028] The following further elaborates on the specific embodiments and effects of the present invention in conjunction with the accompanying drawings.

[0029] Referring to Figure 1 , the landslide risk assessment method based on multimodal learning in this example is implemented as follows:

[0030] Step 1: Extract the required high-resolution remote sensing sub-images IMG from the high-resolution remote sensing images, and extract the altitude data corresponding to all the high-resolution remote sensing sub-images IMG from the digital elevation model to form a regional digital elevation model DEM corresponding one-to-one with the high-resolution remote sensing sub-images IMG.

[0031] The high-resolution remote sensing images are the original data, and the corresponding digital elevation model data plays an auxiliary role in the detection of landslide risk points. In order to better utilize the digital elevation model data, in this example, the digital elevation model is cropped according to the corresponding relationship between geographical coordinate systems. At the same time, since there are significant differences in the altitude data represented by the DEM even for the same terrain in different regions and between different regions, but this has no impact on whether a landslide will occur, so to eliminate the altitude differences between regions, for each DEM point data on the DEM corresponding to each image, the minimum value of each DEM data needs to be subtracted. The specific implementation is as follows:

[0032] 1.1) Crop and extract the required part from the high-resolution remote sensing images to obtain N high-resolution remote sensing sub-images IMG, where N>100;

[0033] 1.2) Select one high-resolution remote sensing sub-image IMG from the extracted high-resolution remote sensing sub-images IMG, and read the geographical coordinate information it contains, as well as the width w and height h of this high-resolution remote sensing sub-image IMG;

[0034] 1.3) Map the geographical coordinate information obtained in 1.2) to the digital elevation model to obtain the upper left corner coordinate information (x, y) of the position to be cropped;

[0035] 1.4) Crop the area in the digital elevation model with (x, y) as the upper left corner and (x+w, y+h) as the lower right corner according to the information size obtained in 1.2) and the information coordinates obtained in 1.3) as the regional digital elevation model DEM corresponding one-to-one to the high-resolution remote sensing sub-image IMG described in 1.1);

[0036] 1.5) Calculate the mapped height h of each pixel in the regional digital elevation model DEM image ′ (i, j):

[0037] h ′ (i, j)=h(i, j)-min(h(i, j))

[0038] where h(i, j) is the value at the corresponding coordinate position (i, j) in the digital elevation model, and min is the function to take the global minimum value;

[0039] 1.6) Integrate the mapped height h′(i, j) of each pixel to obtain the height data corresponding to the high - resolution remote sensing sub - image IMG.

[0040] Step 2, perform semantic segmentation on the high - resolution remote sensing sub - image IMG to obtain the segmentation result R of the landslide risk points seg , and combine it with the geographic coordinate projection information in the high - resolution remote sensing image to obtain the edge E of the landslide risk points.

[0041] In order to improve the accuracy of the overall evaluation model for the original data after adding the regional digital elevation model DEM, in this example, the high - resolution remote sensing sub - image IMG is processed through semantic segmentation to obtain the position and shape information of the landslide risk points, which are stored in the form of a mask, denoted as Mask. Due to the uncertainty of the segmentation task, in addition to the main Mask, a series of small noise masks Mask′0, Mask′1,... are usually generated. Therefore, screening is required according to the area size. Then, the four upper, lower, left, and right edges E seg of the semantic segmentation result R u 、E d 、E l 、E r are obtained through morphological processing, and the four edges are integrated to obtain the edge E. The specific implementation is as follows:

[0042] 2.1) Extract the mask Mask of the landslide risk points and a series of noise masks Mask′0, Mask′1,... through semantic segmentation;

[0043] 2.2) Traverse all the masks and delete the masks with an area smaller than the set threshold. In this example, the threshold is set to 100;

[0044] 2.3) Denote the number of remaining masks as M. Traverse the remaining M masks and calculate the four upper, lower, left, and right edges of each mask where the value of j is 1, 2, 3,..., j,..., M;

[0045] 2.4) Traverse the M upper edges to obtain the minimum value E u of the M upper edges. Similarly, obtain the other three edges e d 、E l 、E r ;

[0046] 2.5) Integrate the above four edges to obtain the edge E of the landslide risk points.

[0047] Step 3: Use the regional digital elevation model (DEM) to calculate the corresponding original slope map S' and original aspect map A of the high-resolution remote sensing sub-image IMG respectively ′ , and crop them according to the edge E of the landslide risk point to obtain the subsequent available slope map S and aspect map A.

[0048] In traditional landslide risk assessment, the learning ability of the assessment model itself is often directly used to process the original regional digital elevation model (DEM), which may lead to the problem that it is difficult to effectively extract the hidden features in the regional digital elevation model (DEM) when the data volume is small. Therefore, prior knowledge needs to be introduced to calculate the slope map and aspect map in advance.

[0049] Refer to Figure 2 , and the specific implementation of this step is as follows:

[0050] 3.1) Calculate the east-west change rate of each pixel in the regional digital elevation model (DEM) and the north-south change rate

[0051]

[0052] where {a, b, c, d, e, f, g, h, i} are all the numbers of the nine-grid pixels centered on a certain pixel in the height mapping image within the dataset HS. The nine pixels are distributed in three layers, with three pixels in each layer. The first layer is {a, b, c} in sequence, the second layer is {d, e, f} in sequence, and the third layer is {g, h, i} in sequence. we_cellsize represents the pixel size in the east-west direction, and sn_cellsize represents the pixel size in the north-south direction;

[0053] 3.2) Calculate the slope value S (i,j) and aspect value A (i,j) corresponding to each pixel of the digital elevation model (DEM) image according to the results of 3.1):

[0054]

[0055] where the constant C is the conversion factor between degrees and radians, and its value is 57.29578;

[0056] 3.3) Merge the slope values S (i,j) belonging to the same regional digital elevation model (DEM) in 3.2) into the original slope map S ′ ;

[0057] 3.4) Merge the aspect values A (i,j) belonging to the same regional digital elevation model (DEM) in 3.2) into the original aspect map A′ ;

[0058] 3.5) According to the edge E distributed in step 2, crop the original slope map S ′ and the original aspect map A ′ to obtain the slope map S and the aspect map A.

[0059] Step 4, perform 16×16 max pooling on the slope map S to obtain the feature sequence F corresponding to the slope map S S .

[0060] By preprocessing the slope map and the aspect map, the difficulty of model learning can be effectively reduced. However, for slope information, since it is more expected that the model can directly obtain the most direct feature information of the current area, here the feature sequence F corresponding to the slope map S is extracted S to achieve this. According to the fact that the greater the slope, the greater the likelihood of landslides occurring, that is, the slope can reflect the characteristics of the current landslide risk points as the maximum value within the area. In this example, the entire slope map is divided into multiple sub-regions through a 16×16 max pooling operation, and the maximum value of each sub-region is obtained respectively to form the feature sequence F corresponding to the slope map S S , and the specific implementation is as follows:

[0061] [[ID=}3]]4.1) Divide the input slope map S into several local regions S of 16×16 i , where the range of i is 1, 2, 3, …, i, …, N;

[0062]

[0063] where H is the height of the high-resolution remote sensing sub-image IMG, and W is the width of the high-resolution remote sensing sub-image IMG;

[0064] 4.2) Traverse each slope value element in the i-th local region S i to obtain the maximum value MS of the slope value i ;

[0065] 4.3) Sort all the maximum values MS i in ascending order according to the number i and concatenate them into a one-dimensional feature sequence F S .

[0066] Step 5, perform 16×16 average pooling on the aspect map A to obtain the feature sequence F corresponding to the aspect map A A .

[0067] Similar to step 4, for the aspect information in the aspect map, in this example, it is also expected that the model can directly obtain the most direct feature information of the current area. Therefore, the feature sequence F corresponding to the slope map W is extracted here AThis is achieved by taking the average value of the slope directions within a region, which can reflect the characteristics of the current landslide risk points. That is, after averaging the slope values within a region, the obtained value can well describe the orientation of the overall slope surface of the landslide risk points. In this example, a 16×16 average pooling operation is used to divide the entire slope direction map into multiple sub-regions, and their average values are obtained respectively to form the feature sequence F corresponding to the slope map A. A , and the specific implementation is as follows:

[0068] 5.1) Divide the input slope direction map A into several local regions A of 16×16 i , where the range of i is 1, 2, 3, …, i, …, N;

[0069]

[0070] where H is the height of the high-resolution remote sensing sub-image IMG, and W is the width of the high-resolution remote sensing sub-image IMG;

[0071] 5.2) Sum up each slope direction value element in the i-th local region A i to obtain the sum SUM of the slope direction values within the current local region A ;

[0072] 5.3) Assume that the number of slope direction values within the current local region is 256, and use the sum SUM of the slope direction values A to obtain the average value of the slope direction values: MA i = SUM A / 256;

[0073] 5.4) Repeat steps 5.2) and 5.3) to obtain the average value MA of each local region i , and sort all the average values MA i in ascending order according to the number i, and splice them into a one-dimensional feature sequence F A .

[0074] Step 6, detect the residential areas in the high-resolution remote sensing sub-image IMG to obtain the detection result R b .

[0075] For the risk assessment of landslides, it not only lies in the slope characteristics of the landslide risk points themselves, but also the bearing points formed by the key targets around the landslide risk points are one of the important factors considered by experts. Among them, the importance of residential areas is self-evident. In order to well describe the residential area information around the landslide risk points. This example takes additional residential area detection to obtain the information of the surrounding residential areas, and the implementation is as follows:

[0076] 6.1) Select K high-resolution remote sensing sub-images IMG for training, and perform residential area annotation, where K > 100;

[0077] 6.2) Randomly sample a part of them as the validation set SV, and the remaining part as the training set ST;

[0078] 6.3) Load the existing residential area detection model D b ;

[0079] 6.4) Load each high - resolution remote sensing sub - image IMG in the training set ST into the residential area detection model D b one by one, and use the gradient descent algorithm to optimize the residential area detection model;

[0080] 6.5) Load each high - resolution remote sensing sub - image IMG in the validation set SV into the residential area detection model D b one by one, and verify the optimized residential area detection model;

[0081] 6.6) Use the optimized residential area detection model D b to detect the residential areas in the high - resolution remote sensing sub - image IMG to be detected, and record the detection result as R b , where R with residential areas b is recorded as 1, and R without residential areas b is recorded as 0.

[0082] Step 7: Detect the roads in the high - resolution remote sensing sub - image IMG to obtain the detection result R r , and record R with roads r as 1, and R without roads r as 0;

[0083] Similar to the residential area information in step 6, the road information around the landslide risk points is also extremely important disaster - bearing points. In order to well describe the road information around the landslide risk points, additional road detection steps are taken here to obtain the surrounding road information, as follows:

[0084] 7.1) Select K high - resolution remote sensing sub - images IMG for training, and label the residential areas, where K > 100;

[0085] 7.2) Load the existing road detection model D r ;

[0086] 7.4) Load each high - resolution remote sensing sub - image IMG in the training set ST into the road detection model D r one by one, and use the gradient descent algorithm to optimize the road detection model;

[0087] 7.5) Load each high - resolution remote sensing sub - image IMG in the validation set SV into the road detection model D r one by one, and verify the optimized road detection model;

[0088] 7.6) Use the completed road detection model D optimized b Perform road detection on the high-resolution remote sensing sub-image IMG to be detected, and record the detection result as R r , where R with roads r is recorded as 1, and R without roads r is recorded as 0.

[0089] Step 8, perform an embedding operation on the semantic segmentation result R seg of the above landslide risk points, slope map S, and aspect map A to obtain the relevant features F of the image data img .

[0090] In order to integrate features of multiple modalities and multiple forms of representation, it is necessary to perform feature embedding on various inputs. In this example, an existing image data embedding method is used for image feature embedding to obtain the relevant features F of the image data img , which is implemented as follows:

[0091] 8.1) Obtain the width W I and height H I of the input image data;

[0092] 8.2) Use a convolutional kernel with a size of 16 and a stride of 16 to perform a convolutional operation on the image, and divide the image into sub-blocks of 3×16×16;

[0093] 8.3) Expand the above sub-blocks into a sequence of 1×768, and the number of sequences is the same as the number of sub-blocks of the image;

[0094] 8.4) Integrate all the sequences in 8.3) to obtain the relevant features F of the embedded image data img .

[0095] Step 9, perform an embedding operation on the feature sequence F S of the slope map and the feature sequence F A of the aspect map to obtain the relevant features F of the sequence data seq .

[0096] The purpose is the same as in Step 8. The embedding of sequence data is also to ensure that data of multiple modalities can be fully aligned so that the subsequent landslide risk assessment model can process different forms of data in the same form. For the embedding of sequence data, there is no embedding method for sequence data in multi-modal data in the existing methods. In this example, the feature sequence F S of the input slope map and the feature sequence F A, a corresponding embedding method is designed, and the specific implementation is as follows:

[0097] 9.1) For the feature sequence F of the input slope map S , embed features according to its length:

[0098] If the length of the feature sequence F S is greater than 768, downsample it to obtain an embedded feature with a length of 768; if the length of the feature sequence F S is less than 768, interpolate it to obtain an embedded feature with a length of 768;

[0099] 9.2) Denote the position coordinate to be interpolated as x, its corresponding value as y, the previous position coordinate and its value as x0 and y0 respectively, and the subsequent position coordinate and its value as x1 and y1;

[0100] 9.3) Calculate the coordinate distance x between the previous position and the subsequent position Δ 0:

[0101] xΔ = x1 - x0;

[0102] 9.4) Calculate the value difference y between the previous position and the subsequent position Δ :

[0103] yΔ = y1 - y0

[0104] 9.5) According to the results of 9.3) and 9.4), calculate the value y that needs to be interpolated at the current position:

[0105]

[0106] 9.6) Determine whether the interpolated y is within the digital distribution range [0, 90] of the slope:

[0107] If y > 90, set it to 90,

[0108] If y < 0, set it to 0;

[0109] 9.7) For the feature sequence F of the input aspect map A , embed features according to the length of its features:

[0110] If the length of the feature sequence F A is greater than 768, downsample it to obtain an embedded feature with a length of 768;

[0111] If the length of the feature sequence F A is less than 768, interpolate it to obtain an embedded feature with a length of 768;

[0112] 9.8) Denote the position coordinates to be interpolated as x ′ , and the corresponding value as y ′ , and denote the previous position coordinate and its value as x ′ 0 and y0 ′ , and denote the next position coordinate and its value as x1 ′ and y1 ′ ;

[0113] 9.9) Calculate the coordinate distance x between the previous position and the next position ′ Δ :

[0114] x ′ Δ = x1 ′ - x ′ 0

[0115] 9.10) Calculate the value difference y between the previous position and the next position Δ ′ , where:

[0116] y Δ ′ = y1 ′ - y0 ′

[0117] 9.11) Calculate the value y to be interpolated at the current position, where:

[0118]

[0119] 9.12) Judge whether the interpolated y ′ is within the digital distribution range [0, 360] of the slope direction:

[0120] If y ′ > 360, then set it to 360,

[0121] If y ′ < 0, then set it to 0;

[0122] 9.13) Integrate the embedded features to obtain the relevant features F of the sequence data seq .

[0123] Step 10, perform point data embedding on the detection results R b of the above-mentioned residential areas r and the detection results R poi of the roads to obtain the relevant features F of the point data

[0124] Since the detection results R b of the residential areas rThe point feature attribute has only a single data 1 or data 0 to represent whether there is a residential area or a road. However, for image data and sequence data, even after embedding, the feature size has more than one data point. To effectively align and splice the point data with other data. In this example, the detection result R of the input residential area b and the detection result R of the road r are embedded, and the specific implementation is as follows:

[0125] 10.1) For the input detection result R of the residential area b , embed the features according to its specific value:

[0126] If R b = 1, generate a data sequence of length 768, where each element is 1;

[0127] If R b = 0, generate a data sequence of length 768, where each element is 0;

[0128] 10.2) For the input detection result R of the road r , embed the features according to its specific value:

[0129] If R r = 1, generate a data sequence of length 768, where each element is 1;

[0130] If R r = 0, generate a data sequence of length 768, where each element is 0;

[0131] 10.3) Integrate the two sequences generated by the embedding operations in 10.1) and 10.3) to obtain the relevant feature F of the point data poi .

[0132] Step 11, input the above three relevant features F imq , F seq and F poi into the existing visual self-attention model for evaluation to obtain the landslide risk level of the landslide risk point.

[0133] After completing the embedding of the three types of data, namely image data, sequence data, and point data, this example obtains the relevant features F img , F seq and F poi with the same dimension. However, for these three types of data, due to their different sources and different dimensions, a model that can balance different dimensions and different modality data is needed. This example uses the existing visual self-attention model for evaluation, and by adding an additional dimensionless classification vector F c, three relevant features F of the landslide risk points img , F seq and f poi are evaluated to obtain the landslide risk level of the final landslide risk points. The specific implementation is as follows:

[0134] 11.1) Concatenate the embedded image data F img , the sequence data F seq and the point data F poi according to the dimension to obtain a feature sequence F of 592×769;

[0135] 11.2) Concatenate a classification vector F of size 1×768 on the feature sequence F c ;

[0136] 11.3) Input the feature sequence F concatenated with the classification vector into the visual self-attention model to obtain an output vector F ′ ;

[0137] 11.4) Extract the classification vector F ′ that has undergone the self-attention model operation from the output vector F c ′ ;

[0138] 11.5) Input the classification vector F c ′ into the multi-layer perceptron to obtain the final landslide risk level.

[0139] The technical effects of the present invention are described below in combination with simulation experiments:

[0140] 1. Simulation conditions

[0141] An Intel Core i7-7800X CPU with a main frequency of 3.50 GHz, 64.0 GB of memory, an Ubuntu18.04 operating system, and a python3.7.1 + pytorch1.7.0 development environment.

[0142] The experimental data is high-resolution imagery with a resolution of 1 m for part of Xiji County, Ningxia Hui Autonomous Region, and the digital elevation model DEM data is digital elevation model data with a resolution of 30 m for the corresponding area.

[0143] There were a total of 953 cases of experimental data. The experimental data was divided into datasets, resulting in 607 cases in the training set, 346 cases in the validation set, corresponding to 209 landslide risk points with annotation information. All landslide risk points with annotation information were screened. After excluding 7 landslide risk points with incomplete information, there were 4 small-scale landslide risk points, 47 medium-scale landslide risk points, 141 large-scale landslide risk points, and 10 extra-large-scale landslide risk points. Among them, the test set contained 1 small-scale landslide and 5 images after data augmentation corresponding to it, and 7 extra-large-scale landslides. All landslide annotation information also included the following attributes.

[0144] The specific attributes of each case of data are shown in Table 1:

[0145] Table 1 Partial attribute information of landslide risk points

[0146]

[0147] 2. Simulation content

[0148] Simulation 1: Use the method of the present invention to simulate the detection of residential areas. As Figure 3 shown, the simulation content is displayed in three scenarios, namely, a single residential area scenario, a multiple residential areas scenario, and a large residential area scenario.

[0149] Simulation 2: Use the method of the present invention to simulate the detection of roads. As Figure 4 shown, the simulation content is displayed in three scenarios, namely, a field scenario, a scenario within a residential area, and a shadow area scenario.

[0150] From Figure 3 , Figure 4 it can be seen that in multiple scenarios, the present invention can effectively achieve the detection of residential areas and roads.

[0151] Simulation No. 3: Compare the accuracy of the landslide risk level assessment by using the method of the present invention and the existing method of evaluating the landslide grade by combining the area-volume mapping method respectively, and evaluate the simulation results using 3 indicators, namely, precision P, recall rate R, and F1 score. The results are shown in Table 1.

[0152] Table 2 Simulation results

[0153]

[0154] As can be seen from Table 2, for small and extra-large landslides, due to the small amount of data, it is difficult to accurately evaluate the results of the present invention and the existing methods; for the classification results of large and medium-sized landslides, since the present invention adds multiple modal data in the evaluation process, a better landslide risk assessment effect can be achieved compared with the existing methods. Specifically, for medium-sized landslides, the present invention improves the F1 score by about 12 percentage points compared with the existing methods, and for large landslides, the present invention improves the F1 score by 40 percentage points compared with the existing methods, indicating that the present invention can effectively improve the accuracy of landslide risk level assessment.

Claims

1. A landslide risk assessment method based on multimodal learning, characterized in that, It includes the following steps: (1) Extract the required high-resolution remote sensing sub-image IMG from the high-resolution remote sensing image, and extract the altitude data corresponding to all high-resolution remote sensing sub-images IMG from the digital elevation model to form a regional digital elevation model DEM corresponding one-to-one with the high-resolution remote sensing sub-image IMG; (2) Perform semantic segmentation on the high-resolution remote sensing sub-image IMG to obtain the segmentation result R of landslide risk points seg , and combine it with the geographic coordinate projection information in the high-resolution remote sensing image to obtain the edge E of landslide risk points; (3) Use the regional digital elevation model DEM to calculate the original slope map S' and the original aspect map A' corresponding to the high-resolution remote sensing sub-image IMG respectively, and crop them according to the landslide risk point edge E to obtain the subsequent available slope map S and aspect map A; (4) Perform 16×16 max pooling on the slope map S to obtain the feature sequence F corresponding to the slope map S S ; (5) Perform 16×16 average pooling on the slope direction map A to obtain the feature sequence F corresponding to the slope direction map A A ; (6) Detect the residential areas in the high-resolution remote sensing sub-image IMG to obtain the detection result R b , and mark the R with residential areas b as 1, and the R without residential areas b as 0; (7) Detect roads in the high-resolution remote sensing sub-image IMG to obtain the detection result R r , and mark the R with roads r as 1, and the R without roads r as 0; (8) The semantic segmentation result R of the above landslide risk points seg and the slope map S and aspect map A are subjected to an embedding operation on the image data to obtain the relevant features F of the image data img ; (9) Embedding operation is performed on the feature sequence F of the above slope map S and the feature sequence F of the slope aspect map A to obtain the relevant features F of the sequence data seq ; (10) Detection result R of the above-mentioned settlement b , detection result R of the road r Perform embedding of point data to obtain relevant features F of the point data pｏi ; (11) Input the above three relevant features F img , F Seq , and F poi into the existing visual self-attention model for evaluation to obtain the landslide risk level of the landslide risk point.

2. The method according to claim 1, characterized in that In the step (1), the extraction of the altitude data corresponding to all high-resolution remote sensing sub-images IMG from the high-resolution remote sensing sub-image IMG is realized as follows: (1a) Cut the digital elevation model information corresponding to the landslide risk point according to the geographical information and store it in a single-channel image corresponding to the size of the corresponding high-resolution remote sensing sub-image IMG; (1b) Calculate the mapped altitude h'(i, j) of each pixel in the single-channel image: h'(i, j) = h(i, j) - min(h(i, j)) where h(i, j) is the value at the corresponding coordinate position (i, j) in the digital elevation model, and min is a function to take the global minimum value; (1c) Integrate the mapped altitude h'(i, j) of each pixel in the single-channel image to obtain the altitude data corresponding to the high-resolution remote sensing sub-image IMG.

3. The method according to claim 1, wherein In the above (4), a 16×16 max pooling operation is used to obtain the feature sequence F corresponding to the slope map S S , and the implementation is as follows: (4a)Divide the input slope map S into a number of 16×16 local regions S i , where the range of i is 1, 2, 3, ..., i, ..., N; where H is the height of the high-resolution remote sensing sub-image IMG, and W is the width of the high-resolution remote sensing sub-image IMG; (4b) Traverse each slope value element in the i-th local area S i to obtain the maximum value MS of the slope value i ; (4c) Sort all the maximum values MS i in ascending order according to the number i and concatenate them into a one-dimensional feature sequence F S .

4. The method according to claim 1, wherein In the above (5), the feature sequence F corresponding to the slope aspect map A is obtained by using 16×16 average pooling, and the implementation is as follows: A , as follows: (5a)Divide the input slope aspect map A into a number of local regions A of 16×16 i , where the range of i is 1, 2, 3,..., i,..., N: where H is the height of the high-resolution remote sensing sub-image IMG, and W is the width of the high-resolution remote sensing sub-image IMG; (5b) Sum the aspect value elements in the $i$-th local area $A$ i to obtain the aspect value $zh$ and $SUM$ within the current local area A ; (5c) Use the sum SUM of the aspect values A Divide by the number 256 of the aspect values in the current local area to obtain the average value MA of the aspect values i (5d) Repeat steps (6b) - (6c) to obtain the average value MA of each local area i , and all the average values MA i are sorted according to the number i and concatenated into a one-dimensional feature sequence F A .

5. The method according to claim 1, characterized in that, In (9) above, the embedding operation using sequence data is performed on the slope map feature sequence F S and the aspect map feature column F A for feature embedding, and the implementation is as follows: (9a) For the feature sequence F of the input slope map S , embed features according to its length: If the feature sequence F S has a sequence length greater than 768, it is downsampled to obtain an embedded feature with a length of 768; If the feature sequence F S has a sequence length less than 768, it is interpolated to obtain an embedded feature with a length of 768; (9b) For the feature sequence F of the input slope aspect map A , perform feature embedding according to the length of its features: If the feature sequence F A has a sequence length greater than 768, it is downsampled to obtain an embedded feature with a length of 768; If the feature sequence F A has a sequence length less than 768, it is interpolated to obtain an embedded feature with a length of 768.

6. The method according to claim 5, wherein Interpolate the feature sequence F with a sequence length less than 768 in (9a) as follows: S This is achieved as follows: (9a1) Denote the position coordinate to be interpolated as x, its corresponding value as y, its previous position coordinate and its value as x0 and y0 respectively, and its subsequent position coordinate and its value as x1 and y1; (9a2) Calculate the coordinate distance x between the previous position and the next position Δ , where: x Δ = x1 - x0 (9a3) Calculate the numerical difference y between the previous position and the next position Δ , where: y Δ = y1 - y0 (9a4) Calculate the value y that needs to be interpolated at the current position, where: (9a5) Judge whether the interpolated y is within the digital distribution range [0, 90] of the slope: If y > 90, then set it to 90, If y < 0, then set it to 0.

7. The method according to claim 5, characterized in that, Interpolate the feature sequence F with a sequence length less than 768 in (9b) as follows: A The implementation is as follows: (9b1) Denote the position coordinate to be interpolated as x, its corresponding value as y, its previous position coordinate and its value as x0 and y0 respectively, and its subsequent position coordinate and its value as x1 and y1; (9b2) Calculate the coordinate distance x between the previous position and the next position Δ , where: x Δ = x1 - x0 (9b3) Calculate the numerical difference y between the previous position and the next position Δ , where: y Δ = y1 - y0 (9b4) Calculate the value y that needs to be interpolated at the current position, where: (9b5) Judge whether the interpolated y is within the digital distribution range [0, 360] of the aspect: If y > 360, then set it to 360, If y < 0, then set it to 0.

8. The method according to claim 1, characterized in that, In the above (10), the embedding operation of point data is used for the residential point detection result R b and the road detection result R r to perform the embedding of point data, and the implementation is as follows: (10a) For the input residential area detection result R b , embed features according to its specific value: If R b = 1, a data sequence of length 768 is generated, where each element is 1; If R b is 0, a data sequence of length 768 is generated, where each element is 0; (10b) For the input road detection result R r , embed features according to its specific value: If R r = 1, a data sequence of length 768 is generated, where each element is 1; If R r is 0, a data sequence of length 768 is generated, where each element is 0.

Images