Landslide hazard point land subsidence prediction method and device based on multi-modal data

Abstract

The application relates to a landslide hidden point land subsidence prediction method and device based on multi-modal data. The method considers various physical factors that have a greater impact on land subsidence. First, the modal data of numerous physical factors are divided into static data, time-varying continuous data and future predictable data according to trends such as change, time and parameter type. Feature extraction is performed on various types of data through a neural network, which can capture complex nonlinear relationships between different modal data. Then, the static features of the static data are used to enhance the future predictable features of the future predictable data and the time-varying features of the time-varying continuous data. Finally, the time-varying features and the future predictable features are integrated using a multi-head attention mechanism to predict the land subsidence state of a rainfall-type landslide hidden point according to the integrated features, thereby improving the accuracy of landslide hidden point land subsidence state prediction.

Description

Technical Field

[0001] This application relates to the field of settlement prediction technology, and in particular to a method and apparatus for predicting land settlement at landslide hazard points based on multimodal data. Background Technology

[0002] Rainfall-induced landslides mainly occur in areas with specific geological conditions (such as shale, mudstone, metamorphic rocks, earthquake zones), topography (such as steep slopes, river valleys, artificially modified areas), and climatic conditions (such as frequent rainstorms and typhoon impacts). Once they occur, they will cause casualties, economic losses, and environmental and ecological damage.

[0003] Landslide susceptibility assessment (LSA) is used to evaluate the likelihood of landslides occurring, helping to mitigate and prevent landslide risks. With the increased availability of high-quality satellite data and landslide logging data, data-driven landslide susceptibility assessment methods have become widely used; however, several issues remain to be addressed, as follows:

[0004] (1) The items in the landslide catalog are mainly historical landslide information obtained through the interpretation of optical images and field surveys. This makes the prediction model trained with these items insensitive to undetectable slope movements, such as slow landslides that have not yet occurred.

[0005] (2) Most of the study areas contain a variety of landslide-prone geographical environments, and a single model cannot adapt well to these situations;

[0006] (3) For rainfall-induced landslides, it is difficult to predict the settlement in the future based solely on static data or satellite remote sensing data. Summary of the Invention

[0007] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0008] The main objective of this disclosure is to propose a method and apparatus for predicting land subsidence at landslide hazard points based on multimodal data, which can improve the accuracy of predicting the land subsidence status at landslide hazard points.

[0009] A first aspect of this application provides a method for predicting land subsidence at landslide hazard points based on multimodal data, the method comprising:

[0010] In response to land subsidence prediction signals at potential landslide sites caused by rainfall, multimodal data of potential landslide sites caused by rainfall are acquired.

[0011] Each modal data in the multimodal data is divided into at least one of the following: first static data, second static data, time-varying continuous data, and future predictable data; the first static data is text-type modal data that does not change with time, or text-type modal data that changes with time and whose fluctuation is less than a first threshold; the second static data is numerical modal data that does not change with time, or numerical modal data that changes with time and whose fluctuation is less than the first threshold; the time-varying continuous data is modal data that changes continuously with time; and the future predictable data is modal data that can be numerically predicted within a future period of time.

[0012] The first static data and the second static data are input into a first neural network to obtain static features of the first static data and the second static data; the time-varying continuous data and the static features are input into a second neural network to obtain time-varying features of the time-varying continuous data enhanced by the static features; the future predictable data and the static features are input into a third neural network to obtain future predictable features of the future predictable data enhanced by the static features.

[0013] By integrating the time-varying features and the predictable future features using a multi-head attention mechanism, an attention matrix output by the multi-head attention mechanism is obtained, and the land subsidence state of the rainfall-induced landslide hazard point is predicted based on the attention matrix.

[0014] The land subsidence prediction method for landslide hazard points based on multimodal data provided in this embodiment has at least the following beneficial effects:

[0015] This method considers multiple physical factors that significantly influence land subsidence. First, the modal data of numerous physical factors are categorized into static data, time-varying continuous data, and future predictable data based on trends such as change, time, and parameter type. A neural network is used to extract features from each type of data, capturing the complex nonlinear relationships between different modal data. Then, the static features of the static data are used to enhance the future predictable features of the future predictable data and the time-varying features of the time-varying continuous data. Finally, a multi-head attention mechanism is used to integrate the time-varying features and the future predictable features to predict the land subsidence state of rainfall-induced landslide hazard points, thereby improving the accuracy of land subsidence state prediction for landslide hazard points.

[0016] A second aspect of this application provides a land subsidence prediction device for landslide hazard points based on multimodal data, the device comprising:

[0017] The data acquisition module is used to acquire multimodal data of rainfall-induced landslide hazard points in response to land subsidence prediction signals.

[0018] The data segmentation module is used to segment each modal data in the multimodal data into at least one of the following: first static data, second static data, time-varying continuous data, and future predictable data; the first static data is text-type modal data that does not change with time, or text-type modal data that changes with time and whose fluctuation is less than a first threshold; the second static data is numerical modal data that does not change with time, or numerical modal data that changes with time and whose fluctuation is less than the first threshold; the time-varying continuous data is modal data that changes continuously with time; and the future predictable data is modal data that can be numerically predicted within a future period of time.

[0019] The feature extraction module is used to input the first static data and the second static data into a first neural network to extract static features of the first static data and the second static data; input the time-varying continuous data and the static features into a second neural network to extract time-varying features of the time-varying continuous data enhanced by the static features; and input the future predictable data and the static features into a third neural network to extract future predictable features of the future predictable data enhanced by the static features.

[0020] The state prediction module is used to integrate the time-varying features and the future predictable features according to the multi-head attention mechanism to obtain the attention matrix output by the multi-head attention mechanism, and predict the land subsidence state of the landslide hazard point according to the attention matrix.

[0021] A third aspect of this application provides an electronic device including at least one controller and a memory for communicatively connecting to the controller; the memory stores instructions executable by the at least one controller, the instructions being executed by the at least one controller to cause the at least one controller to perform a land subsidence prediction method for landslide hazard points based on multimodal data as described above.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1This is a flowchart illustrating an embodiment of a landslide hazard point landslide subsidence prediction method based on multimodal data provided in this application;

[0025] Figure 2 This is a schematic diagram of an embodiment of a landslide hazard point land subsidence prediction device based on multimodal data provided in this application;

[0026] Figure 3 This is a schematic diagram of the structure of an embodiment of an electronic device provided in this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0029] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed or function in a specific orientation, and therefore should not be construed as a limitation of this application.

[0030] like Figure 1 One embodiment of this application provides a method for predicting land subsidence at landslide hazard points based on multimodal data, the method comprising the following steps S100 to S400:

[0031] Step S100: In response to the land subsidence prediction signal of the rainfall-induced landslide hazard point, acquire multimodal data of the rainfall-induced landslide hazard point.

[0032] In step S100, the execution subject of this method can be a computer device, which obtains multimodal data of the landslide hazard point by responding to the land subsidence prediction signal of the landslide hazard point.

[0033] It should be noted that the landslide hazard involved in this embodiment is a rainfall-induced landslide hazard. Rainfall-induced landslides mainly occur in areas with specific geological, topographical and climatic conditions. The rainfall-induced landslide hazard point can be a pre-set point, such as a location that relevant personnel need to monitor in advance. No specific limitation is made here.

[0034] In this step, since the physical model of land subsidence is very complex and closely related to multiple factors such as geological type, water content, and rainfall at each location, the multimodal data includes, but is not limited to: lithology, landslide, vertical deformation, horizontal deformation, distance to river, distance to fault layer, distance to water system, aspect, slope, altitude, profile curvature, vegetation coefficient, and rainfall.

[0035] The following describes the methods for obtaining horizontal and vertical deformations. Other modal data are common knowledge in the field and will not be elaborated here:

[0036] Satellite remote sensing data of rainfall-induced landslide hazard areas are downloaded, and an interferometric image pair network is constructed based on spatiotemporal baseline thresholds to generate a connectivity map. The connectivity map is then subjected to interferometric processing to obtain a filtered differential interferometric map. Finally, through two inversions and geocoding, the deformation results of each rainfall-induced landslide hazard point are output as one of the modal data.

[0037] The following describes the preprocessing procedure for multimodal data:

[0038] The spatiotemporal resolution of all modal data is registered and aligned; all numerical modal data are standardized to eliminate dimensional differences; the covariance matrix is ​​calculated and eigenvalues ​​and eigenvectors are obtained; the top few largest eigenvalues ​​are selected as principal components, and projection matrices are constructed using the corresponding eigenvectors to map the factors to the new principal component space. The purpose here is to use principal component analysis to first screen out factors that are more closely related to sedimentation.

[0039] Then, clustering algorithms can be used to classify multimodal data, for example, clustering based on their numerical value and distance from rainfall-induced landslide hazard points. Alternatively, classification can be based on the characteristics of the modal data itself; therefore, any modal data may be divided into at least two different categories.

[0040] Step S200: Divide each modal data in the multimodal data into at least one of first static data, second static data, time-varying continuous data, and future predictable data; the first static data is text-type modal data that does not change with time, or text-type modal data that changes with time and whose fluctuation is less than a first threshold; the second static data is numerical modal data that does not change with time, or numerical modal data that changes with time and whose fluctuation is less than a first threshold; the time-varying continuous data is modal data that changes continuously with time; and the future predictable data is modal data that can be numerically predicted in the future.

[0041] Taking the modal data mentioned above as an example, the result of its partitioning can be:

[0042] First static data (textual modal data that does not change over time, or textual modal data that changes over time with fluctuations less than a first threshold): lithology, landslides, vertical deformation, horizontal deformation; Second static data (numerical modal data that does not change over time, or numerical modal data that changes over time with fluctuations less than a first threshold): distance to river, distance to fault, distance to drainage system, aspect, slope, altitude, profile curvature; Time-varying continuous data (modal data that changes continuously over time): vegetation coefficient, vertical and horizontal deformation; Future predictable data (modal data that can be numerically predicted over a future period): rainfall, etc.

[0043] It should be noted that the first threshold mentioned above has different values ​​for different modal data, and can be set based on experience.

[0044] Step S300: Input the first static data and the second static data into the first neural network to extract the static features of the first static data and the second static data; input the time-varying continuous data and the static features into the second neural network to extract the time-varying features of the time-varying continuous data enhanced by the static features; input the future predictable data and the static features into the third neural network to extract the future predictable features of the future predictable data enhanced by the static features.

[0045] In some embodiments, the first neural network can be a GRN-gated neural network. A GRN-gated neural network mainly consists of exponential linear activation units (GLUs), gated linear units (GLUs), and layer normalization units (LNs). The GLUs and LNs are primarily used for feature extraction, while the exponential linear activation units (SoftMax function) are mainly used for priority ranking (i.e., weight generation). Finally, the features and weights are weighted and summed to obtain the static features of the first and second static data. Alternatively, the second and third neural networks can also be GRN-gated neural networks. In the second and third neural networks, additional static features are added, that is, static features are used to enhance the time-varying features of time-varying continuous data, and to enhance the future-predictable features of future-predictable data.

[0046] Step S400: Based on the multi-head attention mechanism, integrate time-varying features and future predictable features to obtain the attention matrix output by the multi-head attention mechanism, and predict the land subsidence status of rainfall-induced landslide hazard points based on the attention matrix.

[0047] The land subsidence prediction method for landslide hazard points based on multimodal data provided in this embodiment has at least the following beneficial effects:

[0048] This method considers multiple physical factors that significantly influence land subsidence. First, the modal data of numerous physical factors are categorized into static data, time-varying continuous data, and future predictable data based on trends such as change, time, and parameter type. A neural network is used to extract features from each type of data, capturing the complex nonlinear relationships between different modal data. Then, the static features of the static data are used to enhance the future predictable features of the future predictable data and the time-varying features of the time-varying continuous data. Finally, a multi-head attention mechanism is used to integrate the time-varying features and the future predictable features to predict the land subsidence state of rainfall-induced landslide hazard points, thereby improving the accuracy of land subsidence state prediction for landslide hazard points.

[0049] In some embodiments, the first neural network is a first gated GRN network;

[0050] Step S300, which involves inputting the first static data and the second static data into the first neural network to obtain the static features of the first static data and the second static data, includes:

[0051] Step S3110: Extract the flattening features of the input data based on the gated linear units and layer normalization units of the first gated GRN network; the input data can be any first static data and second static data.

[0052] Step S3120: Determine the importance weights corresponding to the flattened features based on the exponential linear activation function of the first gated GRN network;

[0053] Step S3130: Perform a weighted summation based on each flattened feature and its corresponding importance weight to obtain the static features.

[0054] In this embodiment, the first gated GRN neural network mainly consists of an exponential linear activation unit, a gated linear unit, and a layer normalization unit. The gated linear unit and the layer normalization unit are mainly used to extract features, while the exponential linear activation unit is mainly used to sort priorities (i.e. generate weights). Finally, the features and weights are weighted and summed to obtain the static features of the first static data and the second static data.

[0055] For example, flattening features using gated linear units. Extraction.

[0056] ;

[0057] in, Indicates a time step. , This represents the weights and biases of the linear layer to be trained. The input is a vector representation of any first static data or second static data.

[0058] Then, the relevant flattened features are prioritized using the SoftMax function to generate importance weights. :

[0059] ;

[0060] Next, importance weights With features Weighting yields static features. .

[0061] This method uses a gated GRN network to accurately extract static features, thereby improving the accuracy of the final model's predictions. It's important to note that the first neural network can be replaced by other feature extraction neural networks, such as a Long Short-Term Memory (LSTM) network.

[0062] In this embodiment, the second neural network is a second gated GRN network;

[0063] Before inputting the time-varying continuous data and static features into the second neural network in step S300 to obtain the time-varying features of the time-varying continuous data enhanced by static features, the method further includes:

[0064] Step S1031: Input the static features into the fourth gated GRN network to obtain the first context features of the output static features;

[0065] Step S300 involves inputting time-varying continuous data and static features into a second neural network to obtain time-varying features of the time-varying continuous data enhanced by static features, including:

[0066] Step S3210: The first context feature and time-varying continuous data are fused according to the gating mechanism of the second gated GRN network to obtain the first fused feature;

[0067] Step S3220: Extract the flattened features of the first fusion feature based on the gated linear units and layer normalization units of the second gated GRN network.

[0068] Step S3230: Determine the importance weights corresponding to the flattened features of the first fusion feature based on the exponential linear activation function of the second gated GRN network.

[0069] Step S3240: Perform a weighted summation based on the flattened features of the first fusion feature and their corresponding importance weights to obtain the weighted summation feature;

[0070] Step S3250: Based on the weighted summation feature, obtain the time-varying features of the time-varying continuous data enhanced by static features.

[0071] Unlike steps S3110 to S3130 described above, this step adds steps S1031 and S3210. In step S1031, the extracted static features are used as input, and then the gated linear units of the fourth gated GRN network are used to extract the local context information of the static features to obtain the feature representation. Then, the Softmax function is used to filter the importance relationships, and finally, a weighted sum is performed to obtain the first context feature of the static features. In step S3210, the features of the time-varying continuous data are first extracted using a gating mechanism, and then the first context feature and the corresponding features of the time-varying continuous data are fused. For example, combining the first context feature... Input it into the second-gated GRN network ,Then:

[0072] ;

[0073] ;

[0074] in, The vector representation of the input time-varying continuous data. and For the weights of the linear layer to be trained, This is the bias of the linear layer to be trained.

[0075] Similarly, regarding the flattening feature Prioritize the components and output their importance weights. :

[0076] ;

[0077] The processed features are weighted and combined to obtain the time-varying features of the time-varying continuous data after static feature enhancement. This embodiment utilizes the contextual information corresponding to the static features to enhance the time-varying features of the time-varying continuous data, which can enrich the dynamic input information, provide a comprehensive view of the data, improve the accuracy of model prediction, and also help alleviate the gradient vanishing problem and maintain long-term dependencies.

[0078] In some embodiments of this application, before step S300, which involves inputting time-varying continuous data and static features into a second neural network to obtain time-varying features of the time-varying continuous data enhanced by static features, the method further includes:

[0079] Step S1032: Input the static features into the fifth gated GRN network and the sixth gated GRN network respectively to obtain the second context features of the static features output by the fifth gated GRN network and the third context features of the static features output by the sixth gated GRN network; wherein, the network structure of the fourth gated GRN network to the sixth gated GRN network is the same, but the network parameters are different.

[0080] Step S3250, based on the weighted summation feature, obtains the time-varying features of the time-varying continuous data enhanced by static features, including:

[0081] Step S3251: Input the weighted summation features into the long short-term memory network, and use the second context features and the third context features as the cell state and hidden state of the long short-term memory network to obtain the output first feature sequence;

[0082] Step S3252: Based on the first feature sequence, generate time-varying features of the time-varying continuous data after static feature enhancement.

[0083] To further learn about the temporal relationships in the data, the influence of the first and second static data is introduced. In this embodiment, the second and third contextual features are used as the cell state and hidden state of the LSTM to enhance the time-varying features and improve the accuracy of the model prediction.

[0084] In some embodiments of this application, before inputting the time-varying continuous data and static features into the second neural network in step S300 to obtain the time-varying features of the time-varying continuous data enhanced by static features, the method further includes:

[0085] Step S1033: Input the static features into the seventh gated GRN network to obtain the fourth context features of the static features output by the seventh gated GRN network; the network structure of the fourth gated GRN network is the same as that of the seventh gated GRN network, but the network parameters are different.

[0086] Step S3252, which generates time-varying features of the time-varying continuous data after static feature enhancement based on the first feature sequence, includes:

[0087] The first feature sequence is enhanced based on the fourth context feature to obtain the time-varying features of the time-varying continuous data after static feature enhancement.

[0088] Finally, this embodiment uses the fourth context feature to enhance the first feature sequence, obtaining the time-varying features of the time-varying continuous data after static feature enhancement. This can further introduce the influence of the first static data and the second static data, thereby improving the accuracy of model prediction.

[0089] In some embodiments of this application, the third neural network is a third gated GRN network;

[0090] Before step S300, which involves inputting the future predictable data and static features into the third gated GRN neural network to obtain the future predictable features of the future predictable data enhanced by static features, the following steps are also included:

[0091] Step S1034: Input the static features into the fourth gated GRN network to obtain the first context features of the output static features;

[0092] Step S300 involves inputting the future predictable data and static features into the third gated GRN neural network to obtain the future predictable features of the future predictable data enhanced by static features, including:

[0093] Step S3310: According to the gating mechanism of the third gated GRN network, the first context features and the future predictable data are fused to obtain the second fused features;

[0094] Step S3320: Extract the flattened features of the second fusion feature based on the gated linear units and layer normalization units of the third gated GRN network.

[0095] Step S3330: Determine the importance weights corresponding to the flattened features of the second fusion feature based on the exponential linear activation function of the third gated GRN network.

[0096] Step S3340: Perform a weighted summation based on the flattened features of the second fusion feature and their corresponding importance weights to obtain the weighted summation feature;

[0097] Step S3350: Based on the weighted summation features, obtain the future predictable features of the future predictable data enhanced by static features.

[0098] Similar to steps S3210 to S3250 in the above embodiments, this embodiment extracts features from future predictable data and enhances the future predictable features of future predictable data based on static features. It can utilize the context information corresponding to static features to enhance the data of future predictable features, enrich dynamic input information, provide a comprehensive view of the data, improve the accuracy of model prediction, and also help alleviate the gradient vanishing problem and maintain long-term dependencies.

[0099] In some embodiments of this application, before step S300, which involves inputting the future predictable data and static features into a third gated GRN neural network to obtain the future predictable features of the future predictable data enhanced by static features, the method further includes:

[0100] Step S1035: Input the static features into the fifth gated GRN network and the sixth gated GRN network respectively to obtain the second context features of the static features output by the fifth gated GRN network and the third context features of the static features output by the sixth gated GRN network; wherein, the network structure of the fourth gated GRN network to the sixth gated GRN network is the same, but the network parameters are different.

[0101] Step S3350: Based on the weighted summation features, obtain the future predictable features of the future predictable data enhanced by static features, including:

[0102] Step S3351: Input the weighted summation features into the long short-term memory network, and use the second context features and the third context features as the cell state and hidden state of the long short-term memory network to obtain the output second feature sequence;

[0103] Step S3352: Based on the second feature sequence, generate future predictable features of the future predictable data after static feature enhancement.

[0104] Similar to steps S3251 to S3252 in the above embodiments, in order to further learn the temporal relationships in the data, the influence of the first static data and the second static data is introduced. In this embodiment, the second context features and the third context features are used as the cell state and hidden state of LSTM to enhance the predictable features in the future, so as to improve the accuracy of model prediction.

[0105] In this embodiment of the application, before step S300, which involves inputting the future predictable data and static features into the third gated GRN neural network to obtain the future predictable features of the future predictable data enhanced by static features, the method further includes:

[0106] Step S1036: Input the static features into the seventh gated GRN network to obtain the fourth context features of the static features output by the seventh gated GRN network; the network structure of the fourth gated GRN network is the same as that of the seventh gated GRN network, but the network parameters are different.

[0107] Step S3352, which generates future predictable features of the statically enhanced future predictable data based on the second feature sequence, includes:

[0108] The second feature sequence is enhanced based on the fourth contextual feature to obtain the future predictable features of the future predictable data after static feature enhancement.

[0109] Finally, this embodiment uses the fourth contextual feature to enhance the second feature sequence, obtaining the future predictable features of the future predictable data after static feature enhancement. This can further introduce the influence of the first static data and the second static data, and improve the accuracy of model prediction.

[0110] To facilitate understanding, a set of embodiments is provided, namely, a method for predicting land subsidence at rainfall-induced landslide hazard points based on multimodal data;

[0111] Step S911: Obtain multimodal data of potential landslide sites caused by rainfall;

[0112] Factors related to land subsidence, such as meteorological, hydrological, and geological factors, can be considered. These include lithology, landslides, horizontal deformation, distance to rivers, distance to fault lines, distance to water systems, aspect, slope, altitude, profile curvature, vegetation coefficient, vertical deformation, and rainfall.

[0113] Step S912: Preprocess the multimodal data;

[0114] Step S913: Classify the multimodal data;

[0115] Classification methods include, but are not limited to: principal component analysis, clustering, or expert experience, etc.

[0116] First static data: lithology, landslides, vertical deformation, horizontal deformation; Second static data: distance to river, distance to fault layer, distance to drainage system, aspect, slope, altitude, profile curvature; Time-varying continuous data: vegetation coefficient, vertical and horizontal deformation; Predictable future data: rainfall, etc.

[0117] For discrete variables (static categorical data), entity embedding is used as the feature representation; for continuous data, linear transformation is used.

[0118] The overall model comprises seven gated GRN networks, along with attention mechanisms and linear mappings (such as fully connected layers). It's important to note that these networks form the overall system, which can be trained and updated using a loss function to refine the parameters of each network.

[0119] Step S914: Input the first static data and the second static data into the first gated GRN network. , It consists of the exponential linear activation function SoftMax, the gated linear unit GLU, and the layer normalization unit LN.

[0120] First, the features are flattened using a gated linear unit (GLU). Extraction.

[0121] ;

[0122] in, Indicates a time step. , This represents the weights and biases of the linear layer to be trained. The input is a vector representation of any first static data or second static data.

[0123] Then, the relevant flattened features are prioritized using the SoftMax function to generate importance weights. :

[0124] ;

[0125] Next, importance weights With features Weighting yields static features. :

[0126] Let it exist The static categorical data and static continuous data, among which the gated linear unit (GLU) extracts the first... The flattening characteristics of static categorical data or static continuous data are: Then further:

[0127] ;

[0128] Among them, the final result is the first The static characteristics of time nodes are .

[0129] Step S915, static features Inputs are sent to four encoders (from the fourth gated GRN network to the seventh gated GRN network). to In the composition, four contextual features are output: These contexts are connected to different locations in subsequent networks to enrich dynamic input information, provide a comprehensive view of the data, and at the same time, this combination of non-linear processing and skip connections enhances the model's adaptability, helps alleviate the gradient vanishing problem, and maintains long-term dependencies.

[0130] It is important to note that The contribution varies across different regions. For example, rainfall alone may have a limited impact on landslides, so its contribution to the variable selection module will not be high. However, continuous rainfall over several days has a greater impact on landslides, so the contribution of static data for that region will be higher. Because the entire prediction process is conducted within a large neural network, the parameters of each gated GRN network are automatically updated based on the gradient, which will not be elaborated here.

[0131] Step S916: For time-varying continuous data, combine the first context features Input it into the second-gated GRN network ,Right now:

[0132] ;

[0133] ;

[0134] in, The vector representation of the input time-varying continuous data. and For the weights of the linear layer to be trained, The first context feature output in step S915 This is the bias of the linear layer to be trained.

[0135] Similarly, regarding the flattening feature Prioritize the components and output their importance weights. :

[0136] ;

[0137] The processed features are then weighted and combined:

[0138] Order The extracted time-varying continuous data, of which the first... The characteristics of time-varying continuous data are: :

[0139] ;

[0140] Step S917: Use a Long Short-Term Memory (LSTM) network codec to process time-varying continuous data to predict future time steps;

[0141] To further learn the temporal relationships present in the data, the influence of the first and second static data is introduced, and the second contextual features are incorporated. and third context features Using the cell state and hidden state as the basis for LSTM, the first feature sequence of the output is finally obtained: ;

[0142] Step S918, through the third-gated GRN network Using the fourth context feature For the first feature sequence Static enhancement is performed to obtain the enhanced features. :

[0143] ;

[0144] Step S919: For predictable future data, perform the same operations from steps S960 to S980 to obtain the enhanced features. ;

[0145] Step S920: All features enhanced by the first static data and the second static data are... , Combined into Then, information from static and dynamic features is integrated through a multi-head attention mechanism to output an attention matrix, and then a gated residual mechanism is used for skip connections. This will promote training.

[0146] Step S921 finally uses linear mapping to predict land subsidence at rainfall-induced landslide hazard points.

[0147] This method fully considers the physical factors that significantly influence land subsidence in reality. First, it categorizes the modal data of numerous physical factors into static data, time-varying continuous data, and future predictable data according to trends such as change, time, and parameter type. Then, it extracts features from each type of data using neural networks, capturing the complex nonlinear relationships between different classification modal data. Next, it enhances the future predictable features of the two types of static data and the time-varying features of the time-varying continuous data by adding static attributes to both future predictable and time-varying features. Finally, it integrates the time-varying and future predictable features using a multi-head attention mechanism to predict the land subsidence state of rainfall-induced landslide hazard points based on the integrated features, thereby improving the accuracy of land subsidence state prediction.

[0148] like Figure 2 One embodiment of this application provides a land subsidence prediction device for landslide hazard points based on multimodal data. The device includes:

[0149] The data acquisition module 1100 is used to acquire multimodal data of the land subsidence prediction signal of the landslide hazard point in response to rainfall-induced landslide hazard point;

[0150] The data segmentation module 1200 is used to segment each modal data in the multimodal data into at least one of the following: first static data, second static data, time-varying continuous data, and future predictable data; the first static data is text-type modal data that does not change with time, or text-type modal data that changes with time and whose fluctuation is less than a first threshold; the second static data is numerical modal data that does not change with time, or numerical modal data that changes with time and whose fluctuation is less than a first threshold; the time-varying continuous data is modal data that changes continuously with time; and the future predictable data is modal data that can be numerically predicted in the future.

[0151] The feature extraction module 1300 is used to input the first static data and the second static data into the first neural network to obtain the static features of the first static data and the second static data; input the time-varying continuous data and the static features into the second neural network to obtain the time-varying features of the time-varying continuous data enhanced by the static features; and input the future predictable data and the static features into the third neural network to obtain the future predictable features of the future predictable data enhanced by the static features.

[0152] The state prediction module 1400 is used to integrate time-varying features and future predictable features according to the multi-head attention mechanism to obtain the attention matrix output by the multi-head attention mechanism, and predict the land subsidence state of rainfall-induced landslide hazard points according to the attention matrix.

[0153] It should be noted that the land subsidence prediction device for landslide hazard points based on multimodal data provided in this embodiment is based on the same inventive concept as the land subsidence prediction method for landslide hazard points based on multimodal data described above. Therefore, the content of the land subsidence prediction device for landslide hazard points based on multimodal data described in this embodiment is also applicable to the content of the land subsidence prediction method for landslide hazard points based on multimodal data described above, and will not be repeated here.

[0154] like Figure 3 One embodiment of this application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for predicting land subsidence at landslide hazard points based on multimodal data. The electronic device includes:

[0155] At least one battery;

[0156] At least one memory;

[0157] At least one processor;

[0158] At least one program;

[0159] The program is stored in memory, and the processor executes at least one program to implement the above-described method for predicting land subsidence at landslide hazard points based on multimodal data.

[0160] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0161] The electronic devices according to embodiments of this application will now be described in detail.

[0162] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0163] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to perform a land subsidence prediction method for landslide hazard points based on multimodal data, as described in this disclosure.

[0164] The input / output interface 1800 is used to implement information input and output.

[0165] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0166] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0167] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0168] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described method for predicting land subsidence at landslide hazard points based on multimodal data.

[0169] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0170] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0171] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0172] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0174] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0175] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0179] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A landslide hazard point land subsidence prediction method based on multi-modal data, characterized by, The method comprises: obtaining multi-modal data of a rainfall-type landslide hidden danger point in response to a land subsidence prediction signal of the rainfall-type landslide hidden danger point; dividing each modal data in the multi-modal data into at least one of first static data, second static data, time-varying continuous data, and future predictable data; the first static data is text-type modal data that does not change with time or text-type modal data that changes with time and has a fluctuation less than a first threshold; the second static data is numerical-type modal data that does not change with time or numerical-type modal data that changes with time and has a fluctuation less than the first threshold; the time-varying continuous data is modal data that continuously changes with time; and the future predictable data is modal data that can be numerically predicted in a future period of time; inputting the first static data and the second static data into a first neural network to obtain static features of the first static data and the second static data; inputting the static features into a fourth gated GRN network to obtain first context features outputting the static features; inputting the time-varying continuous data and the static features into a second neural network to obtain time-varying features of the time-varying continuous data enhanced by the static features; and inputting the future predictable data and the static features into a third neural network to obtain future predictable features of the future predictable data enhanced by the static features; the second neural network is a second gated GRN network; and the third neural network is a third gated GRN network; integrating the time-varying features and the future predictable features according to a multi-head attention mechanism to obtain an attention matrix output by the multi-head attention mechanism, and predicting a land subsidence state of the rainfall-type landslide hidden danger point according to the attention matrix; inputting the time-varying continuous data and the static features into a second neural network to obtain time-varying features of the time-varying continuous data enhanced by the static features, comprising: fusing the first context features and the time-varying continuous data according to a gating mechanism of the second gated GRN network to obtain first fused features; extracting flattened features of the first fused features according to a gating linear unit and a layer normalization unit of the second gated GRN network; determining importance weights corresponding to the flattened features of the first fused features according to an exponential linear activation function of the second gated GRN network; performing weighted summation on the flattened features of the first fused features and the importance weights corresponding thereto to obtain weighted summation features; obtaining the time-varying features of the time-varying continuous data enhanced by the static features according to the weighted summation features; inputting the future predictable data and the static features into a third neural network to obtain future predictable features of the future predictable data enhanced by the static features, comprising: fusing the first context features and the future predictable data according to a gating mechanism of the third gated GRN network to obtain second fused features; According to the gating linear unit and layer normalization unit of the third gated GRN network, flat features of the second fusion features are extracted; According to the exponential linear activation function of the third gated GRN network, importance weights corresponding to the flat features of the second fusion features are determined; According to the flat features of the second fusion features and the corresponding importance weights, weighted sum features are obtained; According to the weighted sum features, future predictable features of the future predictable data enhanced by the static features are obtained.

2. The landslide hazard point land subsidence prediction method based on multi-modal data according to claim 1, characterized in that, The first neural network is a first gated GRN network; The inputting of the first static data and the second static data into the first neural network to obtain static features of the first static data and the second static data comprises: According to the gating linear unit and layer normalization unit of the first gated GRN network, flat features of input data are extracted; the input data is any first static data and the second static data; According to the exponential linear activation function of the first gated GRN network, importance weights corresponding to the flat features are determined; According to each of the flat features and the corresponding importance weights, weighted sum features are obtained. 3.The landslide hidden point land subsidence prediction method based on multi-modal data according to claim 1, characterized in that, Before the inputting of the time-varying continuous data and the static features into the second neural network to obtain time-varying features of the time-varying continuous data enhanced by the static features, further comprising: The static features are input into a fifth gated GRN network and a sixth gated GRN network respectively to obtain second context features of the static features output by the fifth gated GRN network and third context features of the static features output by the sixth gated GRN network; wherein the network structures of the fourth gated GRN network to the sixth gated GRN network are the same, and the network parameters are different; According to the weighted sum features, time-varying features of the time-varying continuous data enhanced by the static features are obtained, comprising: The weighted sum features are input into a long short-term memory network, and the second context features and the third context features are taken as cell states and hidden states of the long short-term memory network to obtain output first feature sequences; According to the first feature sequences, time-varying features of the time-varying continuous data enhanced by the static features are generated.

4. The landslide hazard point land subsidence prediction method based on multi-modal data according to claim 3, characterized in that, Before the inputting of the time-varying continuous data and the static features into the second neural network to obtain time-varying features of the time-varying continuous data enhanced by the static features, further comprising: The static features are input into a seventh gated GRN network to obtain fourth context features of the static features output by the seventh gated GRN network; the network structures of the fourth gated GRN network to the seventh gated GRN network are the same, and the network parameters are different; According to the first feature sequences, time-varying features of the time-varying continuous data enhanced by the static features are generated, comprising: The first feature sequence is enhanced according to the fourth context feature, to obtain a time-varying feature of the time-varying continuous data after enhancement of the static feature.

5. The method of claim 1, wherein the method is characterized by: Before the future predictable data and the static feature are input into the third gated GRN neural network to obtain a future predictable feature of the future predictable data after enhancement of the static feature, the method further includes: The static feature is input into a fifth gated GRN network and a sixth gated GRN network respectively, to obtain a second context feature of the static feature output by the fifth gated GRN network, and to obtain a third context feature of the static feature output by the sixth gated GRN network; the network structures of the fourth gated GRN network to the sixth gated GRN network are the same, and the network parameters are different; The future predictable feature of the future predictable data after enhancement of the static feature is obtained according to the weighted sum feature, including: The weighted sum feature is input into a long short-term memory network, and the second context feature and the third context feature are taken as cell states and hidden states of the long short-term memory network, to obtain a second feature sequence output; A future predictable feature of the future predictable data after enhancement of the static feature is generated according to the second feature sequence.

6. The landslide hazard point land subsidence prediction method based on multi-modal data according to claim 5, characterized in that, Before the future predictable data and the static feature are input into the third gated GRN neural network to obtain a future predictable feature of the future predictable data after enhancement of the static feature, the method further includes: The static feature is input into a seventh gated GRN network, to obtain a fourth context feature of the static feature output by the seventh gated GRN network; the network structures of the fourth gated GRN network to the seventh gated GRN network are the same, and the network parameters are different; The future predictable feature of the future predictable data after enhancement of the static feature is generated according to the second feature sequence, including: The second feature sequence is enhanced according to the fourth context feature, to obtain the future predictable feature of the future predictable data after enhancement of the static feature. 7.A landslide hazard point land subsidence prediction device based on multi-modal data, characterized by The device includes: The data acquisition module is configured to acquire multi-modal data of the rainfall-type landslide hidden danger point in response to a land subsidence prediction signal of the rainfall-type landslide hidden danger point. The data division module is configured to divide each modal data in the multi-modal data into at least one of first static data, second static data, time-varying continuous data, and future predictable data; the first static data is text modal data that does not change with time or text modal data that changes with time and has a fluctuation less than a first threshold value, the second static data is numerical modal data that does not change with time or numerical modal data that changes with time and has a fluctuation less than the first threshold value, the time-varying continuous data is modal data that continuously changes with time, and the future predictable data is modal data that can be numerically predicted in a future period of time. The data acquisition module is configured to acquire multi-modal data of the rainfall-type landslide hidden danger point in response to a land subsidence prediction signal of the rainfall-type landslide hidden danger point. The feature extraction module is configured to input the first static data and the second static data into a first neural network to obtain static features of the first static data and the second static data; input the static features into a fourth gated GRN network to obtain first context features outputting the static features; input the time-varying continuous data and the static features into a second neural network to obtain time-varying features of the time-varying continuous data enhanced by the static features; and input the future predictable data and the static features into a third neural network to obtain future predictable features of the future predictable data enhanced by the static features; the second neural network is a second gated GRN network; and the third neural network is a third gated GRN network. The inputting the time-varying continuous data and the static features into the second neural network to obtain the time-varying features of the time-varying continuous data enhanced by the static features includes: fusing the first context features and the time-varying continuous data according to a gating mechanism of the second gated GRN network to obtain first fusion features; extracting flattened features of the first fusion features according to a gating linear unit and a layer normalization unit of the second gated GRN network; determining importance weights corresponding to the flattened features of the first fusion features according to an exponential linear activation function of the second gated GRN network; performing weighted summation on the flattened features of the first fusion features and the importance weights corresponding to the flattened features to obtain weighted summation features; obtaining the time-varying features of the time-varying continuous data enhanced by the static features according to the weighted summation features; The inputting the future predictable data and the static features into the third gated GRN network to obtain the future predictable features of the future predictable data enhanced by the static features includes: fusing the first context features and the future predictable data according to a gating mechanism of the third gated GRN network to obtain second fusion features; extracting flattened features of the second fusion features according to a gating linear unit and a layer normalization unit of the third gated GRN network; determining importance weights corresponding to the flattened features of the second fusion features according to an exponential linear activation function of the third gated GRN network; performing weighted summation on the flattened features of the second fusion features and the importance weights corresponding to the flattened features to obtain weighted summation features; obtaining the future predictable features of the future predictable data enhanced by the static features according to the weighted summation features; The state prediction module is configured to integrate the time-varying features and the future predictable features according to a multi-head attention mechanism to obtain an attention matrix output by the multi-head attention mechanism, and predict a land subsidence state of the rainfall-induced landslide hidden danger point according to the attention matrix.

8. An electronic device, comprising: The method comprises at least one controller and a memory connected in communication with the controller; the memory stores instructions executable by the at least one controller, and the instructions are executed by the at least one controller to enable the at least one controller to perform the landslide hidden point land subsidence prediction method based on multi-modal data according to any one of claims 1 to 6.

Images