Slope intelligent inspection and early warning method and system based on space-time-multimodal fusion

Abstract

The application provides a kind of slope intelligent inspection early warning method and system based on space-time-multimodal fusion, related to slope monitoring technical field.Through the pretreatment of multi-source heterogeneous data, the extraction and fusion of multimodal features, the neural network prediction of physical constraints and the inspection adjustment driven by risk, a complete slope intelligent inspection early warning process is constructed, the complementary information of monitoring nodes, images and point cloud data is effectively integrated, the limitation of limited coverage of single sensor is avoided, the overall state of slope is comprehensively perceived and dynamically warned, thereby solving the technical problems of low efficiency of manual inspection and difficulty of overall state grasping by point sensor, and systematically solving the problems caused by insufficient space-time evolution law mining and shallow multimodal data fusion.

Description

Technical Field

[0001] This invention relates to the field of slope monitoring technology, and more specifically, to a method and system for intelligent slope inspection and early warning based on spatiotemporal-multimodal fusion. Background Technology

[0002] A slope is a sloping rock or soil mass with a certain angle, whether artificially excavated or naturally formed. Numerous slopes are distributed along linear transportation projects such as highways and railways, and their stability directly affects traffic safety and project reliability. Slope instability can lead to geological disasters such as landslides, collapses, and debris flows, causing serious casualties and economic losses.

[0003] Traditional slope monitoring mainly relies on regular manual inspections and physical sensors (such as inclinometers, displacement gauges, and pore water pressure gauges) deployed at key locations. Manual inspections suffer from low efficiency, high cost, high risk, and incomplete coverage; while point sensors offer high accuracy, their coverage is limited, making it difficult to comprehensively grasp the overall condition of the slope. Summary of the Invention

[0004] The problem that this invention aims to solve is that existing inspection and monitoring analysis methods have limited coverage and are difficult to fully grasp the overall condition of slopes.

[0005] To address the aforementioned problems, in a first aspect, this invention provides a method for intelligent slope inspection and early warning based on spatiotemporal-multimodal fusion, comprising:

[0006] The multi-source heterogeneous data obtained from the monitoring slope are preprocessed to obtain a multimodal dataset, which includes monitoring data from multiple monitoring nodes on the slope, slope images, and three-dimensional point cloud data of the slope.

[0007] Based on monitoring data from multiple monitoring nodes on the slope within multiple historical time steps, a spatiotemporal graph convolutional network is used to obtain spatiotemporal features;

[0008] Deep convolutional neural networks are used to extract features from slope images to obtain visual features;

[0009] Point cloud neural networks are used to process the three-dimensional point cloud data of the slope to obtain point cloud features;

[0010] Spatiotemporal features, point cloud features, and visual features are input into a spatiotemporal-multimodal hybrid Transformer fusion model to obtain the final fused features;

[0011] The expected slope location information, final fusion features, and geotechnical parameter information are input into the trained physical information neural network to obtain the displacement field, disease type, safety factor, and instability probability at the expected slope location. The preset rock mechanics constraints are fused into the multi-task loss function, and the physical information neural network is trained using the multi-task loss function.

[0012] The displacement field, safety factor, and instability probability are input into the risk model to obtain the risk score;

[0013] The slope inspection cycle is dynamically adjusted based on the risk score and preset thresholds.

[0014] Optionally, the step of obtaining spatiotemporal features by using a spatiotemporal graph convolutional network based on monitoring data from multiple monitoring nodes on the slope within multiple historical time steps includes:

[0015] Based on monitoring data from multiple monitoring nodes on the slope, a feature vector is constructed for each node at the monitoring time point. The feature vector includes the node's displacement, acceleration, pore water pressure, and water content.

[0016] Based on the relationship between the Euclidean distance and the distance threshold between two nodes, or whether there is a geomechanical association between two nodes, the spatial adjacency matrix between the nodes is obtained;

[0017] Construct a spatiotemporal graph based on node information, spatial adjacency matrix, and eigenvectors;

[0018] Spatiotemporal graphs from multiple time points are input into a stacked spatiotemporal graph convolutional network to obtain spatiotemporal features.

[0019] Optionally, multiple spatiotemporal graph convolutional networks are stacked. The spatiotemporal graph convolutional network includes a spatial graph convolutional layer and a temporal convolutional layer, which are connected in series. The spatial graph convolutional layer is used to aggregate features of nodes in the neighborhood at each node, and the temporal convolutional layer is used to extract the temporal dependencies between multiple time points before the current time point.

[0020] Optionally, the spatiotemporal-multimodal hybrid Transformer fusion model includes a point cloud-image cross-modal attention fusion layer, a graph attention network layer, and a collaborative attention layer.

[0021] The process involves inputting spatiotemporal features, point cloud features, and visual features into a spatiotemporal-multimodal hybrid Transformer fusion model to obtain the final fused features, including:

[0022] Point cloud features and visual features are input into a point cloud-image cross-modal attention fusion layer to obtain preliminary fused features;

[0023] The initial fused features are input into the graph attention network layer to obtain the enhanced fused features;

[0024] The enhanced fusion features and spatiotemporal features are input into the collaborative attention layer to obtain the final fusion features.

[0025] Optionally, the preliminary fusion feature is:

[0026]

[0027] in, Representing point cloud features, Point cloud features representing the fusion of image semantics, A query vector representing point cloud features. Key vectors representing visual features. A value vector representing visual features. Indicates visual characteristics, These represent the projection matrices corresponding to the query vector, key vector, and value vector, respectively.

[0028] Optionally, the enhanced fusion feature Each element inside is

[0029]

[0030] in, This represents the j-th element within the initial fusion features. The projection matrix of the graph attention network layer, The attention weights of the graph attention network layer;

[0031] Employing a bidirectional cross-attention mechanism, spatiotemporal feature queries enhance multimodal fusion features:

[0032]

[0033] in, Indicates spatiotemporal characteristics, These represent the projection matrices corresponding to the spatiotemporal feature query vector, the spatiotemporal feature key vector, and the spatiotemporal feature value vector, respectively. The dimension of the spatiotemporal feature query vector;

[0034] Multimodal enhanced fusion feature query spatiotemporal features:

[0035]

[0036] in, These represent the projection matrices corresponding to the enhanced fusion feature query vector, the enhanced fusion feature key vector, and the enhanced fusion feature value vector, respectively. This indicates the dimension of the enhanced fusion feature query vector;

[0037] The final fusion feature is:

[0038]

[0039] MLP stands for fully connected layer.

[0040] Optionally, the multi-task loss function is:

[0041]

[0042] in, Semantic segmentation loss representing disease categories, Indicates the safety factor for predicted loss. Indicates the safety factor for predicted loss. Cross-entropy loss represents the probability of instability. and These represent the weights corresponding to the semantic segmentation loss, safety factor prediction loss, safety factor prediction loss, and cross-entropy loss for instability probability, respectively, for disease categories.

[0043] The force equilibrium condition based on the limit equilibrium method is incorporated into the physical loss of the physical information neural network. The physical loss of the physical information neural network is defined as the residual of the force equilibrium equation at random sampling points:

[0044]

[0045] For any slope sliding surface, The anti-slip moment along the sliding surface, The sliding torque is the torque applied along the sliding surface. Indicates the slope safety factor. This indicates the total number of sliding surfaces on the slope. Indicates the point of collocation. The network parameters of a neural network representing physical information;

[0046] Semantic segmentation loss of disease categories for

[0047]

[0048] in, The true label of pixel i belonging to category c. This is the predicted probability, where C is the total number of categories. It is the total number of pixels;

[0049] Safety factor predicts loss for

[0050]

[0051] in, This refers to the number of observation points with actual FOS annotations. This represents the predicted value of the slope safety factor. This represents the true value of the slope safety factor.

[0052] Optionally, the risk model is

[0053]

[0054] in, Indicates the risk score. This represents the minimum safety factor for the current slope. This indicates the predicted probability of instability. Indicates displacement rate, These represent the weighting coefficients corresponding to the minimum safety factor, the predicted instability probability, and the displacement rate, respectively.

[0055] Optionally, the step of dynamically adjusting the slope inspection cycle based on the risk score and a preset threshold includes:

[0056] When the risk score is less than the first preset threshold, the slope inspection is adjusted to a regular inspection.

[0057] When the risk score is greater than or equal to the first preset threshold and less than the second preset threshold, the slope inspection is adjusted to a more frequent inspection.

[0058] When the risk score is greater than or equal to the second preset threshold, the slope inspection is adjusted to real-time monitoring inspection.

[0059] Secondly, the present invention also provides a slope intelligent inspection and early warning system based on spatiotemporal-multimodal fusion, comprising:

[0060] The data preprocessing module is used to preprocess the multi-source heterogeneous data obtained from the monitored slope to obtain a multimodal dataset. The multimodal data includes monitoring data from multiple monitoring nodes on the slope, slope images, and three-dimensional point cloud data of the slope.

[0061] The spatiotemporal feature extraction module is used to obtain spatiotemporal features based on monitoring data from multiple monitoring nodes on the slope within multiple historical time steps using a spatiotemporal graph convolutional network.

[0062] The visual feature extraction module is used to extract features from slope images using a deep convolutional neural network to obtain visual features;

[0063] The point cloud feature extraction module is used to process the three-dimensional point cloud data of the slope using a point cloud neural network to obtain point cloud features.

[0064] The feature fusion module is used to input spatiotemporal features, point cloud features, and visual features into the spatiotemporal-multimodal hybrid Transformer fusion model to obtain the final fused features;

[0065] The prediction module is used to input the expected slope location information, final fusion features and geotechnical parameter information into the trained physical information neural network to obtain the displacement field, disease type, safety factor and instability probability at the expected slope location. The preset rock mechanics constraints are fused into the multi-task loss function and the physical information neural network is trained using the multi-task loss function.

[0066] The inspection and adjustment module is used to input the displacement field, safety factor and instability probability into the risk model to obtain a risk score; it is also used to dynamically adjust the slope inspection cycle based on the risk score and preset thresholds.

[0067] This invention provides a method and system for intelligent slope inspection and early warning based on spatiotemporal-multimodal fusion. Compared with existing technologies, it has the following advantages:

[0068] By preprocessing multi-source heterogeneous data, extracting and fusing multimodal features, predicting physical constraints using neural networks, and adjusting inspections based on risks, a complete intelligent slope inspection and early warning process was constructed. This effectively integrates complementary information from monitoring nodes, images, and point cloud data, avoiding the limitations of single sensor coverage. It achieves comprehensive perception and dynamic early warning of the overall slope condition, thus solving the technical problems of low efficiency in manual inspections and the difficulty of point sensors in grasping the overall condition. It also systematically solves the problems caused by insufficient mining of spatiotemporal evolution laws and shallow fusion of multimodal data. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A flowchart illustrating a slope intelligent inspection and early warning method based on spatiotemporal-multimodal fusion provided in an embodiment of the present invention;

[0071] Figure 2 This is a schematic diagram of the structure of a slope intelligent inspection and early warning system based on spatiotemporal-multimodal fusion, provided in an embodiment of the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application are described clearly and completely. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0073] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0074] like Figure 1 As shown in the embodiment of this application, a slope intelligent inspection and early warning method based on spatiotemporal-multimodal fusion is provided, including:

[0075] S1: Preprocess the multi-source heterogeneous data obtained from the monitored slope to obtain a multimodal dataset. The multimodal data includes monitoring data from multiple monitoring nodes on the slope, slope images, and three-dimensional point cloud data of the slope.

[0076] S2: Based on the monitoring data of multiple monitoring nodes on the slope within multiple historical time steps, a spatiotemporal graph convolutional network is used to obtain spatiotemporal features.

[0077] S3: Use a deep convolutional neural network to extract features from the slope images to obtain visual features.

[0078] S4: Use a point cloud neural network to process the three-dimensional point cloud data of the slope to obtain point cloud features.

[0079] S5: Input the spatiotemporal features, point cloud features, and visual features into the spatiotemporal-multimodal hybrid Transformer fusion model to obtain the final fused features.

[0080] S6: Input the expected slope location information, final fusion features and geotechnical parameter information into the trained physical information neural network to obtain the displacement field, disease type, safety factor and instability probability at the expected slope location. The preset rock mechanics constraints are fused into the multi-task loss function, and the physical information neural network is trained using the multi-task loss function.

[0081] S7: Input the displacement field, safety factor, and instability probability into the risk model to obtain the risk score.

[0082] S8: Dynamically adjust the slope inspection cycle based on risk score and preset threshold.

[0083] In this optional embodiment, the multi-source heterogeneous data acquired from the monitored slope are first preprocessed to obtain a multimodal dataset. This dataset includes monitoring data from multiple monitoring nodes on the slope, slope images, and 3D point cloud data of the slope. Based on the monitoring data from multiple monitoring nodes on the slope within multiple historical time steps, a spatiotemporal graph convolutional network is used to extract spatiotemporal features. This process effectively captures the spatial transmission effect and temporal gradual evolution law of soil and rock deformation by constructing node feature vectors and spatial adjacency matrices, achieving joint modeling of spatial dependence and temporal dynamics between nodes, overcoming the deficiency of isolated analysis in being unable to identify gradual evolution laws. Simultaneously, the slope images are processed by a deep convolutional neural network to output visual features, which are used to analyze spatial details such as surface crack propagation, achieving efficient extraction of spatial details such as crack propagation and surface deformation in the images. The 3D point cloud data of the slope is processed by a point cloud neural network to generate point cloud features to quantify the changes in the 3D geometric structure of the slope, achieving accurate analysis of the changes in the 3D geometric structure of the slope. Subsequently, spatiotemporal features, point cloud features, and visual features are input into a spatiotemporal-multimodal hybrid Transformer fusion model. Through a cross-modal attention mechanism, deep interaction of multi-source information is achieved, forming a three-dimensional understanding of the slope condition. This enables the complementarity and fusion of multi-source information, avoiding the breakage of the physical causal chain caused by shallow splicing. Based on this, the expected slope location information, the final fused features, and geotechnical parameters are input into a trained physical information neural network. This network outputs displacement field, disease type, safety factor, and instability probability under preset rock mechanics constraints, where physical constraints are embedded in the training process through a multi-task loss function. Thus, the displacement field, safety factor, and instability probability are passed to the risk model to generate a risk score, and the slope inspection cycle is dynamically adjusted according to preset thresholds, achieving intensive monitoring of high-risk areas and resource optimization of low-risk areas. By preprocessing multi-source heterogeneous data, extracting and fusing multimodal features, predicting physical constraints using neural networks, and adjusting inspections based on risks, a complete intelligent slope inspection and early warning process was constructed. This effectively integrates complementary information from monitoring nodes, images, and point cloud data, avoiding the limitations of single sensor coverage. It achieves comprehensive perception and dynamic early warning of the overall slope condition, thus solving the technical problems of low efficiency in manual inspections and the difficulty of point sensors in grasping the overall condition. It also systematically solves the problems caused by insufficient mining of spatiotemporal evolution laws and shallow fusion of multimodal data.

[0084] The following is a detailed description of each step.

[0085] S1: Preprocess the multi-source heterogeneous data obtained from the monitored slope to obtain a multimodal dataset. The multimodal data includes monitoring data from multiple monitoring nodes on the slope, slope images, and three-dimensional point cloud data of the slope.

[0086] Specifically, industrial-grade drones can carry various sensor payloads to achieve rapid, safe, and high-resolution data acquisition of slopes. The drone platform is equipped with multimodal sensors such as RGB cameras, LiDAR, and thermal infrared cameras. Visible light cameras acquire high-definition images of the slopes for identifying surface defects such as erosion, cracks, and rockfalls. LiDAR can acquire high-precision 3D point cloud data (point density up to hundreds of points / square meter) to construct centimeter-level digital elevation models (DEMs) for deformation monitoring and hidden defect identification. Thermal infrared cameras acquire surface temperature fields to identify anomalies such as seepage channels and cavities. Hyperspectral imagers acquire information from dozens to hundreds of spectral bands for rock and soil mineral composition analysis and vegetation health monitoring. Ground-based IoT devices such as GNSS displacement gauges, pore water pressure gauges, and soil moisture sensors are deployed to construct an IoT sensor array. This ground-deployed IoT sensor array provides continuous time-series monitoring data. GNSS high-precision displacement gauges (millimeter-level accuracy), rain gauges, pore water pressure gauges, soil moisture sensors, inclinometers, etc., form a multi-source heterogeneous data acquisition system integrating air and ground.

[0087] Multi-source data are synchronized in time, unified in spatial coordinates, and normalized to establish a precise mapping relationship between point clouds and images, and to construct a multimodal dataset under a unified spatiotemporal reference.

[0088] S2: Based on the monitoring data of multiple monitoring nodes on the slope within multiple historical time steps, a spatiotemporal graph convolutional network is used to obtain spatiotemporal features.

[0089] Specifically, most existing slope monitoring technologies analyze monitoring data from different times and spatial locations as isolated events. For example, they use Long Short-Term Memory (LSTM) networks to predict the time series of a single displacement monitoring point, or Convolutional Neural Networks (CNNs) to identify defects in images at a single moment. This "seeing the point but not the whole, seeing the tree but not the forest" analytical paradigm completely ignores the fact that the slope is a continuous body, and its deformation field has strong spatial correlation (displacement in one area will affect neighboring areas through the soil and rock mass), and has an evolutionary pattern in time (a gradual process from creep to accelerated deformation to sudden instability). The lack of modeling ability for this spatiotemporal coupling effect makes the system unable to identify the critical precursors of the slope from quantitative to qualitative change, resulting in a serious deficiency in the early warning capability for sudden and catastrophic instability events.

[0090] Monitoring points on slopes are spatially adjacent and exhibit temporal evolutionary trends. Spatiotemporal Graph Convolutional Networks (ST-GCNs) model data as spatiotemporal graphs, and by performing convolution operations on the graph structure, they can simultaneously capture information propagation in the spatial dimension and dynamic evolution in the temporal dimension.

[0091] The specific content of step S2 includes:

[0092] S21: Based on the monitoring data at multiple monitoring nodes on the slope, construct a feature vector for each node at the monitoring time point. The feature vector includes the node's displacement, acceleration, pore water pressure, and water content.

[0093] Specifically, the slope is discretized into Each monitoring node (which can be a GNSS station, sensor location, or virtual grid point) is in... Within each historical time step, each node At any moment The state is determined by the feature vector This indicates that the monitoring data includes the node's displacement, acceleration, pore water pressure, and water content.

[0094] S22: Based on the relationship between the Euclidean distance and the distance threshold between two nodes, or whether there is a geomechanical association between the two nodes, obtain the spatial adjacency matrix between the nodes.

[0095] Specifically, constructing a spatial adjacency matrix If node and nodes The Euclidean distance is less than the threshold Or there may be a geomechanical connection, then ,otherwise Diagonal elements (Self-connection).

[0096] S23: Construct a spatiotemporal graph based on node information, spatial adjacency matrix, and eigenvectors.

[0097] Specifically, a spacetime diagram can be represented as ,in, It is a set of nodes. It is the spatiotemporal feature tensor of the node.

[0098] S24: Input the spatiotemporal graphs at multiple time points into a stacked spatiotemporal graph convolutional network to obtain spatiotemporal features.

[0099] Specifically, multiple spatiotemporal graph convolutional networks are stacked. Each spatiotemporal graph convolutional network includes spatial graph convolutional layers and temporal convolutional layers, which are cascaded. The spatial graph convolutional layers are used to aggregate features of nodes within the neighborhood at each node, while the temporal convolutional layers are used to extract temporal dependencies between multiple time points prior to the current time. Through the spatial graph convolutional layers, it is possible to learn how deformations in different regions of a slope influence each other (e.g., deformation at the top of the slope propagates to the bottom). The temporal convolutional layers enable the model to identify nonlinear evolution patterns from gradual to abrupt changes.

[0100] At a fixed time t, a graph convolution operation is performed on all nodes, using the Chebyshev polynomial approximation of spectral graph convolution (i.e., spatial graph convolution layer):

[0101]

[0102] in, It is the first The node feature matrix of the layer, It is the first The node feature matrix of the layer, It is the k-th order learnable weight matrix. It is a normalized adjacency matrix k-th power ( (It is a degree matrix) It is an activation function (such as ReLU). It is the order of the Chebyshev polynomial, controlling the receptive field. This formula implements "message passing" of node features on the graph. Each node aggregates its features... Features of nodes in the order neighborhood (through) (Control the aggregation range), through learnable weights. Perform feature transformation. The larger the node, the wider the neighborhood it can "see".

[0103] In the time dimension, one-dimensional causal convolution or gated temporal convolution (i.e., temporal convolutional layers) is used to extract temporal dependencies. For node i, its time window... Feature sequences within After temporal convolution:

[0104]

[0105] in, The kernel size is The one-dimensional convolution ensures that the output at time t depends only on time t and previous times (causality).

[0106] Concatenate spatial graph convolution and temporal convolution to form a spatiotemporal convolution block (ST-Conv Block):

[0107]

[0108] Stacking multiple ST-Conv Blocks forms a deep network, which ultimately outputs a spatiotemporal feature tensor. The feature vector of each node It contains comprehensive spatiotemporal evolution information of the node and its neighborhood within the historical time window.

[0109] In step S2, feature vectors are first constructed based on monitoring node data to capture multi-dimensional state information of each node. Then, a spatial adjacency matrix is ​​determined using Euclidean distance thresholds or geomechanical correlations to establish physically reasonable node connections. Next, node information, the adjacency matrix, and feature vectors are integrated into a spatiotemporal graph, forming a unified representation of spatial structure and time series. Finally, the spatiotemporal graph sequences from multiple time points are input into a stacked spatiotemporal graph convolutional network, achieving a synergistic effect of spatial feature aggregation and temporal dynamic extraction through hierarchical processing. This workflow design ensures a progressive transformation from raw data to high-level features. The construction of the spatial adjacency matrix incorporates geomechanical knowledge, making the graph structure more consistent with actual physical mechanisms. The construction of the spatiotemporal graph achieves an organic fusion of spatial and temporal dimensions. The hierarchical architecture of the stacked network effectively decouples and jointly models spatial dependencies and temporal dynamics, thereby enabling accurate identification of the critical evolution process of slope deformation from creep to acceleration.

[0110] S3: Use a deep convolutional neural network to extract features from the slope images to obtain visual features.

[0111] Specifically, a pre-trained deep convolutional neural network (such as ResNet-50 or EfficientNet-B4) is used as the visual feature extractor. The input is high-resolution slope imagery captured by a drone. Through multiple layers of convolution, pooling, and residual connections, high-dimensional visual feature maps are extracted. This includes semantic information such as the texture, color, and crack morphology of the slope.

[0112] S4: Use a point cloud neural network to process the three-dimensional point cloud data of the slope to obtain point cloud features.

[0113] Specifically, point cloud neural networks such as PointNet++ or KPConv are used to process the 3D point clouds acquired by the LiDAR. ,in, Includes spatial coordinates and reflection intensity. PointNet++ extracts local geometric features for each point through hierarchical point cloud sampling, grouping, and feature aggregation:

[0114]

[0115] in, Point The neighborhood of radius r is obtained. Finally, the point cloud feature matrix is ​​obtained. It includes geometric information such as the micro-topography of the slope, surface roughness, and local curvature.

[0116] In addition, by utilizing the drone's attitude parameters (position, attitude angle) and camera intrinsic and extrinsic parameters, a 3D-to-2D perspective projection transformation is performed for each 3D point. Find its corresponding pixel coordinates on the image :

[0117]

[0118] in, It is the camera intrinsic parameter matrix. It refers to the camera's extrinsic parameters (rotation and translation). for The three-dimensional coordinates of the point cloud are mapped. Through this mapping, each point in the point cloud can be associated with a corresponding region on the image feature map.

[0119] S5: Input the spatiotemporal features, point cloud features, and visual features into the spatiotemporal-multimodal hybrid Transformer fusion model (ST-Graph-Multimodal Transformer, STGMT) to obtain the final fused features.

[0120] Specifically, drones and ground sensors can acquire multi-source heterogeneous data on slopes, including high-resolution visible light images, 3D laser point clouds, thermal infrared temperature fields, displacement time series, pore water pressure, and soil moisture content. However, existing technologies often employ simple feature concatenation or decision-making layer voting at the data fusion level, a "hard fusion" approach that fails to achieve deep interaction and complementarity of information from different modalities. For example, it cannot automatically discover the complete physical causal chain of "heavy rainfall leading to a surge in soil moisture content → increased pore water pressure → redistribution of stress within the slope → accelerated propagation of surface cracks." This shallow fusion results in severe loss of information entropy, and the system's perception of slope safety status remains at the level of a "fragmented list of defects," failing to form a dynamic, three-dimensional, and accurate understanding of the overall safety situation.

[0121] The STGMT architecture enables deep information fusion across time, space, and modalities. STGMT consists of a three-stage attention mechanism, and the spatiotemporal-multimodal hybrid Transformer fusion model includes a point cloud-image cross-modal attention fusion layer, a graph attention network layer, and a collaborative attention layer.

[0122] The specific content of step S5 includes:

[0123] S51: Input point cloud features and visual features into the point cloud-image cross-modal attention fusion layer to obtain preliminary fused features.

[0124] Specifically, in Phase 1, point cloud-image cross-modal attention fusion, the goal is to fuse the visual semantic information of the corresponding image region for each point cloud point.

[0125] For point i in the point cloud, its geometric features are: Mapping to image coordinates and extracting the corresponding image features. From feature maps of Location. A cross-attention mechanism is employed, using point cloud features as the query and image features as the key and value. Preliminary feature fusion:

[0126]

[0127] in, Representing point cloud features, Point cloud features representing the fusion of image semantics, A query vector representing point cloud features. Key vectors representing visual features. A value vector representing visual features. Indicates visual characteristics, These represent the projection matrices corresponding to the query vector, key vector, and value vector, respectively. Calculate the similarity score matrix between point cloud features and image features. Normalization yields attention weights, indicating which regions of the image each point should focus on. Through this mechanism, each point in the point cloud not only contains geometric information but also incorporates visual information such as color and texture from the corresponding image region, forming preliminary fusion features.

[0128] S52: Input the preliminary fused features into the graph attention network layer to obtain enhanced fused features.

[0129] Specifically, Stage 2: Graph attention propagation, which integrates the fused features obtained in Stage 1. The graph nodes are treated as new features (note: this graph can be a spatial subgraph of the original spatiotemporal graph, or a new graph constructed based on the proximity relationships of the point cloud), and feature propagation and refinement are performed using a Graph Attention Network (GAT).

[0130] The enhanced fusion features Each element inside is

[0131]

[0132] in, This represents the j-th element within the initial fusion features. The projection matrix of the graph attention network layer, The attention weights for the graph attention network layer are shown. GAT allows different neighboring nodes to contribute different weights to the central node, achieving adaptive information aggregation. This stage enables the fused features to propagate and smooth in space, enhancing the spatial consistency of the features.

[0133] S53: Input the enhanced fusion features and spatiotemporal features into the collaborative attention layer to obtain the final fusion features.

[0134] Specifically, Stage 3: Cooperative attention of spatiotemporal features and multimodal features. This involves integrating the spatial multimodal fusion features obtained in Stage 2. Spatiotemporal features extracted by the aforementioned spatiotemporal graph convolutional network Perform collaborative attention fusion.

[0135]

[0136] in, Indicates spatiotemporal characteristics, These represent the projection matrices corresponding to the spatiotemporal feature query vector, the spatiotemporal feature key vector, and the spatiotemporal feature value vector, respectively. This represents the dimension of the spatiotemporal feature query vector.

[0137] Multimodal enhanced fusion feature query spatiotemporal features:

[0138]

[0139] in, These represent the projection matrices corresponding to the enhanced fusion feature query vector, the enhanced fusion feature key vector, and the enhanced fusion feature value vector, respectively. This indicates the dimension of the enhanced fusion feature query vector.

[0140] The final fusion feature is:

[0141]

[0142] MLP stands for fully connected layer.

[0143] A bidirectional cross-attention mechanism is employed to achieve reciprocal information interaction between spatiotemporal features and multimodal enhanced fusion features. On one hand, spatiotemporal features interact as query vectors with the key and value vectors of multimodal enhanced fusion features, enabling time-series features to dynamically focus on key areas in spatial data. On the other hand, multimodal enhanced fusion features interact as query vectors with the key and value vectors of spatiotemporal features, enabling spatial features to proactively retrieve key states in the time dimension. Finally, the output of the bidirectional attention mechanism is integrated through a fully connected layer to form a unified feature representation that retains the ability of spatiotemporal features to characterize gradual deformation while incorporating the sensitivity of multimodal features to sudden disease outbreaks, thereby achieving a three-dimensional understanding of the overall safety status of the slope.

[0144] Through a three-stage attention mechanism, information from different modalities is gradually integrated and complemented from the bottom layer to the top layer, achieving deep cross-modal interaction. The attention mechanism can adaptively adjust the contribution weights of different modalities according to the actual situation. For example, during periods of heavy rainfall, the model automatically increases the weight of hydrological information in the spatiotemporal features. The final fused features contain comprehensive information including geometric, visual, and spatiotemporal dynamics, resulting in highly condensed information that provides strong feature support for downstream tasks.

[0145] In step S5, a semantic mapping relationship between point cloud features and visual features is first established through a point cloud-image cross-modal attention fusion layer, enabling a physical association between three-dimensional geometric information and two-dimensional image content. Subsequently, a graph attention network layer is used to perform topological enhancement on the preliminary fused features, utilizing the graph structure propagation mechanism to strengthen the stress distribution features in the local area of ​​the slope. Finally, a collaborative attention layer is used to achieve bidirectional interaction between spatiotemporal features and enhanced fused features, enabling the displacement field's time series to dynamically adjust the intensity of attention to multimodal features, thereby fully preserving the correlation of the three dimensions of spatial geometry, visual semantics, and temporal evolution, providing input features with physical causal chain expression capabilities for subsequent physical information neural networks.

[0146] S6: Input the expected slope location information, final fused features, and geotechnical parameter information into the trained Physical Information Neural Network (PINN) to obtain the displacement field, disease type, safety factor, and instability probability at the expected slope location. Pre-defined rock mechanics constraints are fused into a multi-task loss function, which is then used to train the PINN. A Physical Information Neural Network (PINN) is introduced, with the geotechnical mechanics governing equations embedded as soft constraints in the neural network's loss function.

[0147] Specifically, PINN is a fully connected neural network (MLP). Its inputs include: spatial coordinates (location on the slope); the feature vector corresponding to the location in the final fused features; external parameters: rainfall, soil moisture content, and pore water pressure; and material parameters: cohesion, internal friction angle, and unit weight. The network outputs include: the displacement field, stress field (optional), safety factor, disease type, and instability probability at that location.

[0148] The multi-task loss function is:

[0149]

[0150] in, Semantic segmentation loss representing disease categories, Indicates the safety factor for predicted loss. Indicates the safety factor for predicted loss. Cross-entropy loss represents the probability of instability. and These represent the weights corresponding to the semantic segmentation loss, safety factor prediction loss, safety factor prediction loss, and cross-entropy loss for instability probability, respectively, for disease categories.

[0151] For slope stability analysis, the force equilibrium conditions based on the limit equilibrium method are incorporated into the physical loss of the physical information neural network. The physical loss of the physical information neural network is defined as the residual of the force equilibrium equation at random sampling points:

[0152]

[0153] For any slope sliding surface, The anti-slip moment along the sliding surface, The sliding torque is the torque applied along the sliding surface. Indicates the slope safety factor. This indicates the total number of sliding surfaces on the slope. Indicates the point of collocation. The network parameters of the physical information neural network are represented.

[0154] In addition to stability prediction, the system also needs to complete the task of disease identification. Semantic segmentation loss for disease categories is also required. for

[0155]

[0156] in, The true label of pixel i belonging to category c. C is the predicted probability, and C is the total number of categories (such as "stable", "crack", "erosion" etc.). It represents the total number of pixels.

[0157] Safety factor predicts loss for

[0158]

[0159] in, This refers to the number of observation points with actual FOS annotations. This represents the predicted value of the slope safety factor. This represents the true value of the slope safety factor.

[0160] Minimize using backpropagation and gradient descent algorithms (such as the Adam optimizer) The entire network is trained. During the inference phase, for any location on the slope, the network can output the type of damage, safety factor, and instability probability at that location, forming a spatially continuous risk situation map.

[0161] Because physical constraints are embedded, the model's predictions automatically satisfy the principles of geotechnical mechanics, avoiding the counterintuitive outputs of purely data-driven models and improving physical reliability. Physical constraints provide additional "monitoring signals," enabling the model to make reasonable predictions even in data-sparse regions. The model's output, such as the FOS (slope safety factor) distribution and slip surface morphology, has clear physical meaning, making it easy for engineers to understand and trust, and the output results are highly interpretable.

[0162] S7: Input the displacement field, safety factor, and instability probability into the risk model to obtain the risk score.

[0163] The risk model is as follows:

[0164]

[0165] in, Indicates the risk score. This represents the minimum safety factor for the current slope. This indicates the predicted probability of instability. Indicates displacement rate, These represent the weighting coefficients corresponding to the minimum safety factor, the predicted instability probability, and the displacement rate, respectively. To improve monitoring efficiency, the system adaptively adjusts the inspection frequency based on the real-time risk score assessment results.

[0166] S8: Dynamically adjust the slope inspection cycle based on the risk score and preset thresholds. This step specifically includes:

[0167] When the risk score is less than the first preset threshold, it is judged as low risk, and the slope inspection is adjusted to a regular inspection, once a week.

[0168] When the risk score is greater than or equal to the first preset threshold and less than the second preset threshold, it is judged as medium risk, and the slope inspection is adjusted to intensive inspection, once a day.

[0169] When the risk score is greater than or equal to the second preset threshold, it is judged as high risk, and the slope inspection is adjusted to real-time monitoring inspection, once per hour or continuously.

[0170] By establishing a quantitative mapping relationship between risk scores and inspection strategies, a closed-loop feedback mechanism is formed. The risk score, as a comprehensive safety status indicator, is input to the decision-making module in real time. The decision-making module dynamically compares the risk score with preset thresholds and automatically triggers the corresponding inspection strategy switching logic based on the comparison results. When the risk score is below the first preset threshold, the system maintains the regular inspection cycle to conserve resources; when the risk score enters the medium-risk range, the system automatically shortens the inspection interval to increase monitoring density; when the risk score reaches the high-risk threshold, the system immediately activates real-time monitoring mode to ensure immediate response. This mechanism achieves precise matching between monitoring resources and risk levels by adaptively adjusting the inspection intensity driven by risk status.

[0171] like Figure 2 As shown in the figure, an embodiment of this application provides a slope intelligent inspection and early warning system based on spatiotemporal-multimodal fusion, comprising:

[0172] The data preprocessing module 10 is used to preprocess the multi-source heterogeneous data obtained from the monitored slope to obtain a multimodal dataset. The multimodal data includes monitoring data from multiple monitoring nodes on the slope, slope images, and three-dimensional point cloud data of the slope.

[0173] The spatiotemporal feature extraction module 20 is used to obtain spatiotemporal features based on monitoring data from multiple monitoring nodes on the slope within multiple historical time steps, using a spatiotemporal graph convolutional network.

[0174] The visual feature extraction module 30 is used to extract features from the slope image using a deep convolutional neural network to obtain visual features.

[0175] The point cloud feature extraction module 40 is used to process the three-dimensional point cloud data of the slope using a point cloud neural network to obtain point cloud features.

[0176] The feature fusion module 50 is used to input spatiotemporal features, point cloud features and visual features into the spatiotemporal-multimodal hybrid Transformer fusion model to obtain the final fused features.

[0177] The prediction module 60 is used to input the expected slope location information, final fusion features and soil and rock parameter information into the trained physical information neural network to obtain the displacement field, disease type, safety factor and instability probability at the expected slope location. The preset rock mechanics constraints are fused into the multi-task loss function and the physical information neural network is trained using the multi-task loss function.

[0178] The inspection and adjustment module 70 is used to input the displacement field, safety factor and instability probability into the risk model to obtain a risk score; it is also used to dynamically adjust the slope inspection cycle based on the risk score and preset threshold.

[0179] In this embodiment, by combining a spatiotemporal graph convolutional network with a spatiotemporal-multimodal hybrid Transformer fusion model in a deep collaborative manner, and introducing rock mechanics constraints into the training process of the physical information neural network, joint modeling of the strong spatial correlation transmission effect and the gradual evolution law of slope deformation field is achieved. Simultaneously, the physical causal chains between multi-source heterogeneous data are automatically mined, achieving the core advantages of accurately identifying critical precursor signals of slope changes from quantitative to qualitative and constructing a dynamic three-dimensional safety situation awareness system. Specifically, the spatiotemporal feature extraction module aggregates neighborhood features through spatial graph convolutional layers and captures causal dependencies through temporal convolutional layers, effectively overcoming the problem of missed reporting of sudden disasters caused by neglecting spatial correlation in traditional methods; the feature fusion module utilizes a point cloud-image cross-modal attention layer to achieve deep interaction of multimodal features, avoiding information entropy loss caused by shallow fusion; the prediction module integrates rock mechanics constraints into a multi-task loss function, ensuring that displacement field distribution and safety factor prediction strictly follow physical laws, significantly improving the reliability of early warning. Through the above technical solutions, the system not only solves the problem of early warning failure caused by insufficient mining of spatiotemporal evolution laws, but also realizes comprehensive perception and quantitative assessment of the overall safety status of the slope, providing a scientific basis for dynamically optimizing the allocation of inspection resources.

[0180] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0181] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for intelligent slope inspection and early warning based on spatiotemporal-multimodal fusion, characterized in that, include: The multi-source heterogeneous data obtained from the monitoring slope are preprocessed to obtain a multimodal dataset, which includes monitoring data from multiple monitoring nodes on the slope, slope images, and three-dimensional point cloud data of the slope. Based on monitoring data from multiple monitoring nodes on the slope within multiple historical time steps, a spatiotemporal graph convolutional network is used to obtain spatiotemporal features; Deep convolutional neural networks are used to extract features from slope images to obtain visual features; Point cloud neural networks are used to process the three-dimensional point cloud data of the slope to obtain point cloud features; Spatiotemporal features, point cloud features, and visual features are input into a spatiotemporal-multimodal hybrid Transformer fusion model to obtain the final fused features; The expected slope location information, final fusion features, and geotechnical parameter information are input into the trained physical information neural network to obtain the displacement field, disease type, safety factor, and instability probability at the expected slope location. The preset rock mechanics constraints are fused into the multi-task loss function, and the physical information neural network is trained using the multi-task loss function. The displacement field, safety factor, and instability probability are input into the risk model to obtain the risk score; The slope inspection cycle is dynamically adjusted based on the risk score and preset threshold. The spatiotemporal-multimodal hybrid Transformer fusion model includes a point cloud-image cross-modal attention fusion layer, a graph attention network layer, and a collaborative attention layer. The process involves inputting spatiotemporal features, point cloud features, and visual features into a spatiotemporal-multimodal hybrid Transformer fusion model to obtain the final fused features, including: Point cloud features and visual features are input into a point cloud-image cross-modal attention fusion layer to obtain preliminary fused features; The initial fused features are input into the graph attention network layer to obtain the enhanced fused features; The enhanced fusion features and spatiotemporal features are input into the collaborative attention layer to obtain the final fusion features.

2. The intelligent slope inspection and early warning method based on spatiotemporal-multimodal fusion as described in claim 1, characterized in that, The process involves using a spatiotemporal graph convolutional network to obtain spatiotemporal features based on monitoring data from multiple monitoring nodes on the slope across multiple historical time steps. These features include: Based on monitoring data from multiple monitoring nodes on the slope, a feature vector is constructed for each node at the monitoring time point. The feature vector includes the node's displacement, acceleration, pore water pressure, and water content. Based on the relationship between the Euclidean distance and the distance threshold between two nodes, or whether there is a geomechanical association between two nodes, the spatial adjacency matrix between the nodes is obtained; Construct a spatiotemporal graph based on node information, spatial adjacency matrix, and eigenvectors; Spatiotemporal graphs from multiple time points are input into a stacked spatiotemporal graph convolutional network to obtain spatiotemporal features.

3. The intelligent slope inspection and early warning method based on spatiotemporal-multimodal fusion as described in claim 2, characterized in that, Multiple spatiotemporal graph convolutional networks are stacked. The spatiotemporal graph convolutional network includes a spatial graph convolutional layer and a temporal convolutional layer. The spatial graph convolutional layer is used to aggregate the features of the neighboring nodes at each node, and the temporal convolutional layer is used to extract the temporal dependencies between multiple time points before the current time point.

4. The intelligent slope inspection and early warning method based on spatiotemporal-multimodal fusion as described in claim 1, characterized in that, The preliminary fusion features are: in, Representing point cloud features, Point cloud features representing the fusion of image semantics, A query vector representing point cloud features. Key vectors representing visual features. A value vector representing visual features. Indicates visual characteristics, These represent the projection matrices corresponding to the query vector, key vector, and value vector, respectively.

5. The intelligent slope inspection and early warning method based on spatiotemporal-multimodal fusion as described in claim 4, characterized in that, The enhanced fusion features Each element inside is in, This represents the j-th element within the initial fusion features. The projection matrix of the graph attention network layer, The attention weights of the graph attention network layer; Employing a bidirectional cross-attention mechanism, spatiotemporal feature queries enhance multimodal fusion features: in, Indicates spatiotemporal characteristics, These represent the projection matrices corresponding to the spatiotemporal feature query vector, the spatiotemporal feature key vector, and the spatiotemporal feature value vector, respectively. The dimension of the spatiotemporal feature query vector; Multimodal enhanced fusion feature query spatiotemporal features: in, These represent the projection matrices corresponding to the enhanced fusion feature query vector, the enhanced fusion feature key vector, and the enhanced fusion feature value vector, respectively. This indicates the dimension of the enhanced fusion feature query vector; The final fusion feature is: MLP stands for fully connected layer.

6. The intelligent slope inspection and early warning method based on spatiotemporal-multimodal fusion as described in claim 1, characterized in that, The multi-task loss function is: in, Semantic segmentation loss representing disease categories, Indicates the safety factor for predicted loss. The physical loss of a neural network representing physical information. Cross-entropy loss represents the probability of instability. and These represent the weights corresponding to the semantic segmentation loss for disease categories, the safety factor prediction loss, the physical loss of the physical information neural network, and the cross-entropy loss for instability probability, respectively. The force equilibrium condition based on the limit equilibrium method is incorporated into the physical loss of the physical information neural network. The physical loss of the physical information neural network is defined as the residual of the force equilibrium equation at random sampling points: For any slope sliding surface, The anti-slip moment along the sliding surface, The sliding torque is the torque applied along the sliding surface. Indicates the slope safety factor. This indicates the total number of sliding surfaces on the slope. Indicates the point of collocation. The network parameters of a neural network representing physical information; Semantic segmentation loss of disease categories for in, The true label of pixel i belonging to category c. This is the predicted probability, where C is the total number of categories. It is the total number of pixels; Safety factor predicts loss for in, This refers to the number of observation points with actual FOS annotations. This represents the predicted value of the slope safety factor. This represents the true value of the slope safety factor.

7. The intelligent slope inspection and early warning method based on spatiotemporal-multimodal fusion as described in claim 1, characterized in that, The risk model is as follows: in, Indicates the risk score. This represents the minimum safety factor for the current slope. This indicates the predicted probability of instability. Indicates displacement rate, These represent the weighting coefficients corresponding to the minimum safety factor, the predicted instability probability, and the displacement rate, respectively.

8. The intelligent slope inspection and early warning method based on spatiotemporal-multimodal fusion as described in claim 7, characterized in that, The dynamic adjustment of the slope inspection cycle based on risk scores and preset thresholds includes: When the risk score is less than the first preset threshold, the slope inspection is adjusted to a regular inspection. When the risk score is greater than or equal to the first preset threshold and less than the second preset threshold, the slope inspection is adjusted to a more frequent inspection. When the risk score is greater than or equal to the second preset threshold, the slope inspection is adjusted to real-time monitoring inspection.

9. A slope intelligent inspection and early warning system based on spatiotemporal-multimodal fusion, characterized in that, include: The data preprocessing module is used to preprocess the multi-source heterogeneous data obtained from the monitored slope to obtain a multimodal dataset. The multimodal data includes monitoring data from multiple monitoring nodes on the slope, slope images, and three-dimensional point cloud data of the slope. The spatiotemporal feature extraction module is used to obtain spatiotemporal features based on monitoring data from multiple monitoring nodes on the slope within multiple historical time steps using a spatiotemporal graph convolutional network. The visual feature extraction module is used to extract features from slope images using a deep convolutional neural network to obtain visual features; The point cloud feature extraction module is used to process the three-dimensional point cloud data of the slope using a point cloud neural network to obtain point cloud features. The feature fusion module is used to input spatiotemporal features, point cloud features, and visual features into the spatiotemporal-multimodal hybrid Transformer fusion model to obtain the final fused features; The prediction module is used to input the expected slope location information, final fusion features and geotechnical parameter information into the trained physical information neural network to obtain the displacement field, disease type, safety factor and instability probability at the expected slope location. The preset rock mechanics constraints are fused into the multi-task loss function and the physical information neural network is trained using the multi-task loss function. The inspection and adjustment module is used to input the displacement field, safety factor and instability probability into the risk model to obtain a risk score; it is also used to dynamically adjust the slope inspection cycle based on the risk score and preset thresholds. The spatiotemporal-multimodal hybrid Transformer fusion model includes a point cloud-image cross-modal attention fusion layer, a graph attention network layer, and a collaborative attention layer. The process involves inputting spatiotemporal features, point cloud features, and visual features into a spatiotemporal-multimodal hybrid Transformer fusion model to obtain the final fused features, including: Point cloud features and visual features are input into a point cloud-image cross-modal attention fusion layer to obtain preliminary fused features; The initial fused features are input into the graph attention network layer to obtain the enhanced fused features; The enhanced fusion features and spatiotemporal features are input into the collaborative attention layer to obtain the final fusion features.

Images