Multi-modal slope deformation early warning method, electronic equipment and storage medium

Abstract

The invention discloses a multi-modal slope deformation early warning method, electronic equipment and a storage medium, and relates to the technical field of industrial data processing, and the method comprises the steps: obtaining multi-modal slope monitoring data, carrying out the preprocessing of the multi-modal slope monitoring data, obtaining the preprocessed data of each modal, and carrying out the early warning of the deformation of the multi-modal slope. And inputting the preprocessed modal data into a multi-modal prediction segmentation model, and carrying out coding, fusion, joint reasoning prediction and segmentation on the features to obtain a slope deformation risk early warning graph. Due to the fact that the preprocessed multi-modal data are input into the multi-modal prediction segmentation model for feature coding, fusion and joint reasoning prediction, the model can deeply mine internal association and causal laws among multi-source information, the depicting capacity of a slope instability mechanism is enhanced, and due to the fact that the method does not depend on a specific sensor type, the stability of a slope is improved. Various modal data are flexibly processed through a unified framework, extensible fusion from a finite modal to an N modal is realized, and the accuracy of slope deformation early warning is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of industrial data processing technology, and in particular to a multimodal slope deformation early warning method, electronic equipment, and storage medium. Background Technology

[0002] Accurate prediction of slope deformation in open-pit coal mines is crucial for disaster early warning and risk management, but the complex environment presents numerous challenges to prediction. Current deep learning-based prediction methods mostly rely on single-modal data for prediction. While these methods have achieved some degree of prediction in specific scenarios, they neglect the potential relationships between deformation and other influencing factors due to modal isolation.

[0003] Current early warning methods mostly rely on single data modes for prediction. While they can reflect some deformation trends, the failure to effectively integrate multimodal information makes it difficult for models to capture the complete driving mechanism and precursor features of slope instability, thus hindering a fundamental improvement in early warning accuracy. Furthermore, the deep coupling between model architecture and data modes limits model generalization ability, and data fusion methods suffer from insufficient modal breadth and poor compatibility with heterogeneous modes such as imagery, waveforms, and point clouds, making it difficult to support intuitive risk assessment and on-site decision-making.

[0004] Therefore, improving the accuracy of slope deformation early warning is an urgent problem to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a multimodal slope deformation early warning method, electronic device and storage medium, aiming to solve the technical problem of how to improve the accuracy of slope deformation early warning.

[0006] To achieve the above objectives, this application proposes a multimodal slope deformation early warning method, the method comprising: Acquire slope monitoring data in multiple modes; The slope monitoring data of the various modes are preprocessed to obtain preprocessed data of each mode. The preprocessing is used to unify the spatiotemporal reference of the data. The preprocessed modal data are input into a multimodal prediction and segmentation model to encode, fuse, jointly infer and predict, and segment the features to obtain a slope deformation risk warning map.

[0007] In one embodiment, the slope monitoring data of multiple modalities includes time-series monitoring data, radar monitoring data, optical image monitoring data, underground monitoring data, and geological model monitoring data. The step of preprocessing the slope monitoring data of multiple modalities to obtain preprocessed modal data includes: Time-series monitoring data are resampled and missing values ​​are imputed to obtain preprocessed time-series monitoring data. The radar monitoring data is subjected to coordinate transformation, spatial downsampling and interpolation to generate radar images. The radar images are then filled with decoherent regions and resampled over time to obtain preprocessed radar monitoring data. The optical image monitoring data is spatially resampled and aligned with the spatial grid of the radar image to obtain preprocessed optical image monitoring data. The underground monitoring data is timestamped and aligned with the time series of the time-series monitoring data to obtain preprocessed underground monitoring data. The geological model monitoring data is spatially aligned with the radar monitoring data using coordinate system one to obtain preprocessed geological model monitoring data.

[0008] In one embodiment, the step of inputting the preprocessed modal data into a multimodal prediction and segmentation model, encoding, fusing, jointly inferring and predicting the preprocessed modal data, and segmenting it to obtain a slope deformation risk early warning map includes: The preprocessed modal data are respectively feature-encoded to obtain the feature representation of each modality; The feature representations of each modality are mapped to a unified feature space and then fused to obtain joint features. Based on the joint features, a text description of the risk area is generated through a large language model. The visual feature representation, learnable mask, and text description of the risk area are input into a conditional mask generator. By performing pixel-level segmentation on areas of different risk levels, a slope deformation risk assessment map is generated.

[0009] In one embodiment, the step of performing feature encoding on each of the preprocessed modal data to obtain the feature representation of each modality includes: The preprocessed time-series monitoring data and the preprocessed underground monitoring data are time-series encoded, and time-series feature representations are obtained through patch partitioning, linear projection, and attention alignment operations. The preprocessed radar monitoring data is spatiotemporally encoded, and spatiotemporal feature representations are obtained through patch partitioning, position encoding, vector projection, and attention alignment operations. The preprocessed optical image monitoring data is visually encoded, and visual feature representations are obtained through patch segmentation, position encoding, and image encoding. The preprocessed geological model monitoring data is geometrically encoded, and geometric feature representations are obtained through downsampling and transformer encoding.

[0010] In one embodiment, the step of performing time-series encoding on the preprocessed time-series monitoring data and the preprocessed underground monitoring data, and obtaining a time-series feature representation through patch partitioning, linear projection, and attention alignment operations, includes: The preprocessed time-series monitoring data and the preprocessed underground monitoring data are subjected to reversible instance normalization to obtain normalized time-series data. The normalized time series data is divided into patch blocks, and the patch blocks are linearly projected and embedded to obtain the time series patch embedding. By aligning the text embedding with the temporal patch embedding using an attention mechanism, a temporal feature representation is obtained.

[0011] In one embodiment, the step of performing spatiotemporal encoding on the preprocessed radar monitoring data, and obtaining a spatiotemporal feature representation through patch partitioning, location encoding, vector projection, and attention alignment operations, includes: Each frame of the preprocessed radar monitoring data is segmented into multiple non-overlapping patch blocks, and vector projection is performed on the multiple non-overlapping patch blocks to obtain patch embedding. Spatial location encoding and temporal location encoding are added to the patch embedding respectively to obtain the spatiotemporal patch block embedding; The spatiotemporal patch block embedding is subjected to patch discarding processing to obtain the processed patch embedding; The processed patch embedding is subjected to spatiotemporal coding to obtain coded features, and the coded features are subjected to temporal pooling to obtain spatiotemporal feature representations.

[0012] In one embodiment, the step of geometrically encoding the preprocessed geological model monitoring data, and obtaining a geometric feature representation through downsampling and transformer encoding, includes: The preprocessed geological model monitoring data is converted to obtain a point cloud representation; The point cloud representation is downsampled to obtain a downsampled point cloud; The downsampled point cloud is encoded by a transformer, and the geometric relationships between the points are modeled to obtain a geometric feature representation.

[0013] In one embodiment, the step of generating a text description of the risk area based on the joint features using a large language model, and inputting the visual feature representation, a learnable mask, and the text description of the risk area into a conditional mask generator to generate a slope deformation risk assessment map by performing pixel-level segmentation of areas with different risk levels, includes: The task instruction prompts and category prompts are encoded to obtain the task embedding vector; The task embedding vector and the joint feature are input into a large language model for joint inference, and a textual description of the risk area is output. The visual feature representation, the learnable mask, and the text description of the risk area are input into the conditional mask generator to generate a segmentation mask and a category label. The segmentation mask and the category label are merged to obtain the final category label for each pixel; Based on the final category label of each pixel, the areas with different risk levels are segmented at the pixel level to generate a slope deformation risk assessment map.

[0014] In addition, to achieve the above objectives, this application also proposes an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multimodal slope deformation early warning method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the multimodal slope deformation early warning method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: Multiple modal slope monitoring data were acquired and preprocessed to obtain preprocessed data for each modality. This preprocessing was used to unify the spatiotemporal reference of the data. By acquiring and preprocessing multiple modal slope monitoring data to unify the spatiotemporal reference, the information limitations of a single modality were overcome from the data source perspective. This provides the model with a more comprehensive description of the slope condition, thus laying the foundation for solving prediction biases caused by incomplete information.

[0017] The preprocessed modal data is input into a multimodal prediction and segmentation model for feature encoding, fusion, joint inference prediction, and segmentation, resulting in a slope deformation risk early warning map. By inputting the preprocessed multimodal data into the multimodal prediction and segmentation model for feature encoding, fusion, joint inference prediction, and segmentation, the model can deeply explore the intrinsic correlations and causal patterns among multiple sources of information such as deformation sequences, groundwater levels, and microseismic activity, rather than analyzing single signals in isolation. This enhances the ability to characterize slope instability mechanisms and significantly improves the causality and accuracy of predictions. Since the model architecture itself does not depend on specific sensor types and can flexibly process heterogeneous data such as images, waveforms, and point clouds through a unified framework, it achieves scalable fusion from finite modes to N modes. This allows the model to adapt to the differentiated monitoring conditions of different mining areas. By continuously optimizing its performance using richer data resources, its generalization ability is improved, ultimately outputting a slope deformation risk early warning map with higher accuracy and more intuitive spatial representation, providing a reliable basis for accurate early warning and decision-making.

[0018] Task instruction prompts and category prompts are encoded to obtain task embedding vectors. These task embedding vectors, along with joint features, are input into a large language model for joint inference, outputting textual descriptions of risk areas. Visual feature representations, learnable masks, and textual descriptions of risk areas are input into a conditional mask generator to generate segmentation masks and category labels. The segmentation masks and category labels are merged to obtain the final category label for each pixel. Based on the final category label for each pixel, different risk level areas are segmented at the pixel level to generate a slope deformation risk assessment map. Encoding task instruction prompts and category prompts into task embedding vectors and inputting them along with joint features into a large language model for joint inference, the natural language instructions explicitly constrain the model's output format and content scope, generating textual descriptions that include risk location, level, and mechanistic explanations. This solves the problem of traditional methods lacking semantic information and causal explanations in their early warning results. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the multimodal slope deformation early warning method of this application; Figure 2This is a flowchart illustrating the second embodiment of the multimodal slope deformation early warning method of this application; Figure 3 This is a schematic diagram of the multimodal prediction segmentation model structure in an embodiment of this application; Figure 4 This is a schematic diagram of the conditional mask generator structure in an embodiment of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, the multimodal slope deformation early warning method, multimodal slope deformation early warning device, multimodal slope deformation early warning equipment and storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0025] It should be noted that the multimodal slope deformation early warning method can be applied to the terminal, and can be executed by the hardware or software in the terminal.

[0026] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0027] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0028] This application provides a multimodal slope deformation early warning method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multimodal slope deformation early warning method of this application.

[0029] In this embodiment, the multimodal slope deformation early warning method includes steps S10~S30: Step S10: Obtain slope monitoring data for multiple modes.

[0030] For example, multimodal slope monitoring data can include, but is not limited to, UAV topographic imagery, UAV orthophotos, GNSS (Global Navigation Satellite System) data, GBSAR (Ground-Based Synthetic Aperture Radar) data, underground rock fracture vibration data, geological structure models, and subsurface depth displacement data. Among these, UAV topographic imagery is DSM (Digital Surface Model) data, UAV orthophotos is DOM (Digital Orthophoto Map) data, and underground rock fracture vibration data and subsurface depth displacement data belong to subsurface monitoring data. Correlation analysis between surface and subsurface data lays the foundation for subsequently revealing the causal and temporal relationships between multimodal data.

[0031] Step S20: Preprocess the slope monitoring data of multiple modes to obtain preprocessed data of each mode. The preprocessing is used to unify the spatiotemporal reference of the data.

[0032] For example, slope monitoring data from multiple modalities can be categorized into time-series monitoring data and image monitoring data. Time-series monitoring data can be resampled to standardize time intervals, while image monitoring data can be resampled spatially to standardize spatial resolution. Furthermore, coordinate system unification can be applied to the slope monitoring data from multiple modalities, ensuring that all data are based on the same spatial reference.

[0033] Step S30: Input the preprocessed modal data into the multimodal prediction and segmentation model to encode, fuse, jointly infer and predict and segment the features to obtain the slope deformation risk warning map.

[0034] It should be noted that the multimodal prediction and segmentation model can be understood as a deep learning network that simultaneously processes and fuses data from multiple modalities. For example, the sub-networks of the multimodal prediction and segmentation model convert the preprocessed data from each modality into high-dimensional, abstract feature vectors, which are then integrated at specific layers of the model to generate a unified feature representation containing multi-source information. Based on the fused multimodal features, a unified decision analysis is performed, simultaneously completing deformation trend prediction and risk region segmentation.

[0035] In this embodiment, multi-modal slope monitoring data is acquired and preprocessed to unify the spatiotemporal benchmark. This breaks through the information limitations of a single modality at the data source level, providing the model with a more comprehensive description of the slope state, thus laying the foundation for solving prediction biases caused by incomplete information. By inputting the preprocessed multi-modal data into the multi-modal prediction and segmentation model for feature encoding, fusion, joint inference prediction, and segmentation, the model can deeply explore the intrinsic correlations and causal laws among multiple sources of information such as deformation sequences, groundwater levels, and microseismic activity, rather than analyzing single signals in isolation. This enhances the ability to characterize slope instability mechanisms and significantly improves the causality and accuracy of predictions. Finally, because the model architecture itself does not depend on specific sensor types and can flexibly process heterogeneous data such as images, waveforms, and point clouds through a unified framework, it achieves scalable fusion from finite modes to N modes. This allows the model to adapt to the differentiated monitoring conditions of different mining areas. By continuously optimizing its performance using richer data resources, while improving generalization, it ultimately outputs a more accurate and spatially intuitive slope deformation risk early warning map, providing a reliable basis for accurate early warning and decision-making.

[0036] In one implementation, the multimodal slope monitoring data includes time-series monitoring data, radar monitoring data, optical image monitoring data, subsurface monitoring data, and geological model monitoring data. It should be noted that the multimodal slope monitoring data can be categorized by data type into time-series monitoring data, radar monitoring data, optical image monitoring data, subsurface monitoring data, and geological model monitoring data. Specifically, GNSS data, underground rock fracturing vibration data, and subsurface depth displacement data belong to time-series monitoring data; GBSAR data belongs to radar monitoring data; DSM data and DOM data belong to optical image monitoring data; and geological structural models belong to geological model monitoring data.

[0037] Based on this, step S20 includes: performing time resampling and missing value interpolation on the time-series monitoring data to obtain preprocessed time-series monitoring data; performing coordinate transformation, spatial downsampling, and interpolation processing on the radar monitoring data to generate radar images, and performing decoherence region filling and time resampling on the radar images to obtain preprocessed radar monitoring data; performing spatial resampling on the optical image monitoring data and aligning it with the spatial grid of the radar images to obtain preprocessed optical image monitoring data; performing time stamp uniform calibration on the underground monitoring data and aligning it with the time series of the time-series monitoring data to obtain preprocessed underground monitoring data; and performing coordinate system one on the geological model monitoring data and spatially aligning it with the radar monitoring data to obtain preprocessed geological model monitoring data.

[0038] For example, for GNSS data, the data is first resampled to a 1-hour interval, and then missing values ​​are imputed. For GBSAR data, the raw binary data is converted to CSV (Comma-Separated Values) format, then converted to a coordinate system and downsampled to a 1-meter grid. The converted scatter data is then interpolated to generate GeoTIFF (Georeferenced Tagged Image File Format) radar radial displacement images, and incoherent areas on the images are filled with interpolation. The images are then resampled temporally. For DSM and DOM data, spatial resampling can be used to uniformly sample to a 1-meter spatial resolution, aligned with the radar image grid. For subsurface depth displacement data and subsurface rock fracture vibration data, the timestamps are uniformly calibrated to UTC (Standard Time Reference) time and aligned with the time series of GNSS data. For geological structure models, the entire model is embedded in the same coordinate system as the GBSAR data and uses the same spatial reference, such as CGCS2000 (Standard Spatial Reference).

[0039] In this embodiment, by performing time resampling and missing value interpolation on the time-series data, the problem of inconsistent acquisition frequencies and discontinuous data from different devices is solved, providing a standardized and unified time reference for subsequent fusion. By performing coordinate transformation, spatial downsampling, interpolation to generate images, and filling in incoherent areas on the radar monitoring data, it is transformed from scattered signals in relative device coordinates into raster data with absolute geographical significance, complete data, and a regular structure. Simultaneously, by spatially resampling the optical image to align it with the radar image grid, the heterogeneity of data from different spatial sensors in terms of scale, resolution, and spatial reference is resolved, achieving precise matching of multi-source spatial data at the pixel level, laying the foundation for feature-level fusion. Furthermore, by uniformly calibrating and aligning the timestamps of the underground monitoring data with the time-series data, and by unifying the coordinate system and aligning the geological model with space, high-quality and highly consistent input is provided for subsequent deep and accurate multimodal joint inference of the model, fundamentally ensuring and significantly improving the accuracy and reliability of the final slope deformation risk early warning map.

[0040] Reference Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the multimodal slope deformation early warning method of this application, based on the above. Figure 1 The first embodiment shown presents a second embodiment of the multimodal slope deformation early warning method of this application.

[0041] In the second embodiment, step S30 includes: Step S301: Perform feature encoding on each of the preprocessed modal data to obtain the feature representation of each modality.

[0042] It should be noted that different encoders can be built for data of different modalities. By encoding the preprocessed data of each modality, it can be transformed into feature depth with unified dimension and semantics, providing high-quality joint feature representation for downstream tasks, and making the processed data flexibly expandable to N modalities.

[0043] Step S302: Map the feature representations of each modality to a unified feature space and perform feature fusion to obtain joint features.

[0044] It should be noted that feature fusion can be understood as the operation of integrating the feature representations of various modalities that have been mapped to a unified feature space. Fusion methods can include concatenation, weighted addition, or attention-based mechanisms. The joint feature is the output of the feature fusion step, which can be understood as a unified high-dimensional feature vector that integrates multimodal information.

[0045] Step S303: Based on joint features, generate textual descriptions of risk areas through a large language model, and input visual feature representations, learnable masks, and textual descriptions of risk areas into a conditional mask generator. By performing pixel-level segmentation on areas of different risk levels, a slope deformation risk assessment map is generated.

[0046] For example, the slope area can be divided into pixel-level or unit-level sections to identify areas with different risk levels, and the slope deformation in the future can be inferred based on historical and current data.

[0047] In this embodiment, a targeted neural network model is used to extract deep features related to deformation from each modality of data, achieving efficient extraction of heterogeneous information and preparing high-quality feature representations for deep fusion. Mapping and fusing these modal feature representations to a unified feature space breaks down the inherent structural and dimensional barriers between different modal data, generating joint features that comprehensively reflect the slope state. This overcomes the limitations of existing single-modal perspectives and lays the foundation for accurately capturing complex deformation mechanisms. Based on the joint features, a large language model is used to generate textual descriptions of risk areas and a risk assessment map is generated accordingly. The data-driven joint features are transformed into text containing causal explanations, and this deep reasoning result is ultimately mapped intuitively onto a spatial map, enabling the final output risk assessment map to provide an accurate spatial risk distribution.

[0048] In some embodiments, step S301 includes: performing temporal encoding on preprocessed time-series monitoring data and preprocessed underground monitoring data, obtaining temporal feature representations through patch partitioning, linear projection, and attention alignment operations; performing spatiotemporal encoding on preprocessed radar monitoring data, obtaining spatiotemporal feature representations through patch partitioning, location encoding, vector projection, and attention alignment operations; performing visual encoding on preprocessed optical image monitoring data, obtaining visual feature representations through patch partitioning, location encoding, and image encoding processing; and performing geometric encoding on preprocessed geological model monitoring data, obtaining geometric feature representations through downsampling and transformer encoding processing.

[0049] Understandably, different encoders need to be built for different modalities of data in order to solve the fusion difficulties caused by poor compatibility of heterogeneous modalities such as time series, images, and point clouds. See the following implementation methods for details.

[0050] It should be noted that the steps for spatially resampling the optical image monitoring data and aligning it with the spatial grid of the radar image to obtain the preprocessed optical image monitoring data can be found in the following example.

[0051] For example, the preprocessed DOM image and DSM images Divided into and Each patch block is of equal size. A learnable linear projection is used to linearly project each patch block into a token, and then positional encoding is performed to obtain a feature sequence. and Then, the image is encoded using a pre-trained CLIP visual encoder to obtain a feature sequence. and .

[0052] In one implementation, the steps of performing time-series encoding on preprocessed time-series monitoring data and preprocessed underground monitoring data, and obtaining time-series feature representations through patch partitioning, linear projection, and attention alignment operations include: performing reversible instance normalization on the preprocessed time-series monitoring data and preprocessed underground monitoring data to obtain normalized time-series data; dividing the normalized time-series data into patch blocks and performing linear projection embedding on the patch blocks to obtain time-series patch embeddings; and aligning the text embeddings with the time-series patch embeddings through an attention mechanism to obtain time-series feature representations.

[0053] It should be noted that GNSS data, underground rock fracture vibration data, and underground depth displacement data are all time-series monitoring data, and the GBSAR encoder will be used as an example for illustration. For example,

[0054] GNSS time series data for each dimension The data is processed using Reversible Instance Normalization (RevIN); then the data is divided into... A series of consecutive overlapping or non-overlapping patch blocks The patch is then embedded into a feature vector through feature mapping. The corresponding formula is: in, The number of time series numbers is Dimensions Slip() represents the patch block partitioning function; This indicates the number of patches for each dimension of the GNSS data; L is the length of each patch. Linear() represents a linear projection layer; This is the dimension after embedding the patch using a linear layer. To describe time-series changes, a new set of text representations can be learned from the original pre-trained word embedding matrix using linear probe learning, resulting in new pre-trained text embeddings. To encode time series segments into text representations, a cross-attention mechanism can be used to match the time series segments with the new text representations. This can be achieved by embedding time series patches after convolutional mapping. As a query and for new pre-trained text embeddings Perform convolution mapping as key Sum The corresponding formula is:

[0055] in, , and These represent the weights of the query, key, and value, respectively. , and These represent the corresponding biases. They can be... and The matching process yields a weight matrix. This represents the correlation between each text and the time series. Then, based on the weight matrix, using... We perform weighted summation to obtain an encoded time series segment representation. , represented as:

[0056] in, express and Dimensions This is the activation function.

[0057] In this embodiment, reversible instance normalization is used to process time-series and underground monitoring data. This eliminates the interference caused by differences in the dimensions of different physical quantities during model training, while preserving the normalization parameters to provide a pathway for subsequent recovery of physical dimensions. This lays the foundation for solving the problem of unclear physical meaning in model prediction results. The technique of dividing normalized data into patch blocks and performing linear projection embedding transforms continuous time series into a set of structured high-dimensional feature fragments, enabling the model to capture local temporal patterns and dependencies like image patches. Alignment between time series and natural language enhances the time-series understanding and reasoning capabilities of the large language model.

[0058] In one implementation, the preprocessed radar monitoring data is spatiotemporally encoded, and a spatiotemporal feature representation is obtained through patch partitioning, position encoding, vector projection, and attention alignment operations. This includes: segmenting each frame of the preprocessed radar monitoring data into multiple non-overlapping patch blocks, and performing vector projection on the multiple non-overlapping patch blocks to obtain patch embeddings; adding spatial position encoding and temporal position encoding to the patch embeddings respectively to obtain spatiotemporal patch block embeddings; performing patch discarding processing on the spatiotemporal patch block embeddings to obtain processed patch embeddings; performing spatiotemporal encoding processing on the processed patch embeddings to obtain encoded features, and performing temporal pooling processing on the encoded features to obtain a spatiotemporal feature representation.

[0059] For example, the preprocessed GBSAR frame sequence Each frame of the image is segmented into Each patch is composed of non-overlapping patches of equal size, and these patches are feature-mapped into feature vectors. Then, positional encoding, including spatial positional encoding, is performed on each patch embedding. and time location encoding Obtain the spacetime patch block embedding The formula is:

[0060] in, This indicates an operation that performs vector projection on the patch; Spatial location coding is a set of learnable parameters used to preserve spatial location information; The temporal position encoding is a set of learnable parameters that encodes the temporal relationships between frames.

[0061] To enhance model robustness and prevent overfitting, spatiotemporal patch embedding is used. To discard a patch, the corresponding formula is: , in, Table patch discard operation; This is the output result after the discard operation; This is the probability of being dropped, which is set to 0.5 here; This indicates the number of patches after the patch discarding process; It is the dimension after feature mapping of the patch.

[0062] Furthermore, spatiotemporal coding can be performed based on the CLIPVision Transformer model from the HuggingFace Transformers library. Spatial attention is used to handle patch relationships within the same frame, while temporal attention is used to handle corresponding patch relationships between different frames, yielding the output. The corresponding formula is: , in, This represents the spatiotemporal coding operation. The CLS Token (Classification Token) of each encoded frame is subjected to temporal mean pooling, merging into a single vector while preserving global features of temporal information. The corresponding formula is:

[0063] in, This represents the output of the GNSS encoder, which is an embedded feature representation of GNSS time-series data; Indicates the feature normalization layer; Indicates the number of frames.

[0064] In this embodiment, by segmenting radar images into non-overlapping patches and performing vector projection, continuous image data can be transformed into a structured patch embedding sequence, providing standardized input for subsequent Transformer-based model processing and solving the problem of insufficient feature extraction capability of traditional convolutional networks for irregular deformation fields. Furthermore, spatial and temporal location encodings are added to the patch embeddings, explicitly injecting the spatial orientation and temporal context information of each data point, enabling the model to clearly distinguish and associate deformation features at different locations and times. In addition, due to the patch discarding process, some input information is randomly masked during training, forcing the model to avoid overfitting to local, accidental noise patterns and instead learn to recover complete information from a more global spatiotemporal context, thus significantly enhancing the model's robustness to data loss and noise and its generalization ability. Finally, by employing a spatiotemporal encoding processing combined with temporal pooling, the long-range dependencies between any spatiotemporal points in the deformation field are fully exploited through a self-attention mechanism, aggregating complex temporal dynamics into a compact and comprehensive spatiotemporal feature representation, enabling this feature to deeply characterize the overall dynamic process of slope deformation.

[0065] In one embodiment, the steps of geometrically encoding the preprocessed geological model monitoring data and obtaining a geometric feature representation through downsampling and transformer encoding include: performing data transformation on the preprocessed geological model monitoring data to obtain a point cloud representation; performing downsampling processing on the point cloud representation to obtain a downsampled point cloud; performing transformer encoding on the downsampled point cloud and modeling the geometric relationships between points to obtain a geometric feature representation.

[0066] It should be noted that geological structure models typically exist in mesh form; for example, all vertices of the mesh can be extracted. (in , Represents a set The Chinese Chamber of Commerce has (The original point cloud is converted from 100 vertices to a point cloud representation.) To reduce computational cost and normalize the input, farthest-point sampling can be used to downsample the original point cloud, ultimately obtaining a point cloud with a fixed number of points. (in , Represents a set The Chinese Chamber of Commerce has (Point clouds contain vertices) can be modeled using the PointTransformer architecture to model the relationships between points. Its self-attention mechanism can effectively model long-range dependencies in point clouds and capture the global geometric context of geological structures such as faults and folds. The formula is as follows:

[0067] in, express Adjacent points; Point and Relative position information; , and These represent the eigenvalues ​​after linear transformation; , and These represent linear operations with different parameters; It is a mapping function of an MLP layer.

[0068] In this embodiment, by transforming complex three-dimensional geological structures into regular, uniform point set data, the data complexity and computational burden are effectively reduced, providing feasibility for efficient processing by subsequent deep learning models. Transformer encoding is used to process the downsampled point cloud, and a self-attention mechanism dynamically captures the spatial relative positions and connectivity between any two points in the point cloud, achieving in-depth mining and extraction of the global geometric morphology and structural features of key geological structures such as stratigraphic interfaces and faults. Through geometric relationship modeling capabilities, the resulting geometric feature representation can accurately encode the control effect of geological structures on slope stability.

[0069] Based on the above embodiments and implementation methods, step S302 includes: mapping the modal feature representations to a unified semantic space through a learnable linear transformation layer, and concatenating the mapped modal feature vectors to obtain joint features.

[0070] For example, the output text and visual tokens of each modal encoder are... , , , , , , The cross-modal embedding layer uniformly maps to the same embedding space, and the output dimension corresponds to the input size of the large language model based on LLaVA. The modal features processed by the embedding layer are then mapped to (…). , , , , , , Feature fusion is performed to obtain joint features. The formula is as follows: in, This indicates that cross-modal embedding layer processing operations can map the feature vectors of all modalities to a unified high-dimensional semantic subspace through learnable linear transformations. This indicates a feature fusion operation.

[0071] In one implementation, step S303 includes: encoding the task instruction prompts and category prompts to obtain a task embedding vector; inputting the task embedding vector and joint features into a large language model for joint inference to output a text description of the risk area; inputting the visual feature representation, learnable mask, and text description of the risk area into a conditional mask generator to generate a segmentation mask and category labels; merging the segmentation mask and category labels to obtain a final category label for each pixel; and performing pixel-level segmentation of areas with different risk levels based on the final category label for each pixel to generate a slope deformation risk assessment map.

[0072] It should be noted that the conditional mask generator can be understood as a pre-defined neural network module. Using visual feature representations and textual descriptions of risk regions as joint inputs, it can generate corresponding region labels in the image space corresponding to the visual features based on the semantic information defined by the textual descriptions.

[0073] For example, multimodal prediction segmentation models such as Figure 3 As shown, to introduce task priors and domain knowledge, task instruction prompts (such as "Please predict the slope displacement in the next 24 hours and classify the risk level of the displacement according to the category prompt to achieve risk area segmentation") and category prompts (risk area level definition) can be designed. These are then processed into token sequences using a pre-trained text tokenizer and passed through a linear layer to obtain an embedded representation. and Embed task instructions Category embedding Joint features Joint inference is performed on a large language model based on the LLaVA architecture, and low-rank adaptive learning (LoRA) is used to fine-tune the model, reducing computational resources and improving training efficiency. The large model ultimately outputs a text description token for the risk region, denoted as [Token Name]. To overcome the limitation of large language models being limited to text output, the language output is transformed into a spatially displayed risk assessment map to achieve the prediction and segmentation task. The structure diagram of the conditional mask generator is shown below. Figure 4 As shown, visual features encoded in DOM images Text description of risk areas decoded by LLM The input consists of a set of learnable masks, which are binary images representing which segmentation region each pixel in the image belongs to. As a query feature, it is used to represent the target region that needs to be segmented. As key-value features, they represent the information of each pixel in the image. A conditional mask generator predicts the class label and segmentation mask for each pixel in the DOM image. By merging the segmentation mask with the class label, the final class label for each pixel is obtained. An image is then drawn using the class label of each pixel, with each different class label corresponding to a different color, thus obtaining the final slope deformation risk level warning map.

[0074] In this implementation, task instruction prompts and category prompts are encoded into task embedding vectors and input together with joint features into a large language model for joint inference. Natural language instructions explicitly constrain the model's output format and content scope, transforming the open-ended generation task into a controllable structured analysis. Simultaneously, the powerful causal inference capabilities of the large language model are utilized to deeply interpret multimodal joint features, generating textual descriptions that include risk location, level, and mechanistic explanations. This addresses the problem of traditional methods lacking semantic information and causal explanations in early warning results. Furthermore, by inputting visual feature representations and this textual description together into a conditional mask generator, the semantic information of the textual description is used as a generation condition, guiding the model to accurately map abstract language inference results back to specific spatial locations at the pixel level, achieving a transformation from semantic concepts to spatial segmentation. Finally, by merging the generated segmentation mask and category labels, a slope deformation risk assessment map integrating risk level, spatial distribution, and causal mechanisms is obtained. This makes the early warning results not only accurately located but also possess clear physical meaning and rich contextual information, significantly improving the intuitiveness, interpretability, and decision support effectiveness of risk assessment.

[0075] In some embodiments, an electronic device provided in this application includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the above-described multimodal slope deformation early warning method.

[0076] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0077] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0078] This application also provides a computer-readable medium storing a computer program thereon, which, when executed by a processor, implements the multimodal slope deformation early warning method described above. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments or implementations of this application.

[0079] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0080] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multimodal slope deformation early warning method, characterized in that, The method includes: Acquire slope monitoring data in multiple modes; The slope monitoring data of the various modes are preprocessed to obtain preprocessed data of each mode. The preprocessing is used to unify the spatiotemporal reference of the data. The preprocessed modal data are input into a multimodal prediction and segmentation model to encode, fuse, jointly infer and predict, and segment the features to obtain a slope deformation risk warning map.

2. The method as described in claim 1, characterized in that, The slope monitoring data of the various modalities includes time-series monitoring data, radar monitoring data, optical image monitoring data, underground monitoring data, and geological model monitoring data. The step of preprocessing the slope monitoring data of the various modalities to obtain preprocessed modal data includes: Time-series monitoring data are resampled and missing values ​​are imputed to obtain preprocessed time-series monitoring data. The radar monitoring data is subjected to coordinate transformation, spatial downsampling and interpolation to generate radar images. The radar images are then filled with decoherent regions and resampled over time to obtain preprocessed radar monitoring data. The optical image monitoring data is spatially resampled and aligned with the spatial grid of the radar image to obtain preprocessed optical image monitoring data. The underground monitoring data is timestamped and aligned with the time series of the time-series monitoring data to obtain preprocessed underground monitoring data. The geological model monitoring data is spatially aligned with the radar monitoring data using coordinate system one to obtain preprocessed geological model monitoring data.

3. The method as described in claim 2, characterized in that, The steps of inputting the preprocessed modal data into a multimodal prediction and segmentation model, encoding, fusing, jointly inferring and predicting the preprocessed modal data, and segmenting it to obtain a slope deformation risk early warning map include: The preprocessed modal data are feature-encoded to obtain the feature representation of each modality; The feature representations of each modality are mapped to a unified feature space and then fused to obtain joint features. Based on the joint features, a text description of the risk area is generated through a large language model. The visual feature representation, learnable mask, and text description of the risk area are input into a conditional mask generator. By performing pixel-level segmentation on areas of different risk levels, a slope deformation risk assessment map is generated.

4. The method as described in claim 3, characterized in that, The step of performing feature encoding on each of the preprocessed modal data to obtain the feature representation of each modality includes: The preprocessed time-series monitoring data and the preprocessed underground monitoring data are time-series encoded, and time-series feature representations are obtained through patch partitioning, linear projection, and attention alignment operations. The preprocessed radar monitoring data is spatiotemporally encoded, and spatiotemporal feature representations are obtained through patch partitioning, position encoding, vector projection, and attention alignment operations. The preprocessed optical image monitoring data is visually encoded, and visual feature representations are obtained through patch segmentation, position encoding, and image encoding. The preprocessed geological model monitoring data is geometrically encoded, and geometric feature representations are obtained through downsampling and transformer encoding.

5. The method as described in claim 4, characterized in that, The step of performing time-series encoding on the preprocessed time-series monitoring data and the preprocessed underground monitoring data, and obtaining time-series feature representations through patch partitioning, linear projection, and attention alignment operations, includes: The preprocessed time-series monitoring data and the preprocessed underground monitoring data are subjected to reversible instance normalization to obtain normalized time-series data. The normalized time series data is divided into patch blocks, and the patch blocks are linearly projected and embedded to obtain the time series patch embedding. By aligning the text embedding with the temporal patch embedding using an attention mechanism, a temporal feature representation is obtained.

6. The method as described in claim 4, characterized in that, The step of performing spatiotemporal encoding on the preprocessed radar monitoring data, and obtaining spatiotemporal feature representations through patch partitioning, position encoding, vector projection, and attention alignment operations, includes: Each frame of the preprocessed radar monitoring data is segmented into multiple non-overlapping patch blocks, and vector projection is performed on the multiple non-overlapping patch blocks to obtain patch embedding. Spatial location encoding and temporal location encoding are added to the patch embedding respectively to obtain the spatiotemporal patch block embedding; The spatiotemporal patch block embedding is subjected to patch discarding processing to obtain the processed patch embedding; The processed patch embedding is subjected to spatiotemporal coding to obtain coded features, and the coded features are subjected to temporal pooling to obtain spatiotemporal feature representations.

7. The method as described in claim 4, characterized in that, The step of geometrically encoding the preprocessed geological model monitoring data, and obtaining geometric feature representations through downsampling and transformer encoding, includes: The preprocessed geological model monitoring data is converted to obtain a point cloud representation; The point cloud representation is downsampled to obtain a downsampled point cloud; The downsampled point cloud is encoded by a transformer, and the geometric relationships between the points are modeled to obtain a geometric feature representation.

8. The method as described in claim 3, characterized in that, The steps of generating textual descriptions of risk areas based on the joint features using a large language model, and inputting visual feature representations, learnable masks, and the textual descriptions of the risk areas into a conditional mask generator to generate slope deformation risk assessment maps by pixel-level segmentation of areas with different risk levels include: The task instruction prompts and category prompts are encoded to obtain the task embedding vector; The task embedding vector and the joint feature are input into a large language model for joint inference, and a textual description of the risk area is output. The visual feature representation, the learnable mask, and the text description of the risk area are input into the conditional mask generator to generate a segmentation mask and a category label. The segmentation mask and the category label are merged to obtain the final category label for each pixel; Based on the final category label of each pixel, the areas with different risk levels are segmented at the pixel level to generate a slope deformation risk assessment map.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multimodal slope deformation early warning method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the multimodal slope deformation early warning method as described in any one of claims 1 to 8.

Images