A method for completing highway slope monitoring data based on multimodal data fusion

Abstract

This invention relates to the field of highway slope geological disaster monitoring technology, and discloses a method for highway slope monitoring data completion based on multimodal data fusion. The method includes acquiring multi-source sensor time-series data from monitoring points; aligning the multi-source sensor time-series data according to a preset time granularity to construct a sliding window sample; constructing ternary heterogeneous multimodal features based on the sliding window sample; dynamically fusing the ternary heterogeneous multimodal features using a gated prefix self-attention fusion mechanism to obtain fused features; inputting the fused features into a pre-trained time-series completion model, and outputting the predicted values ​​of the missing positions through a decoder to obtain the completed monitoring time-series data. This invention can fully utilize the complementary relationships between different modalities to achieve reliable reconstruction of missing monitoring values.

Description

Technical Field

[0001] This invention relates to the field of highway slope geological disaster monitoring technology, specifically to a method for supplementing highway slope monitoring data based on multimodal data fusion. Background Technology

[0002] Geological hazards on highway slopes are characterized by their suddenness, insidiousness, and destructiveness. Engineering monitoring typically relies on long-term continuous observation using various devices such as GNSS displacement sensors, accelerometers, tilt sensors, and rain gauges. However, due to factors such as extreme weather, power outages, communication failures, equipment maintenance, or sensor damage, field monitoring data frequently suffers from random or continuous data loss, and even prolonged disconnections of critical data channels. Missing data not only disrupts the time alignment of multiple sensors but also reduces the reliability of subsequent deformation trend analysis, disaster early warning, and risk assessment.

[0003] Existing missing data completion methods mainly include traditional interpolation, statistical prediction, machine learning, and deep learning methods. Linear interpolation, spline interpolation, and mean imputation methods are computationally simple, but they usually assume that the monitoring sequence changes steadily or approximately linearly within the missing interval, making it difficult to characterize the nonlinear displacement evolution under the combined effects of rainfall infiltration, slope structure changes, and external disturbances. When the missing segment lasts for several hours or even longer, these methods are prone to oversmoothing or deviating from the true trend.

[0004] Statistical models such as ARIMA offer some interpretability, but typically rely on stationarity and regular sampling conditions. While deep learning models such as LSTM, GRU, BiLSTM, and the standard Transformer can capture some nonlinear temporal relationships, most still rely on a single numerical sequence as input. When displacement sensors experience continuous disconnections, the local context available to the model decreases drastically, making it difficult to obtain effective compensation from rainfall, dip angle, acceleration, and geological semantic information.

[0005] In recent years, masked autoencoders and multi-source data fusion mechanisms have provided new pathways for reconstructing missing data. Masked modeling can learn the data distribution by reconstructing masked segments, and self-attention mechanisms can capture long-range dependencies. However, existing methods mostly remain at the level of single-modal mask reconstruction or simple multi-modal splicing, lacking deep collaborative mechanisms for the three heterogeneous information of time-series numerical data, semantic text, and Lomb spectrum in slope monitoring, and also lacking gating strategies that dynamically adjust the contribution ratio of each modality based on the current monitoring status.

[0006] Therefore, the existing technology has at least the following shortcomings:

[0007] (1) Most deep learning completion models only use single time series values ​​and cannot fully integrate external text data and non-uniform sampling frequency domain priors. They lack cross-modal compensation capabilities when key displacement channels are missing for a long time.

[0008] (2) Existing multimodal fusion methods mostly use simple splicing or fixed weight fusion, which cannot dynamically adjust the modal weights according to sudden increases in rainfall, accelerated displacement, enhanced frequency domain fluctuations, or changes in sensor noise.

[0009] (3) Existing mask training often uses random point masks or continuous value regression loss, which is prone to degenerate into local neighbor interpolation. For scenarios with missing continuous blocks, the completion effect needs to be improved. Summary of the Invention

[0010] To address the aforementioned shortcomings in existing technologies, this invention provides a method for completing highway slope monitoring data based on multimodal data fusion. When multi-source sensor data such as displacement, tilt angle, acceleration, and rainfall exhibit localized or continuous gaps, or non-uniform sampling, this method fully leverages the complementary relationships between different modes to reliably reconstruct missing monitoring values.

[0011] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0012] A method for completing highway slope monitoring data based on multimodal data fusion includes the following steps:

[0013] Acquire time-series data from multi-source sensors at monitoring points;

[0014] Align the multi-source sensor time-series data according to a preset time granularity to construct a sliding window sample;

[0015] A ternary heterogeneous multimodal feature is constructed based on the sliding window sample; the ternary heterogeneous multimodal feature includes temporal numerical features, semantic text features, and spectral features; wherein, the spectral feature is the frequency domain energy distribution extracted based on the non-uniform sampling sequence in the sliding window sample;

[0016] The ternary heterogeneous multimodal features are dynamically fused using a gated prefix self-attention fusion mechanism to obtain fused features;

[0017] The fused features are input into a pre-trained time-series completion model, and the decoder outputs the predicted value of the missing position to obtain the completed monitoring time-series data.

[0018] Furthermore, the method for constructing the time-series numerical features includes:

[0019] Extract target variable channel data from the sliding window sample;

[0020] Auxiliary covariate channel data are extracted from the sliding window sample, the auxiliary covariate channel data including at least one auxiliary monitoring quantity physically associated with the target variable;

[0021] The target variable channel data and the auxiliary covariate channel data are concatenated into a multi-channel time series numerical matrix to obtain the time series numerical features.

[0022] Furthermore, the method for constructing the semantic text features includes:

[0023] Based on the statistical information of multi-source sensors within the current time window, a structured natural language description is automatically generated. The structured natural language description includes data integrity status, abnormal event records, and physical association features.

[0024] The structured natural language description is encoded by a pre-trained large language model to obtain the semantic text features.

[0025] Furthermore, the method for constructing the spectral features includes:

[0026] For target displacement sequences containing missing or non-uniform sampling within the current time window, the Lomb-Scargle periodogram algorithm is used to extract the spectral power distribution.

[0027] The spectral power distribution is converted into a one-dimensional spectral vector of a preset length to obtain the spectral features.

[0028] Furthermore, a gated prefix self-attention fusion mechanism is used to dynamically fuse the ternary heterogeneous multimodal features to obtain fused features, including:

[0029] The time-series numerical features, semantic text features, and spectral features are encoded respectively to obtain encoded features;

[0030] Calculate the dynamic gating coefficient based on the coding characteristics;

[0031] The spectral features are weighted using the dynamic gating coefficients to obtain the gated spectral features;

[0032] The gated spectral features and the semantic text features are used as global prefix dimensions and concatenated to the front end of the time-series numerical feature sequence to obtain the concatenated sequence.

[0033] The fused features are obtained by globally interacting with the spliced ​​sequence using a Transformer self-attention layer.

[0034] Furthermore, the time-series numerical features, semantic text features, and spectral features are encoded respectively to obtain encoded features, including:

[0035] After binning and discretizing the target displacement sequence, it is input into a learnable embedding layer, and then auxiliary covariates and position codes after linear projection are added to obtain the temporal backbone representation.

[0036] Semantic text features are input into a fully connected layer with GELU activation and mapped to a textual latent representation;

[0037] The spectral features are input into a multilayer perceptron with residual connections and a layer normalization layer to extract the spectral latent representation.

[0038] Furthermore, calculating the dynamic gating coefficient based on the coding features includes:

[0039] Extract the global context vector of the current time window from the implicit representation of the temporal backbone;

[0040] The global context vector is concatenated with the spectral implicit representation, and the dynamic gating coefficients are calculated using a Sigmoid gating network.

[0041] Furthermore, the training process of the temporal completion model includes:

[0042] In the first training phase, an artificial continuous block mask is applied to the target variable channel data in the sliding window sample. Based on the masked data, the ternary heterogeneous multimodal features are constructed and fused to obtain the fused features. The training model predicts the target variable channel data at the masked position.

[0043] In the second training phase, the pre-trained weights from the first training phase are loaded and fine-tuned using the true missing mask.

[0044] Furthermore, applying an artificial contiguous block mask to the target variable channel data in the sliding window sample includes:

[0045] According to a preset masking ratio, a continuous time segment is randomly selected from the target variable channel data and a mask is applied;

[0046] The model parameters are optimized using the cross-entropy loss function, enabling the model to learn the distribution of target variable channel data at the masked location based on the fused features.

[0047] Furthermore, the predicted values ​​for the missing locations output by the decoder include:

[0048] Based on the historical observation distribution of the target variable, the boundary values ​​of a preset number of discrete bins are calculated using the quantile method, and the center value of each discrete bin is used as the predicted representative value of that bin.

[0049] The decoder outputs the probability distribution of the missing location on each discrete bin;

[0050] Based on the probability distribution and bin center values, a weighted expectation is calculated to obtain the predicted value of the continuous target variable.

[0051] The present invention has the following beneficial effects:

[0052] This invention utilizes three types of information—temporal numerical values, semantic text, and Lomb spectrum—to collaboratively reconstruct missing displacements. It maintains stable accuracy even with long periods of continuous missing time, avoiding the over-smoothing of traditional linear interpolation over long missing intervals. Through continuous block mask pre-training, the model no longer relies on local interpolation shortcuts between adjacent time points, but instead learns cross-modal mapping relationships between rainfall, tilt angle, acceleration, semantic trends, and frequency domain periods, making it suitable for scenarios involving power outages or communication interruptions in the field.

[0053] Lomb periodic spectral features can directly extract frequency domain information from irregular sampling sequences, avoiding the spurious periodicity problem caused by interpolation followed by spectral analysis. The gating mechanism can dynamically adjust the contribution of auxiliary modes based on the current monitoring window status. When heavy rainfall or accelerated displacement causes frequency domain changes, the model can increase the weight of spectral information; when spectral noise is strong, the model can reduce its impact. Through discrete binning and Softmax probability distribution, the model not only outputs a single completed value but also implicitly includes the confidence level of each candidate displacement interval, which is beneficial for subsequent uncertainty analysis in the early warning system. This method can be embedded in a slope monitoring and early warning platform to automatically supplement key monitoring data during sensor disconnection periods, maintain the continuity of monitoring curves, and provide a more reliable data foundation for landslide trend analysis, risk level determination, and emergency response. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the process for a method to complete highway slope monitoring data based on multimodal data fusion according to the present invention. Detailed Implementation

[0055] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0056] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for supplementing highway slope monitoring data based on multimodal data fusion, comprising the following steps S1 to S5:

[0057] S1. Acquire time-series data from multi-source sensors at monitoring points;

[0058] In an optional embodiment of the present invention, step S1 involves acquiring multi-source sensor data from the target slope or landslide monitoring point. The sensor data includes one or more of GNSS displacement data, acceleration data, tilt angle data, and rainfall data. The raw data undergoes timestamp correction, hourly granularity alignment, outlier identification, missing value marking, and standardization.

[0059] ;

[0060] in, This represents the raw observation value of the d-th channel at time step t. and Let represent the mean and standard deviation of the d-th channel in the training set, respectively. To prevent extremely small constants with a denominator of zero, This represents the observation value after standardization.

[0061] For missing locations caused by power outages or communication interruptions, a dedicated invalid numerical marker for the missing location and its corresponding mask are retained to avoid revealing local patterns. The missing mask is constructed synchronously. :

[0062] ;

[0063] in, This indicates the missing state of the d-th channel at time step t.

[0064] Specifically, this embodiment deploys or acquires data from various monitoring devices at the target landslide site, including GNSS displacement sensors, acceleration sensors, dual-axis tilt sensors, and rain gauges. All channels of data from these four types of sensors are aligned at the hourly granularity to form a multi-dimensional time-series matrix containing channels for total displacement, horizontal displacement, vertical displacement, triaxial acceleration, tilt angle, and rainfall, while simultaneously recording any missing locations.

[0065] S2. Align the multi-source sensor time-series data according to a preset time granularity to construct a sliding window sample;

[0066] In an optional embodiment of the present invention, step S2 involves aligning and normalizing the multidimensional monitoring sequence. A sliding window is set up, with a preferred window length of 24 hours and a preferred sliding step size of 1 hour, to obtain multiple sliding window samples:

[0067] ;

[0068] ;

[0069] in, Indicates the first The sliding window sample constructed from a sliding window; For standardized multi-source monitoring sequences, Indicates from sequence Extracting the first to The continuous time steps and the subsequences of all D channels are used as the first... The original monitoring data of each window, among which This is the window length. For the first The missing mask sequence corresponding to each sliding window. Indicates from the global mask sequence Extracting the first to Subsequences of all D channels across continuous time steps are used to obtain window-specific missing masks. Each sample includes a target reconstruction channel, an auxiliary covariate channel, a window start and end time, and a corresponding missing mask. The target reconstruction channel is preferably total displacement, horizontal displacement, and vertical displacement, while the auxiliary covariate channel is preferably monitored quantities such as triaxial acceleration, tilt angle, and rainfall.

[0070] Specifically, in this embodiment, a sliding window sample is generated with a sliding window length of 24 hours and a sliding step size of 1 hour. For each time window, the target displacement missing mask is retained, auxiliary covariates are extracted, and the training set, validation set, and test set are divided according to time order.

[0071] This embodiment addresses the issues of minute-level drift in sampling time, asynchronous device power-on time, and irregular missing segments in multi-source sensors. By establishing a unified hour-level time alignment, missing mask retention, and sliding window sample generation mechanism through steps S1 and S2.

[0072] S3. Construct a ternary heterogeneous multimodal feature based on the sliding window sample; the ternary heterogeneous multimodal feature includes temporal numerical features, semantic text features, and spectral features; wherein, the spectral feature is the frequency domain energy distribution extracted based on the non-uniform sampling sequence in the sliding window sample;

[0073] In an optional embodiment of the present invention, the method for constructing the time-series numerical features in step S3 includes:

[0074] Extract target variable channel data from the sliding window sample;

[0075] Auxiliary covariate channel data are extracted from the sliding window sample, the auxiliary covariate channel data including at least one auxiliary monitoring quantity physically associated with the target variable;

[0076] The target variable channel data and the auxiliary covariate channel data are concatenated into a multi-channel time series numerical matrix to obtain the time series numerical features.

[0077] The methods for constructing semantic text features in step S3 include:

[0078] Based on the statistical information of multi-source sensors within the current time window, a structured natural language description is automatically generated. The structured natural language description includes data integrity status, abnormal event records, and physical association features.

[0079] The structured natural language description is encoded by a pre-trained large language model to obtain the semantic text features.

[0080] The method for constructing the spectral features in step S3 includes:

[0081] For target displacement sequences containing missing or non-uniform sampling within the current time window, the Lomb-Scargle periodogram algorithm is used to extract the spectral power distribution.

[0082] The spectral power distribution is converted into a one-dimensional spectral vector of a preset length to obtain the spectral features.

[0083] This embodiment addresses the problem of insufficient information in single numerical sequences with long-term missing data by constructing three heterogeneous features: time-series numerical data, semantic text, and Lomb-Scargle spectrum, forming a ternary multimodal input for missing data completion. This embodiment constructs three modal features for each sliding window sample: time-series numerical modality, semantic text modality, and Lomb periodogram spectrum modality.

[0084] The first category is time-series numerical modes. The standardized multidimensional sensor readings within the window are organized into a W×D matrix, where... Let D be the window length and D be the number of channels. The preferred target displacement channels include total displacement, horizontal displacement, and vertical displacement. Auxiliary covariate channels include the maximum change in the X-axis direction, maximum change in the Y-axis direction, and maximum change in the Z-axis direction of the triaxial accelerometer; the angle between the X-axis and the horizontal plane of the tilt sensor; the angle between the Y-axis and the horizontal plane of the tilt sensor; the angle between the XY-axis and the horizontal plane of the tilt sensor; the angle between the Z-axis and the horizontal plane of the tilt sensor; and the hourly cumulative rainfall. This time-series mode retains direct numerical information such as slope deformation, rainfall-induced factors, tilt changes, and vibration response.

[0085] The second category is semantic text modality. Based on statistical features such as window start and end times, average displacement, maximum displacement, horizontal displacement, vertical displacement, cumulative rainfall, maximum acceleration, missing proportion, and deformation level, a structured natural language description is automatically generated; this description is then encoded into a multi-dimensional semantic embedding vector using a pre-trained Sentence-BERT model. Time steps marked as power outages or masked are masked during text generation to avoid leaking the target to be reconstructed to the model.

[0086] The third category is Lomb-Scargle spectral modes. For target displacement sequences with missing or non-uniform sampling, the Lomb-Scargle periodogram algorithm is used to directly extract the frequency domain energy distribution, generating a one-dimensional spectral power vector with an optimal length of 128. This method does not require forced linear interpolation of the original missing sequence and can preserve stable frequency components and periodic priors under irregular sampling conditions.

[0087] Specifically, this embodiment calculates statistical features such as average total displacement, maximum total displacement, average horizontal displacement, maximum horizontal displacement, average vertical displacement, maximum vertical displacement, cumulative rainfall, and maximum acceleration for each window, and automatically generates a structured text description. For example: "Time period: 2024-12-05 01:00 to 2024-12-06 00:00. Average total displacement: 2.54 mm, Max total displacement: 3.00 mm, displacement level: normal range. Total rainfall in window: 0.00 mm, Max acceleration magnitude: 0.2256 mg." This is then encoded into a multi-dimensional vector using Sentence-BERT, preferably 384 dimensions.

[0088] This embodiment calculates a Lomb-Scargle periodogram for each window containing the missing target displacement sequence and converts it into a one-dimensional spectral power vector. This step does not perform forced linear interpolation on the missing intervals to avoid spurious frequency components.

[0089] This embodiment adopts a ternary heterogeneous sample construction method for missing data completion in slope monitoring. It unifies the time series data of multi-source sensors, window statistical semantic text, and Lomb periodic spectrum vector into a multimodal input. The stable frequency components in the missing sequence are directly extracted through Lomb periodogram, providing frequency consistency constraints for missing reconstruction without simple interpolation.

[0090] S4. The ternary heterogeneous multimodal features are dynamically fused using a gated prefix self-attention fusion mechanism to obtain fused features;

[0091] In an optional embodiment of the present invention, step S4 employs a gated prefix self-attention fusion mechanism to dynamically fuse the ternary heterogeneous multimodal features to obtain fused features, including:

[0092] The time-series numerical features, semantic text features, and spectral features are encoded respectively to obtain encoded features;

[0093] Calculate the dynamic gating coefficient based on the coding characteristics;

[0094] The spectral features are weighted using the dynamic gating coefficients to obtain the gated spectral features;

[0095] The gated spectral features and the semantic text features are used as global prefix dimensions and concatenated to the front end of the time-series numerical feature sequence to obtain the concatenated sequence.

[0096] The fused features are obtained by globally interacting with the spliced ​​sequence using a Transformer self-attention layer.

[0097] Specifically, the time-series numerical features, semantic text features, and spectral features are encoded to obtain encoded features, including:

[0098] After binning and discretizing the target displacement sequence, it is input into a learnable embedding layer, and then auxiliary covariates and position codes after linear projection are added to obtain the temporal backbone representation.

[0099] Semantic text features are input into a fully connected layer with GELU activation and mapped to a textual latent representation;

[0100] The spectral features are input into a multilayer perceptron with residual connections and a layer normalization layer to extract the spectral latent representation.

[0101] The calculation of dynamic gating coefficients based on coding features includes:

[0102] Extract the global context vector of the current time window from the implicit representation of the temporal backbone;

[0103] The global context vector is concatenated with the spectral implicit representation, and the dynamic gating coefficients are calculated using a Sigmoid gating network.

[0104] To enhance the model's robustness to field monitoring noise and outliers, this embodiment does not directly perform continuous value regression on the target displacement. Instead, it first constructs a discrete vocabulary containing K bins based on the value range of the target displacement channels in the training set, mapping the target displacement channel data in the time-series numerical features to discrete terms:

[0105] ;

[0106] in, This represents the discrete binning mapping operation for continuous displacement values. Indicates the first Each time step Target displacement value for each target displacement channel; to For the container boundary, Indicates the number of boxes. More specifically, This represents the discrete bin label at time step t, when hour , For the box number, the masked position uses a dedicated MASK identifier:

[0107] ;

[0108] in, Indicates the first Some time steps are either artificially obscured or genuinely missing. Indicates the first Each time step was neither artificially obscured nor missing. Indicates a dedicated MASK identifier. Indicates the first The target displacement word after masking at each time step.

[0109] This embodiment addresses the issue of significant differences in modal structures and dimensions by setting up an independent modal encoder to map discretized displacement terms, auxiliary covariates, multidimensional semantic embeddings, and multidimensional spectral vectors to a unified latent space. The target displacement channel data in the time-series numerical features is mapped to d through a learnable embedding layer. model The dimensional features and auxiliary covariates are mapped to the same dimension through linear projection, and the sum of the two forms the temporal backbone latent representation. :

[0110] ;

[0111] in, This represents the target shifted word sequence after masking. This indicates a learnable embedding layer; This represents the auxiliary covariate matrix, including channels for rainfall, tilt angle, acceleration, etc. Indicates a linear projection layer; This indicates a learnable positional encoding.

[0112] The text modality encoder takes the multidimensional semantic vector output by Sentence-BERT as input and maps it to d through a fully connected layer with GELU activation. model Uyghur text lexicon :

[0113] ;

[0114] in, This represents the semantic text embedding obtained from Sentence-BERT. This represents the GELU activation function. This represents the weight matrix of the text modality encoder. This represents the bias vector of the text modal encoder.

[0115] The Lomb spectral modal encoder takes a one-dimensional power spectral vector as input and extracts the spectral latent representation through a multilayer perceptron with residual connections, layer normalization, and nonlinear activation. :

[0116] ;

[0117] in, This represents the one-dimensional spectral power vector obtained from the Lomb-Scargle periodogram. Representation layer normalization, This represents a multilayer perceptron. This represents the weight matrix of the residual connections; each encoder output is mapped to d. model The unified latent space provides a structural alignment basis for subsequent gated prefix fusion and Transformer global self-attention interaction.

[0118] To avoid introducing noise through fixed weights or simple concatenation of text and spectral features, this invention proposes a gated prefix self-attention fusion mechanism that adaptively adjusts the contributions of spectral and text modalities to the reconstruction task based on the current temporal context. First, the global context vector of the current window is extracted from the temporal backbone latent representation, for example, by using temporal mean pooling to obtain the global context vector. Then the global context vector Lomb spectral implicit representation The data is spliced ​​together, and a Sigmoid gating network is used to output dynamic gating coefficients to adjust the proportion of spectral information flowing in. :

[0119] ;

[0120] in, This represents the gated activation function, and ⊙ represents element-wise multiplication. and These are learnable parameters. This gating mechanism can increase the spectral mode weights during heavy rainfall, displacement acceleration, or enhanced frequency domain changes, and suppress frequency domain interference during steady creep or when there is strong spectral noise.

[0121] Then, the dynamic gating coefficients are multiplied element-wise with the implicit spectral representation to obtain the gated spectral features. ;

[0122] ;

[0123] Subsequently, the gated Lomb spectrum lexical units and text semantic lexical units are used as two global prefix lexical units and concatenated before the temporal backbone lexical sequence to form a unified cross-modal input sequence. :

[0124] ;

[0125] If the window length is 24 hours, the total sequence length after adding the two prefix tokens is 26.

[0126] The unified cross-modal input sequence Hall is input into a multi-layer Transformer encoder, and the association between any time step and the global text and spectral prefix is ​​calculated through multi-head self-attention.

[0127] Through the above design, any masked target displacement term can access the time axis context and bypass the local missing blind zone to directly obtain macro trends and periodic priors from the text prefix and spectral prefix, thereby improving the reconstruction robustness of continuous missing segments.

[0128] This embodiment employs a multimodal independent coding and gated prefix self-attention fusion mechanism. Addressing the data structure differences among temporal numerical, semantic text, and Lomb spectrum modalities, embedding projection, text projection, and residual MLP encoders are set up respectively. The contribution of Lomb spectrum features is dynamically adjusted based on the current window's temporal context, and the spectrum and text representation are used as global prefix lexical inputs to the Transformer, enabling missing positions to be directly filled using cross-modal global priors.

[0129] S5. Input the fused features into the pre-trained time-series completion model, and output the predicted value of the missing position through the decoder to obtain the completed monitoring time-series data.

[0130] In an optional embodiment of the present invention, the training process of the temporal completion model in step S5 includes:

[0131] In the first training phase, an artificial continuous block mask is applied to the target variable channel data in the sliding window sample. Based on the masked data, the ternary heterogeneous multimodal features are constructed and fused to obtain the fused features. The training model predicts the target variable channel data at the masked position.

[0132] In the second training phase, the pre-trained weights from the first training phase are loaded and fine-tuned using the true missing mask.

[0133] Applying an artificial continuous block mask to the target variable channel data in the sliding window sample includes:

[0134] According to a preset masking ratio, a continuous time segment is randomly selected from the target variable channel data and a mask is applied;

[0135] The model parameters are optimized using the cross-entropy loss function, enabling the model to learn the distribution of target variable channel data at the masked location based on the fused features.

[0136] This embodiment addresses the problem of insufficient accuracy in completing consecutive missing segments by proposing a two-stage mask pre-training method. First, cross-modal mapping rules are learned using manually generated continuous block masks, and then fine-tuning is performed using real offline disconnected masks. This embodiment employs a two-stage mask pre-training strategy, progressing from easy to difficult. The first stage is pre-training with manually generated continuous block masks, and the second stage is fine-tuning with real disconnected masks.

[0137] In the first stage, for the target displacement channel data in the temporal numerical features of the training samples, consecutive time segments are randomly selected for masking, with a masking ratio preferably of 20%. The target words at the masked locations are replaced with MASK identifiers. The model predicts the discrete bin labels of the masked locations based on the unmasked temporal context, auxiliary covariates, textual semantic words, and Lomb spectral words. This stage employs cross-entropy loss. The optimization objective is calculated only at the mask location:

[0138]

[0139] in, Represents the set of locations of artificial contiguous block masks. For real discrete bin labels, Let θ be the predicted output probability when the model parameters are θ. for The unmasked target displacement term. Compared to directly using mean squared error regression, the cross-entropy objective represents the model output as a conditional probability distribution, which can better describe the uncertainty in complex nonlinear systems and reduce the impact of outliers on the gradient.

[0140] In the second stage, the pre-trained weights obtained in the first stage are loaded and fine-tuned using real-world offline power outage masks or irregular missing masks found in the dataset. This stage allows the model to migrate from controlled simulated missing data to real-world engineering disconnection scenarios, further learning cross-modal compensation patterns under conditions of sudden rainfall changes, sensor noise, and long-term missing data.

[0141] Specifically, in this embodiment, during the first-stage mask pre-training, a 20% continuous block mask is applied to the effective target displacement sequence. The model outputs a discrete bin probability distribution of the mask positions, and training is performed using cross-entropy loss. Gradient pruning and learning rate scheduling strategies are used during training to enhance convergence stability.

[0142] In the second stage of fine-tuning the real disconnection mask, the model is trained using missing masks caused by power outages, communication interruptions, or equipment maintenance in the actual collected data, so that the model can adapt to the missing modes in real engineering.

[0143] This embodiment adopts a two-stage training mechanism that combines continuous block mask pre-training with real disconnected mask fine-tuning. It transforms continuous displacement reconstruction into a discrete bin classification task, uses cross-entropy loss to learn the conditional probability distribution of missing positions, and recovers continuous displacement values ​​through probability-weighted expectation, thereby improving the model's adaptability to outliers and multi-peak uncertainties.

[0144] In an optional embodiment of the present invention, step S5, which outputs the predicted value of the missing position through the decoder, includes:

[0145] Based on the historical observation distribution of the target variable, the boundary values ​​of a preset number of discrete bins are calculated using the quantile method, and the center value of each discrete bin is used as the predicted representative value of that bin.

[0146] The decoder outputs the probability distribution of the missing location on each discrete bin;

[0147] Based on the probability distribution and bin center values, a weighted expectation is calculated to obtain the predicted value of the continuous target variable.

[0148] This embodiment addresses the issue of continuous value regression being susceptible to outliers and multimodal uncertainties. It discretizes the target displacement into bin tokens, transforms missing value completion into discrete classification probability estimation, and recovers the continuous displacement value during the inference phase using probability-weighted expectation. During model inference, the classification head outputs the Softmax probability distribution of each missing location across K bins. The center values ​​of each bin are used as candidate continuous displacement values, and the continuous displacement is recovered through probability-weighted summation.

[0149]

[0150] in, Let be the predicted probability that the element belongs to the k-th bin when the element is missing at time step t. Let the center value of the k-th bin be... The displacements used to complete the model.

[0151] Finally, the original valid observations and model-completed values ​​are merged using the true missing value mask to obtain the complete monitoring sequence:

[0152]

[0153] in, For complete monitoring sequences, For global mask sequence, These are the original valid observations. To complete the model with values.

[0154] In engineering applications, when the monitoring system detects continuous disconnection of displacement sensors, it automatically generates a missing mask and simultaneously calls auxiliary sensors, text semantic descriptions, and Lomb spectral features to output the reconstructed displacement values ​​for the missing time steps. When the sensors return to normal, the newly added data can be incorporated into periodic fine-tuning to adapt to changes in seasonal rainfall and slope evolution stages.

[0155] The effects achieved by the method of the present invention will be analyzed and explained below with specific examples.

[0156] Using real landslide monitoring data as the object, we simulated continuous missing conditions of 4 hours, 8 hours, 12 hours, and 16 hours, and compared them with BERT-PIN, BiLSTM, Transformer, linear interpolation, and K-nearest neighbor algorithms. The main indicators include mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), and mean absolute percentage error (MAPE). The comparison results are shown in Table 1.

[0157] Table 1. Summary of ablation experiment results for different modal configurations under different missing durations.

[0158]

[0159] Experimental results show that the method of this invention has a more significant advantage over the mainstream baseline in scenarios with long-term missing values ​​of 12 hours and 16 hours. The modal ablation results further illustrate that text semantics and Lomb spectrum can provide effective compensation when the observable context is limited, and the anti-degradation ability of the full modal configuration is better than the model that only uses time-series numerical values.

[0160] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0164] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for completing highway slope monitoring data based on multimodal data fusion, characterized in that, Includes the following steps: Acquire time-series data from multiple sensor sources at monitoring points; Align the multi-source sensor time-series data according to a preset time granularity to construct a sliding window sample; A ternary heterogeneous multimodal feature is constructed based on the sliding window sample; the ternary heterogeneous multimodal feature includes temporal numerical features, semantic text features, and spectral features; wherein, the spectral feature is the frequency domain energy distribution extracted based on the non-uniform sampling sequence in the sliding window sample; The ternary heterogeneous multimodal features are dynamically fused using a gated prefix self-attention fusion mechanism to obtain fused features; The fused features are input into a pre-trained time-series completion model, and the decoder outputs the predicted value of the missing position to obtain the completed monitoring time-series data.

2. The method for supplementing highway slope monitoring data based on multimodal data fusion according to claim 1, characterized in that, The method for constructing the time-series numerical features includes: Extract target variable channel data from the sliding window sample; Auxiliary covariate channel data are extracted from the sliding window sample, the auxiliary covariate channel data including at least one auxiliary monitoring quantity physically associated with the target variable; The target variable channel data and the auxiliary covariate channel data are concatenated into a multi-channel time series numerical matrix to obtain the time series numerical features.

3. The method for supplementing highway slope monitoring data based on multimodal data fusion according to claim 1, characterized in that, The method for constructing the semantic text features includes: Based on the statistical information of multi-source sensors within the current time window, a structured natural language description is automatically generated. The structured natural language description includes data integrity status, abnormal event records, and physical association features. The structured natural language description is encoded by a pre-trained large language model to obtain the semantic text features.

4. The method for supplementing highway slope monitoring data based on multimodal data fusion according to claim 1, characterized in that, The method for constructing the spectral features includes: For target displacement sequences containing missing or non-uniform sampling within the current time window, the Lomb-Scargle periodogram algorithm is used to extract the spectral power distribution. The spectral power distribution is converted into a one-dimensional spectral vector of a preset length to obtain the spectral features.

5. The method for supplementing highway slope monitoring data based on multimodal data fusion according to claim 1, characterized in that, The ternary heterogeneous multimodal features are dynamically fused using a gated prefix self-attention fusion mechanism to obtain fused features, including: The time-series numerical features, semantic text features, and spectral features are encoded respectively to obtain encoded features; Calculate the dynamic gating coefficient based on the coding characteristics; The spectral features are weighted using the dynamic gating coefficients to obtain the gated spectral features; The gated spectral features and the semantic text features are used as global prefix dimensions and concatenated to the front end of the time-series numerical feature sequence to obtain the concatenated sequence. The fused features are obtained by globally interacting with the spliced ​​sequence using a Transformer self-attention layer.

6. The method for supplementing highway slope monitoring data based on multimodal data fusion according to claim 5, characterized in that, The time-series numerical features, semantic text features, and spectral features are encoded respectively to obtain encoded features, including: After binning and discretizing the target displacement sequence, it is input into a learnable embedding layer, and then auxiliary covariates and position codes after linear projection are added to obtain the temporal backbone representation. Semantic text features are input into a fully connected layer with GELU activation and mapped to a textual latent representation; The spectral features are input into a multilayer perceptron with residual connections and a layer normalization layer to extract the spectral latent representation.

7. The method for supplementing highway slope monitoring data based on multimodal data fusion according to claim 6, characterized in that, Calculating dynamic gating coefficients based on coding features includes: Extract the global context vector of the current time window from the implicit representation of the temporal backbone; The global context vector is concatenated with the spectral implicit representation, and the dynamic gating coefficients are calculated using a Sigmoid gating network.

8. The method for supplementing highway slope monitoring data based on multimodal data fusion according to claim 1, characterized in that, The training process of the temporal completion model includes: In the first training phase, an artificial continuous block mask is applied to the target variable channel data in the sliding window sample. Based on the masked data, the ternary heterogeneous multimodal features are constructed and fused to obtain the fused features. The training model predicts the target variable channel data at the masked position. In the second training phase, the pre-trained weights from the first training phase are loaded and fine-tuned using the true missing mask.

9. The method for supplementing highway slope monitoring data based on multimodal data fusion according to claim 1, characterized in that, Applying an artificial contiguous block mask to the target variable channel data in the sliding window sample includes: According to a preset masking ratio, a continuous time segment is randomly selected from the target variable channel data and a mask is applied; The model parameters are optimized using the cross-entropy loss function, enabling the model to learn the distribution of target variable channel data at the masked location based on the fused features.

10. A method for supplementing highway slope monitoring data based on multimodal data fusion according to claim 1, characterized in that, The predicted values ​​for the missing locations output by the decoder include: Based on the historical observation distribution of the target variable, the boundary values ​​of a preset number of discrete bins are calculated using the quantile method, and the center value of each discrete bin is used as the predicted representative value of that bin. The decoder outputs the probability distribution of the missing location on each discrete bin; Based on the probability distribution and bin center values, a weighted expectation is calculated to obtain the predicted value of the continuous target variable.

Images