A multi-source perception road surface disaster early warning system based on multi-modal fusion

Abstract

The application discloses a multi-source perception pavement disaster early warning system based on multi-modal fusion, which comprises a data preprocessing module, a feature extraction module, a feature fusion module and a disaster early warning module.The feature extraction module is used for extracting a text feature extraction vector, a video feature extraction vector and a radar feature extraction vector from text data, pavement surface disease video data and radar image data of the preprocessed environment respectively, wherein the radar feature extraction vector is obtained by splicing and processing statistical distribution features, spatial structure features and dynamic evolution features.The feature fusion module is used for aligning and fusing the text feature extraction vector, the video feature extraction vector and the radar feature extraction vector to obtain a final fusion vector.The disaster early warning module comprises an LSTM-Transformer hybrid model.The application realizes full-element monitoring of road infrastructure from macro regional deformation to micro surface disease, from environmental influence factors to internal structure state, and realizes multi-source data advanced perception of pavement performance and disaster early warning.

Description

Technical Field

[0001] This invention relates to the field of road health detection technology, specifically a multi-source sensing road disaster early warning system based on multimodal fusion. Background Technology

[0002] With the continuous improvement of my country's transportation infrastructure network, road maintenance and management has shifted from a large-scale construction phase to a refined operation phase focused on safety and durability. Road surface performance degradation and geological disasters not only threaten driving safety but also cause huge economic and social losses.

[0003] Traditional pavement performance testing primarily relies on manual inspections and single-source data sensing methods. This approach offers limited sensing dimensions, identifying only surface defects and failing to detect internal subgrade problems. Furthermore, inconsistent data formats and coordinate systems across different testing devices lead to fragmented data, hindering the collaborative processing of multi-source data. Pavement disaster early warning systems based on traditional testing methods are ill-suited to the engineering requirements of early identification and accurate assessment of pavement hazards. Therefore, developing a pavement disaster early warning system capable of deeply integrating multimodal data for collaborative analysis, enabling accurate, efficient, and proactive perception of pavement conditions and risk prediction, has become extremely urgent. Summary of the Invention

[0004] To address the shortcomings of the aforementioned situation, this invention aims to solve the problem of proposing a multi-source perception road disaster early warning system based on multimodal fusion. This system integrates environmental text data, road surface defect video data, and radar image data through multimodal data fusion, breaking down the barriers between different types and sources of data, establishing a multi-source perception system encompassing air, space, and ground, and achieving comprehensive monitoring of road infrastructure from macro-regional deformation to micro-surface defects, and from environmental influencing factors to internal structural conditions. This enables advanced perception of road performance and disaster early warning based on multi-source data.

[0005] The technical solution adopted by the present invention to solve the aforementioned technical problem is as follows:

[0006] A multi-source sensing road disaster early warning system based on multimodal fusion, the system comprising:

[0007] The data preprocessing module is used to preprocess environmental text data, road surface defect video data, and radar image data.

[0008] Feature extraction module: Extracts text feature vectors from preprocessed environmental text data using Multilayer Perceptron (MLP); extracts video feature vectors from preprocessed road surface defect video data using a 3D CNN model; extracts statistical distribution features, spatial structure features, and dynamic evolution features from preprocessed radar image data, and concatenates these features to obtain radar feature extraction vectors.

[0009] The feature fusion module aligns and fuses text feature extraction vectors, video feature extraction vectors, and radar feature extraction vectors. This includes spatiotemporal alignment, a multimodal projection network, a Transformer deep fusion architecture, and a modality gating mechanism. The multimodal projection network comprises a text modality projection sub-network, a video modality projection sub-network, and a radar modality projection sub-network. The text modality projection sub-network consists of a fully connected layer followed by a normalization layer followed by a fully connected layer, receiving spatiotemporally aligned text features to obtain text projection features. The video modality projection sub-network consists of a convolutional layer followed by a pooling layer followed by a fully connected layer, receiving spatiotemporally aligned video features to obtain video projection features. The radar modality projection sub-network consists of a convolutional layer followed by a dropout layer followed by a fully connected layer, receiving spatiotemporally aligned radar features to obtain radar projection features. Finally, the outputs of each sub-network are normalized to ensure that all three modalities are mapped to a common fusion semantic space of the same dimension.

[0010] The projection features of the three modalities output by the multimodal projection network are concatenated and then input into the Transformer deep fusion architecture for fusion. The deep fusion feature matrix is ​​output. The contribution weights of the three modalities are dynamically learned through the modal gating mechanism. The projection features of the three modalities are weighted, summed and standardized to obtain the final fusion vector.

[0011] The disaster early warning module includes an LSTM-Transformer hybrid model that is self-supervised and reconstructed based on the final fusion vector output by the feature fusion module. It automatically calculates the reconstruction error and trend deviation, and fuses them to generate a quantitative road risk index to achieve disaster early warning.

[0012] Furthermore, the final fused vector is constructed using a time feature sequence with a 100-meter station number as the spatial reference and a natural year as the time step, and then input into the LSTM-Transformer hybrid model. The LSTM-Transformer hybrid model includes an LSTM hidden layer, a Transformer encoding layer, and a mask reconstruction task head.

[0013] The mask reconstruction task header outputs the reconstructed time feature sequence, calculates the reconstruction error and trend deviation, normalizes the reconstruction error and trend deviation and weights them to generate a road risk index, and determines the low, medium, high and emergency levels of risk based on the preset risk level threshold.

[0014] Furthermore, the preprocessing of the text data in the environment involves: for structured text data including road surface historical performance data, meteorological data, traffic data, and road maintenance records, the data is unified into a timestamp format and spatially converted into station coordinates based on a road linear reference system, based on 3D... The principle is to conduct outlier detection to identify outliers in environmental parameters that deviate from the normal range, while also identifying invalid and missing values, removing invalid values, filling missing values ​​in the data by calculating the arithmetic mean of all non-missing values, and finally normalizing data from different units and ranges to form consistent structured data.

[0015] The preprocessing process for the road surface defects video data is as follows: For the road surface defects video data collected by vehicle-mounted cameras and fixed monitoring equipment, the data is split into individual video frames at a fixed frame rate. The precise acquisition time and corresponding latitude and longitude of each video frame are obtained and converted into station coordinates based on the road linear reference system. All video frames are classified by year and grouped by 100-meter station intervals. A video frame sequence with "100-meter station - year" as the sample is constructed. The core video frame sequence is extracted from each group of video frame sequences. The image quality of the core video frame sequence is optimized by brightness equalization and noise reduction. The resolution and format are unified simultaneously. Finally, the processed image is normalized and converted into a five-dimensional tensor composed of the number of samples, the number of channels, the number of core video frame sequences, the video frame height, and the video frame width.

[0016] The preprocessing process for radar image data is as follows: For the raw radar image data acquired by satellite, ENVI software is used to crop and process the road vector data according to time series to generate an annual deformation rate raster map; finally, GIS software is used to overlay and analyze the road vector data and the annual deformation rate raster map to obtain settlement raster pixels in each 100-meter station interval.

[0017] Furthermore, the process of obtaining the radar feature extraction vector is as follows: for each settlement grid cell in the 100-meter station interval, its statistical distribution features are extracted, including the annual average settlement rate. Standard deviation of annual average settlement rate Maximum annual average settlement rate and the minimum annual average settlement rate Next, the center location of the settlement area within each 100-meter chainage is calculated to obtain the spatial structural characteristics; for each 100-meter chainage interval across multiple years, the average annual settlement rate within each interval is calculated with year t as the horizontal axis. Using the vertical axis as the ordinate, a time series settlement curve is generated for each 100-meter station interval through quadratic polynomial fitting. The settlement acceleration is determined based on the coefficient of the quadratic term in the settlement curve. The settlement acceleration is used as the dynamic evolution feature. The statistical distribution feature, spatial structure feature, and dynamic evolution feature are concatenated and normalized by a fully connected neural network to obtain the radar feature extraction vector.

[0018] Furthermore, the spatiotemporal alignment process is as follows: In the spatial dimension, the entire road is divided into continuous and fixed station intervals, with each station interval serving as a basic spatial analysis unit, and all data is uniformly mapped to the road's station interval; in the temporal dimension, alignment is uniformly performed using a "one-year" time window. Through temporal and spatial alignment, each station interval has a set of aligned text, video, and radar feature extraction vector data structures for each year, thus achieving spatiotemporal alignment.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0020] 1. A comprehensive and integrated multi-source data perception system has been constructed. This invention innovatively integrates environmental text data, pavement surface defect video data, and radar image data, and for the first time constructs a spatiotemporal integrated "air-ground-space" multi-source perception technology system with road linear reference system chainage as a unified spatial benchmark and "natural year" as a unified time window. This effectively overcomes the limitations of traditional single-source monitoring methods and the fragmentation of existing multi-source data. The system achieves synchronous perception from environmental influencing factors and microscopic surface defects to macroscopic regional deformation, forming a comprehensive "environment-surface-internal" monitoring network. It not only solves the problem of accurate alignment and unified mapping of multi-source heterogeneous data in the spatiotemporal dimension, but also fully leverages the complementary advantages of the three types of data in the spatial, temporal, and semantic dimensions through multimodal collaborative analysis. It comprehensively perceives road conditions from multiple dimensions and levels, thereby effectively identifying early defects and potential risks that are difficult to detect with a single modality, and significantly improving the accuracy of pavement performance degradation and disaster evolution identification and early warning timeliness.

[0021] 2. This invention achieves deep feature fusion from data alignment to semantic understanding, enabling in-depth mining and precise modeling of road condition evolution patterns. Addressing the problem that existing multi-source fusion technologies often remain at the level of simple data splicing and traditional methods frequently focus only on surface defects or local deformations, failing to correlate internal and external factors and thus leading to early warning biases, this invention proposes for the first time a multimodal projection and fusion model. This model is a three-level progressive fusion architecture consisting of a "multimodal projection network - Transformer deep fusion architecture - modal gating mechanism." By effectively integrating and dynamically weighting multi-source information into a unified semantic space, it truly achieves the leap from "data splicing" to "semantic understanding," thereby more accurately capturing the inherent patterns of defect development and the coupling effects of multiple factors. This provides a more scientific and reliable basis for in-depth mining of road condition evolution patterns and the establishment of precise early warning models.

[0022] 3. This invention constructs an LSTM-Transformer hybrid model for intelligent pavement risk assessment that eliminates the need for manual labeling, achieving a paradigm shift from "relying on human experience" to "data-driven intelligence" early warning. Traditional methods rely on periodic manual inspections, experience-based thresholds, or models trained with limited labels, resulting in high costs and significant time lag. This invention utilizes only historical, routine monitoring time-series data to automatically identify abnormal road sections deviating from long-term pavement evolution patterns, transforming passive response into proactive early warning. Ultimately, a pavement risk index of 0-100 points and its corresponding four risk levels (emergency, high, medium, and low) are calculated, which can be directly integrated with the maintenance management system, providing a clear and quantitative scientific basis for resource priority allocation and improving the intelligence and precision of maintenance management. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall structure of the multi-source sensing road disaster early warning system based on multimodal fusion of the present invention.

[0024] Figure 2 This is a schematic diagram of the feature fusion module according to an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of a disaster early warning module according to an embodiment of the present invention. Detailed Implementation

[0026] The present invention will be further explained below with reference to the embodiments and accompanying drawings, but this is not intended to limit the scope of protection of this application.

[0027] This invention is based on a multi-modal fusion multi-source sensing road disaster early warning system, such as... Figure 1 As shown, it mainly consists of four parts: data preprocessing module, feature extraction module, feature fusion module, and disaster early warning module.

[0028] I. Data Preprocessing Module

[0029] Environmental text data preprocessing: Historical data of the target road is collected, including: pavement performance data, meteorological data, traffic data, and road maintenance records, etc., in the form of structured text data. For the pavement performance data, meteorological data, traffic data, and road maintenance records, the data is standardized with a timestamp format and spatially converted into station coordinates based on the road linear reference system. Based on this, and using 3D modeling... The principle is to conduct outlier detection to identify abnormal values ​​of environmental parameters that deviate from the normal range, fill in missing values ​​by calculating the arithmetic mean of all non-missing values, and finally normalize data from different units and ranges to form consistent structured data.

[0030] The road surface performance data is obtained through periodic inspections of the road every year since its construction. In this embodiment, the length unit for the road surface performance data is set to 100 meters, and the time unit is one year.

[0031] Meteorological data includes annual average temperature, annual cumulative rainfall, and the number of days with extreme temperatures in a year.

[0032] Traffic data includes average daily traffic volume and passenger-to-freight ratio.

[0033] Road maintenance records are digitally coded and recorded for different maintenance methods corresponding to each 100-meter marker interval.

[0034] Using station number as the row index and year, average annual temperature, annual cumulative rainfall, number of days with extreme temperatures in a year, average daily traffic volume, passenger-to-freight ratio, maintenance records, and historical pavement performance data as column indexes, the data is statistically organized and stored in CSV format, enabling the data to be aligned with the benchmark in the spatiotemporal dimensions.

[0035] Subsequently, the data was cleaned: through 3 The principle of statistical methods is used to detect outliers, identify, verify and correct erroneous values ​​that deviate significantly from the reasonable range; at the same time, invalid values ​​and missing values ​​are identified, invalid values ​​are removed, and missing values ​​are filled by calculating the arithmetic mean of all non-missing values.

[0036] Next, the text dataset is divided and standardized to form consistent structured data: the cleaned data is divided into training set, validation set and test set in chronological order. Based on the mean and standard deviation parameters calculated from the training set, Z-Score standardization is performed on the training set, validation set and test set respectively to eliminate the influence of dimensions between features.

[0037] Finally, all the standardized data were organized into two-dimensional data with each "100-meter station number-year" as a sample, which was then used as input for subsequent feature extraction.

[0038] Road surface distress video data preprocessing: Using professional vehicle-mounted inspection equipment (such as vehicle-mounted cameras) or fixed monitoring equipment, inspections are conducted four times a year, once per quarter, to collect road surface distress video data. This data includes the inspection time, latitude and longitude, and distress information. The collected video data is broken down into individual video frames at a fixed frame rate. The timestamp and GPS latitude and longitude of each video frame are extracted simultaneously and converted to station coordinates based on a road linear reference system. All video frames are categorized by year and grouped by station interval (in 100 meters). A core video frame sequence is extracted from each group. The core video frame sequence undergoes brightness equalization and noise reduction to optimize image quality, while simultaneously unifying resolution and format. Finally, the processed images are normalized and converted to a five-dimensional tensor format. Specifically:

[0039] 1) In terms of time, the calendar year is used as the time base, and the cumulative period is defined as "January 1, 00:00 to December 31, 23:59 of the current year". All video data are assigned to the corresponding year according to the timestamp of the video frame and sorted in ascending order to form a continuous video frame sequence with the year as the unit. In terms of space, the latitude and longitude coordinates of each video frame are uniformly converted into station numbers under the road linear reference system through spatial projection of GIS software, and the video frame sequence is grouped according to the station number interval of 100 meters, constructing a video frame sequence with "station number-year" as the sample.

[0040] 2) Within each video frame sequence group, extract 32 core video frame sequences from the video frame sequence of each "100-meter marker - year" sample. If the number of frames is insufficient, copy the last frame to make up to 32 frames. If the number of frames is too long, select the core video frame sequence based on the defect area ratio optimization algorithm. The defect area ratio optimization algorithm is as follows: calculate the defect area ratio within each video frame, slide through all video frame sequences of the "100-meter marker - year" sample with a fixed window of 32 frames and a step size of a single video frame, and select the 32 consecutive frames with the highest average defect area ratio within the window as the core video frame sequence.

[0041] 3) Adaptive brightness equalization and noise reduction optimization are performed on the extracted 32 core video frame sequences. A bilinear interpolation algorithm is used to uniformly scale each video frame to a standard size of 224×224 pixels, and simultaneously convert each video frame to RGB three-channel format with data type converted to float32. The specific process of the bilinear interpolation algorithm is as follows: Four source pixels (2×2 pixels) are selected around the target pixel, and linear interpolation calculations are performed twice, once horizontally and once vertically.

[0042] 4) After extracting the core frame sequence and processing the images, divide all samples into training set, validation set and test set according to time order.

[0043] The maximum and minimum values ​​are calculated based on the pixel values ​​of all video frames in the training set. These two parameters are then used to perform max-min normalization on the training set, validation set, and test set, normalizing them to the range [0,1].

[0044] 5) Finally, stack the core video frame sequences processed in step 4) to form a five-dimensional tensor (number of samples, number of channels, number of core video frame sequences, video frame height, and video frame width), where the number of samples is the number of station numbers × the year, the number of channels is 3, the number of core video frame sequences is 32, the video frame height is 224, and the video frame width is 224.

[0045] Radar image data preprocessing: Taking Sentinel-1 radar image data as an example, download Sentinel-1 single-view complex radar images covering the study area from the European Space Agency Copernicus Open Access Hub, and select multiple SAR data in interferometric wide swath (IW) mode with the same orbit number and covering the same area, on a calendar year basis.

[0046] Temporal InSAR processing was performed using ENVI Sarscape software.

[0047] First, import the IW mode SAR data into ENVI Sarscape software, then crop it according to the road vector data, remove redundant areas, and focus on the target area.

[0048] Here, the SABS-InSAR method is used as an example for temporal InSAR. The file, which is imported from ENVI Sarscape software after being cropped according to road vector data, is input into the SABS-InSAR temporal processing workflow. The following steps are executed sequentially: Connection Graph, Interferometric Process, Refinement and Re-Flattening, Inversion: First Step, Inversion: Second Step, and Geocoding. Finally, an annual deformation rate raster map is generated, where the pixel value of each raster cell represents the deformation rate of that location over one year. Positive values ​​indicate uplifted raster cells, and negative values ​​indicate subsidence raster cells. The pixel value unit is millimeters. In this embodiment, only subsidence raster cells are considered, as uplifted raster cells have a relatively small impact on road surface hazards.

[0049] The road vector data and the annual deformation rate raster map generated after time-series InSAR processing were all imported into the GIS software. Starting from the road starting point, a station point was generated every 100 meters, dividing the road into continuous 100-meter station intervals. The annual deformation rate raster map was overlaid and analyzed with the road vector data divided into 100-meter station intervals. The features of the settlement raster pixels in the annual deformation rate raster map falling within each 100-meter station interval were extracted to obtain the settlement raster pixels in each 100-meter station interval.

[0050] II. Feature Extraction Module

[0051] Feature extraction from environmental text data: An MLP model is constructed to extract features from the environmental text data. The MLP model includes an input layer, multiple hidden layers, and an output layer. The training set of preprocessed environmental text data is input into the MLP model for training, and the determination coefficient of the MLP model is calculated on the validation set to evaluate its generalization ability. The trained MLP model extracts features from the preprocessed environmental text data, outputting a 256-dimensional feature vector, which is then normalized to obtain a normalized 256-dimensional text feature extraction vector.

[0052] Feature extraction from pavement surface distress video data: A 3D CNN model is used for end-to-end feature extraction from pavement surface distress video data, including an input layer, a spatiotemporal feature extraction network layer, a feature compression layer, and an output layer. The 3D CNN model is trained using a five-dimensional tensor obtained from the preprocessed pavement surface distress video data as input, and its performance is evaluated on a validation set. The spatiotemporal feature extraction network layer extracts features by stacking 3D convolutional layers and pooling layers. The feature compression layer uses 3D global average pooling to transform high-dimensional features into fixed-dimensional vectors. The output layer uses a fully connected layer with 256 neurons, outputting a 256-dimensional feature vector, which is then normalized to obtain a normalized 256-dimensional video feature extraction vector.

[0053] Feature extraction from radar image data:

[0054] Statistical distribution feature extraction: For each settlement grid cell in the 100-meter station interval, its statistical distribution features are extracted, including the annual average settlement rate. (Arithmetic mean of the sum of settlement rate values ​​of all settlement raster cells), standard deviation of annual average settlement rate Extreme values ​​of annual average settlement rate (including maximum values) and minimum value );

[0055] Spatial structural feature extraction: For each settlement grid cell within a 100-meter chainage interval, calculate the center position of the settlement area within each 100-meter chainage interval. , The calculation formula is:

[0056]

[0057] in: The total number of all settlement grid cells within each 100-meter station interval; Within each 100-meter station interval The absolute value of the settlement rate of each settlement raster cell; , Within each 100-meter station interval The center coordinates of a settling raster cell.

[0058] Dynamic evolution feature extraction: For each 100-meter station interval across multiple years, based on year... The horizontal axis represents the average annual settlement rate within each 100-meter station interval. Using the vertical axis as the ordinate, time-series settlement curves for each 100-meter station interval are generated through quadratic polynomial fitting. ,

[0059] in, Represents the settlement curve. , , These represent the fitting coefficients of the settlement curves;

[0060] The settlement acceleration is determined by the coefficient of the quadratic term in the settlement curve. Positive acceleration indicates a decrease in the settlement rate, while negative acceleration indicates an increase in the settlement rate.

[0061] Finally, all the feature values ​​extracted from each station segment are concatenated into a form like [ , , , , , , The feature vector is input into a fully connected neural network. Through linear mapping and nonlinear activation, a 256-dimensional feature vector is output. This vector is then normalized to obtain a normalized 256-dimensional radar feature extraction vector.

[0062] In this embodiment, the fully connected neural network consists of an input layer, a hidden layer, and an output layer. The input layer has 7 neurons, the hidden layer consists of two fully connected layers with 64 and 128 neurons respectively, and the output layer is also a fully connected layer with 256 neurons. The data dimension is changed from 7 to 64 to 128 to 256 through the fully connected neural network.

[0063] III. Feature Fusion Module

[0064] First, data-level spatiotemporal alignment is achieved: spatially, the continuous and fixed 100-meter marker intervals of this road are used as the benchmark, and temporally, a "one-year" time window is used as the time window. All three types of data after feature extraction are mapped to the corresponding marker intervals and corresponding years, thus constructing a structured data system of "100-meter marker - year - feature extraction vectors of each of the three modalities".

[0065] Next, a multimodal projection and fusion model is constructed: the multimodal projection and fusion model includes a multimodal projection network, a Transformer deep fusion architecture, and a modal gating mechanism, and its input is a three-modal feature extraction vector that has been aligned to the data level spatiotemporal dimensions.

[0066] The multimodal projection network includes three independent projection sub-networks: a text modal projection sub-network, a video modal projection sub-network, and a radar modal projection sub-network, which unify the original semantic spaces where text, video, and radar features differ.

[0067] The text modality projection subnetwork consists of "fully connected layer - normalization layer - fully connected layer", and the text projection features are obtained by inputting the text feature extraction vector.

[0068] The video modal projection subnetwork consists of "convolutional layer - pooling layer - fully connected layer", and the video projection features are obtained by inputting the video feature extraction vector.

[0069] The radar modal projection subnetwork consists of "convolutional layer - dropout layer - fully connected layer", and the radar projection features are obtained by inputting the radar feature extraction vector.

[0070] Finally, the outputs of each sub-network are normalized to ensure that all three modalities are mapped to a common fusion semantic space of the same dimension.

[0071] The projected features (i.e., the projected features of the three modalities) enter the cross-modal fusion stage, using the Transformer deep fusion architecture:

[0072] The projected features of the three modalities are concatenated to form the input matrix X, which is then input into the Transformer deep fusion architecture. This architecture has multiple identical Transformer encoder layers, each consisting of a multi-head self-attention sub-layer and a feedforward network sub-layer, with residual connections and normalization applied after each sub-layer. The multi-head attention matrix X1 output from the multi-head self-attention sub-layer of the first encoder layer is directly added to the input matrix X. After addition, a residual connection is performed and normalization is applied. The normalized matrix is ​​then input into the feedforward network sub-layer for nonlinear transformation, followed by another residual connection and normalization to obtain the output of the first encoder layer. The output of the first encoder layer is used as the input to the second encoder layer. After multiple iterations, the output of the last encoder layer is the deep fusion feature matrix X. final .

[0073] Then the deep fusion feature matrix X final The input modal gating mechanism for intermediate information sources dynamically adjusts the contribution ratios of the three modalities. This mechanism is based on X. final The encoded context information is used to dynamically generate a set of gating weights. This set of weights is then weighted and fused with the projection features of the three previously input modalities, and the fusion vector is directly calculated using the formula "weight × corresponding modal projection feature". The formula is as follows:

[0074]

[0075] in: This represents the m-th modality, with values ​​of 1, 2, and 3, corresponding to text modality, video modality, and radar modality, respectively. Indicates the first Modal projection features; The corresponding first generation generated for the modal gating mechanism Dynamic weights for class modalities.

[0076] Finally, the fusion vector will be... Standardization is performed to obtain the final fusion vector. .

[0077] This invention takes into account the differences and adaptability of the outputs of different projection subnetworks, and innovatively introduces weight coefficients to perform weighted fusion of the contribution weights of different modes.

[0078] The quality of the multimodal projection and fusion model is evaluated by a multi-objective loss function, which consists of projection network alignment loss, feature fusion loss, and model consistency loss. The loss value of the multi-objective loss function continuously decreases and stabilizes at a low level, indicating that the model has been successfully optimized.

[0079] IV. Disaster Early Warning Module

[0080] The disaster early warning module includes an LSTM-Transformer hybrid model that uses the final fusion vector output by the feature fusion module to reconstruct a self-supervised model based on a mask. This model automatically calculates the reconstruction error and trend deviation, and fuses them to generate a quantitative road risk index for disaster early warning. The LSTM-Transformer hybrid model undergoes self-supervised learning, trained through a mask reconstruction task head. It learns the long-term evolution of the road surface by predicting the randomly masked portion, and finally outputs the reconstructed time feature sequence. By calculating the reconstruction error and trend deviation in the original and reconstructed time feature sequences, the two are normalized and fused according to preset weights to generate a quantitative road risk index of 0-100. Finally, based on preset thresholds, it outputs four risk levels: emergency, high, medium, and low.

[0081] First, the final fusion vector output by the feature fusion module... The time feature sequence is organized with the 100-meter station number as the spatial reference and the year as the time step, and is denoted as the original time feature sequence.

[0082] Organize all time feature series as The tensor format constitutes the early warning dataset, where: Number of samples (station number × year); For time step; for The feature dimensions.

[0083] Using the early warning dataset, a hybrid LSTM-Transformer model was selected for self-supervised training and prediction, without relying on any labeled data. Its architecture consists of "input layer—unidirectional LSTM hidden layer—Transformer encoding layer—mask reconstruction task head". The training process is as follows: first, the time feature sequence of each station number is processed according to a preset time scale. Split into Each time block contains non-overlapping time blocks, and each time block includes The final fusion vector for consecutive years is obtained. Then, the masking ratio is determined to be 15% based on the characteristics of road surface time series data. Uniform random sampling is used to extract time blocks to be masked from all time blocks. Only the final fusion vector is partially masked in the selected time blocks, while the station number and year are kept intact. This yields a damage time feature sequence consistent with the original time feature sequence in this step. The model input layer receives the damage time feature sequence, and the local temporal dependencies are captured by the gating mechanism of the LSTM hidden layer. Its output is then fed into the Transformer encoding layer, where the correlation weights between global time steps are dynamically calculated using a multi-head attention mechanism. Finally, it is fed into the mask reconstruction task header, and the reconstructed time feature sequence is output.

[0084] The LSTM-Transformer hybrid model uses a composite loss function consisting of "mask reconstruction loss" and "time trend constraint loss", the formula of which is:

[0085]

[0086]

[0087]

[0088] in, This is a composite loss; Loss due to mask reconstruction; Loss is constrained by time trend; These are the loss weighting coefficients; This represents the total number of final fused vectors masked within all masked time blocks in the corrupted time feature sequence; The feature dimension of the final fused vector; The first element in the original time feature sequence The final fused vector at the nth mask position is the first... 3D eigenvalues; For the reconstructed time feature sequence, the first... The final fused vector at the nth mask position is the first... 3D eigenvalues; The time step of the time feature sequence; The first element in the original time feature sequence The final fusion vector corresponding to the nth time step 3D eigenvalues; The first element in the original time feature sequence The final fusion vector corresponding to the nth time step 3D eigenvalues; For the reconstructed time feature sequence, the first... The final fusion vector corresponding to the nth time step 3D eigenvalues; For the reconstructed time feature sequence, the first... The final fusion vector corresponding to the nth time step 1-dimensional eigenvalues.

[0089] After the LSTM-Transformer hybrid model is trained, the time feature sequence to be evaluated is input into the trained LSTM-Transformer hybrid model, the corresponding reconstructed time feature sequence is output, and the reconstruction error is calculated. and trend deviation Next, the reconstruction error of all stations in the training set is calculated. and trend deviation The maximum and minimum values, and the reconstruction error for each 100-meter station based on these maximum and minimum values. and trend deviation Max-min normalization is performed to normalize to the range [0, 1]. Then, a weighted fusion is performed according to a pre-set weight (70% for reconstruction error and 30% for trend deviation). The result is then multiplied by 100 to convert to a percentage, thus obtaining the pavement risk index for that station. The project risk levels are divided into four categories based on preset thresholds (see Table 1): 90-100 is considered emergency risk, 70-89 is considered high risk, 50-69 is considered medium risk, and 0-49 is considered low risk.

[0090] Reconstruction error formula:

[0091] Trend deviation formula:

[0092] Road surface risk index:

[0093] in, For the first Reconstruction error of a 100-meter station number; For the first Trend deviation at each 100-meter marker; For the normalized first Reconstruction error of a 100-meter station number; For the normalized first Trend deviation at each 100-meter marker;

[0094] For the first Time step for each 100-meter marker; The first time feature sequence to be evaluated 100-meter marker The final fusion vector corresponding to the nth time step 3D eigenvalues; The first time feature sequence to be evaluated 100-meter marker The final fusion vector corresponding to the nth time step 3D eigenvalues; To reconstruct the time feature sequence of the first 100-meter marker The final fusion vector corresponding to the nth time step 3D eigenvalues; To reconstruct the time feature sequence of the first 100-meter marker The final fusion vector corresponding to the nth time step 3D eigenvalues; For the first The road surface risk index at each 100-meter marker.

[0095] Table 1 shows the comparison table of road risk index levels.

[0096]

[0097] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A multi-source perception road hazard early warning system based on multi-modal fusion, characterized in that, The system comprises: a data preprocessing module for preprocessing text data, pavement surface disease video data, and radar image data of an environment; a feature extraction module for extracting a text feature extraction vector from the preprocessed text data of the environment using a multilayer perceptron (MLP), extracting a video feature extraction vector from the preprocessed pavement surface disease video data using a 3D CNN model, and extracting statistical distribution features, spatial structure features, and dynamic evolution features from the preprocessed radar image data, and obtaining a radar feature extraction vector by concatenating the statistical distribution features, spatial structure features, and dynamic evolution features; a feature fusion module for aligning and fusing the text feature extraction vector, video feature extraction vector, and radar feature extraction vector, including spatiotemporal dimension alignment, a multi-modal projection network, a Transformer deep fusion architecture, and a modal gating mechanism; the multi-modal projection network comprises a text modal projection subnetwork, a video modal projection subnetwork, and a radar modal projection subnetwork, the text modal projection subnetwork is composed of a "full connection layer-normalization layer-full connection layer", and the text projection feature is obtained by inputting the spatiotemporal dimension aligned text feature; the video modal projection subnetwork is composed of a "convolutional layer-pooling layer-full connection layer", and the video projection feature is obtained by inputting the spatiotemporal dimension aligned video feature; the radar modal projection subnetwork is composed of a "convolutional layer-Dropout layer-full connection layer", and the radar projection feature is obtained by inputting the spatiotemporal dimension aligned radar feature; finally, the outputs of the subnetworks are normalized to ensure that the three types of modalities are mapped to the same dimension of the public fusion semantic space; the projection features of the three types of modalities output by the multi-modal projection network are concatenated and input into the Transformer deep fusion architecture for fusion, and a deep fusion feature matrix is output, which is then input into the modal gating mechanism to dynamically learn the contribution weights of the three types of modalities, and the projection features of the three types of modalities are weighted and summed to obtain a final fusion vector; a disaster warning module comprising an LSTM-Transformer hybrid model based on mask reconstruction self-supervised training, which automatically calculates reconstruction error and trend deviation, and fuses to generate a quantitative pavement risk index to realize disaster warning. The process of preprocessing the text data of the environment is: for the text structured data including road surface historical performance data, meteorological live data, traffic data, road maintenance records, the data is uniformly timestamped, and is converted into pile number coordinates based on the road linear reference system in space, based on the 3 The principle is to carry out outlier detection to identify environmental parameter outliers deviating from the normal range, identify invalid values and missing values, remove invalid values, fill in missing values by calculating the arithmetic mean of all non-missing values, and finally normalize data of different units and different ranges to form consistent structured data. The process of preprocessing the road surface apparent disease video data is: for the collected road surface apparent disease video data, split into single video frames according to a fixed frame rate, obtain the accurate collection time and corresponding latitude and longitude of each video frame and convert them into pile number coordinates based on the linear reference system of the road, classify all video frames by year, group the video frame sequences by 100-meter pile number intervals, construct video frame sequences with "100-meter pile number-year" as samples, and extract core video frame sequences in each group of video frame sequences, optimize the image quality of the core video frame sequences through brightness equalization and denoising, synchronize the resolution and format, and finally normalize the processed images and convert them into five-dimensional tensors composed of sample number, channel number, core video frame sequence frame number, video frame height and video frame width; The process of preprocessing the radar image data is: for the satellite-acquired radar original image data, use ENVI software to crop and perform time series processing according to the road vector data, generate annual deformation rate raster images; finally, overlay the road vector data and the annual deformation rate raster images through GIS software to obtain the settlement grid pixels in each 100-meter pile number interval.

2. The multi-modal fusion based multi-source perception road hazard warning system according to claim 1, wherein, The final fusion vector is constructed into a time feature sequence with 100-meter pile number as the spatial reference and natural year as the time step, which is input into the LSTM-Transformer hybrid model; the LSTM-Transformer hybrid model includes LSTM hidden layers, Transformer encoding layers and mask reconstruction task heads. The mask reconstruction task head outputs the reconstructed time feature sequence, calculates the reconstruction error and trend bias, normalizes and weights the reconstruction error and trend bias to generate the road risk index, and determines the low, medium, high and urgent four levels of risk according to the threshold of the preset risk level.

3. The multi-modal fusion based multi-source perception road hazard warning system according to claim 1, wherein, The process of obtaining the radar feature extraction vector is: for the settlement grid pixels in each 100-meter pile number interval, extract their statistical distribution features, The statistical distribution features include annual average settlement rate , annual average settlement rate standard deviation , annual average settlement rate maximum value , and annual average settlement rate minimum value ; then the center position of the settlement area within each 100-meter stake number is calculated to obtain the spatial structure features; for each 100-meter stake number interval of multiple years, taking year t as the horizontal axis and the annual average settlement rate within each 100-meter stake number interval as the vertical axis, a time series settlement curve of each 100-meter stake number interval is generated through quadratic polynomial fitting, the settlement acceleration is determined according to the coefficient of the quadratic term in the settlement curve, and the settlement acceleration is taken as the dynamic evolution feature. After the statistical distribution features, the spatial structure features, and the dynamic evolution feature are spliced and normalized through a fully connected neural network, a radar feature extraction vector is obtained.

4. The multi-modal fusion based multi-source perception road hazard warning system of claim 1, wherein, The process of aligning the space-time dimensions is: in the spatial dimension, divide the entire road into continuous and fixed pile number intervals, each pile number interval as a basic spatial analysis unit, and map all data to the pile number interval of the road; in the time dimension, align with "one year" as the time window, through the alignment of time and space, in each pile number interval, there is a group of aligned text, video and radar feature extraction vector data structures for each year, realizing the alignment of space-time dimensions. The process of aligning the space-time dimensions is: in the spatial dimension, divide the entire road into continuous and fixed pile number intervals, each pile number interval as a basic spatial analysis unit, and map all data to the pile number interval of the road; in the time dimension, align with "one year" as the time window, through the alignment of time and space, in each pile number interval, there is a group of aligned text, video and radar feature extraction vector data structures for each year, realizing the alignment of space-time dimensions.

Images