Tunnel pavement spillage warning method based on multimodal perception

Abstract

The present invention relates to the technical field of tunnel safety management, and in particular to a tunnel pavement spillage warning method based on multimodal perception, comprising the following steps: obtaining multimodal data of a target tunnel pavement; performing feature extraction on a visible light image to obtain feature information of the spillage; inputting three-dimensional point cloud data into an LSTM network to obtain the short-term spatial displacement of the spillage; constructing an ST-GNN network, inputting the short-term spatial displacement of the spillage into the ST-GNN network to obtain an enhanced spatiotemporal feature vector; inputting visible light images and thermal radiation images into a Transformer network to obtain long-term distribution heat map features; combining the enhanced spatiotemporal feature vector and the long-term distribution heat map features to obtain a spatiotemporal distribution prediction result of the spillage; and obtaining a tunnel traffic risk warning result based on the multimodal data and the spatiotemporal distribution prediction result of the spillage. The present invention can provide timely and effective warning information.

Description

Technical Field

[0001] The present invention relates to the technical field of tunnel safety management, and in particular to a method for predicting the spatiotemporal evolution of tunnel pavement spillage based on multimodal perception. Background Art

[0002] With the rapid development of modern transportation infrastructure, tunnels, as a vital component of road and rail networks, are attracting increasing attention for their safety and stability. However, the presence of debris (such as oil, debris, and accumulated water) in tunnels not only poses a direct threat to driving safety but can also cause secondary hazards such as fires, structural damage, and traffic congestion. In recent years, a number of technologies have been applied to the detection and management of tunnel debris, including image recognition based on traditional cameras, temperature monitoring using infrared thermal imaging, and spatial scanning using LiDAR. These technologies have achieved some success in specific scenarios. For example, a single camera system can initially identify the presence of debris, infrared sensors can detect high-temperature objects, and LiDAR provides basic spatial distribution information. Furthermore, existing technologies have also explored some preliminary approaches to debris prediction, such as predicting the area change of debris using simple threshold analysis or estimating the location and movement of debris using traditional time series models (such as ARIMA). These prediction methods can provide some reference in static scenarios, but overall they remain relatively rudimentary.

[0003] However, existing tunnel debris detection and risk assessment technologies present significant technical challenges. First, detection methods primarily rely on single sensors or traditional image processing methods, which are limited by complex environmental factors such as viewing angle limitations, uneven lighting, and fog interference, significantly compromising data accuracy and comprehensiveness. For example, in low-light conditions, cameras struggle to capture details of debris. In high-temperature or smoky environments, single infrared and LiDAR data fail to fully reflect the characteristics of debris. Second, existing technologies are particularly deficient in predicting the spatiotemporal evolution of debris. Prediction methods based on threshold analysis are overly simplistic, neglecting interactions between debris and the impact of environmental dynamics. Traditional time series models (such as ARIMA) can only handle single-dimensional positional changes and lack the ability to model multimodal features (such as temperature and type) and their spatial relationships. This lack of predictive capability makes it difficult for existing technologies to accurately predict the spread and potential risks of debris, and consequently, to provide tunnel managers with timely and effective early warning information. Summary of the Invention

[0004] The purpose of the present invention is to provide a tunnel pavement spillage warning and prediction method based on multimodal perception, which realizes the dynamic prediction of the spatiotemporal evolution of spillage and the early warning of potential risks.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] The tunnel pavement spillage warning method based on multimodal perception includes:

[0007] Acquiring multimodal data of a target tunnel pavement, wherein the multimodal data includes visible light images, thermal radiation images, and three-dimensional point cloud data;

[0008] Performing feature extraction on the visible light image to obtain feature information of the scattered objects;

[0009] Input the three-dimensional point cloud data into the LSTM network to obtain the spatial displacement of the scattered object in the short term;

[0010] Constructing an ST-GNN network based on the characteristic information of the scattered object and the multimodal data, inputting the spatial displacement of the scattered object in a short period of time into the ST-GNN network to obtain an enhanced spatiotemporal feature vector;

[0011] Inputting the visible light image and the thermal radiation image into a Transformer network to obtain long-term distribution heat map features;

[0012] Combining the enhanced spatiotemporal feature vector and the long-term distribution heat map feature to obtain a spatiotemporal distribution prediction result of the scattered objects;

[0013] A tunnel traffic risk warning result is obtained based on the multimodal data and the spatiotemporal distribution prediction result of the scattered objects.

[0014] Optionally, obtaining multimodal data of the target tunnel pavement includes:

[0015] Acquire an initial visible light image using a high-resolution camera, perform non-local mean denoising and Retinex illumination compensation on the initial visible light image, and obtain the visible light image;

[0016] Collecting the thermal radiation image by an infrared thermal imaging sensor;

[0017] Initial three-dimensional point cloud data is collected by laser radar, and the initial three-dimensional point cloud data is aligned with the visible light image to obtain the three-dimensional point cloud data.

[0018] Optionally, performing feature extraction on the visible light image to obtain feature information of the scattered object includes:

[0019] The visible light image is input into the YOLOv11 network, the CNN network and the ViT network respectively, and the location and category of the spilled object, the local feature map and the global feature map are output respectively;

[0020] The intermediate feature map of the YOLOv11 network, the local feature map of the CNN network, and the global feature map of the ViT network are fused to obtain feature information of the scattered objects.

[0021] Optionally, the method for fusing the intermediate feature map of the YOLOv11 network, the local feature map of the CNN network, and the global feature map of the ViT network is:

[0022] ;

[0023] Among them, Fusion is a comprehensive feature representation, α, β, γ are weight coefficients, 、 、 They are the intermediate feature map of YOLOv11, the local feature map of CNN, and the global feature map of ViT.

[0024] Optionally, constructing a spatiotemporal graph neural network includes: constructing the spatiotemporal graph neural network by using nodes to represent the spatial positions of spilled objects and edges to represent the spatial relationships between adjacent spilled objects or regions, wherein the node feature representation includes coverage area, type of spilled objects, temperature and characteristic information of the spilled objects.

[0025] Optionally, calculation of the short-term spatial displacement of the spilled object includes:

[0026] ;

[0027] Among them, ΔA is the spatial displacement of the scattered objects in the short term, and is the coverage area at adjacent time points, is the time interval.

[0028] Optionally, obtaining a tunnel traffic risk warning result based on the multimodal data and the spatiotemporal distribution prediction result of the spilled objects includes: calculating a current traffic risk index and a heat risk index based on the multimodal data, and calculating a future traffic risk index and a future heat risk index based on the spatiotemporal distribution prediction result of the spilled objects;

[0029] The current traffic risk index is:

[0030] ;

[0031] ;

[0032] in, is the adjusted current traffic risk index, k is the thermal sensitivity coefficient, is the temperature difference, is the adjustment coefficient of the spatial displacement of the scattered objects in the short term, A is the current coverage area, P is the current position weight, R is the current traffic risk index, is the weight coefficient of coverage area, is the weight coefficient of position weight;

[0033] The heat risk index is:

[0034] ;

[0035] in, is the short-term temperature change value predicted by ST-GNN, is the adjustment coefficient of short-term temperature change value, H is the heat risk index;

[0036] The future traffic risk index is:

[0037] ;

[0038] The future heat wave risk index is:

[0039] ;

[0040] in, is the future traffic risk index, is the future heat wave risk index, 、 、 The coverage area, position, and temperature change values for the predicted future time steps.

[0041] Optionally, obtaining the tunnel traffic risk warning results also includes: inputting the type of spilled objects, the current location and the spatial displacement of the spilled objects in the short term into the Bayesian network, and outputting the congestion and collision probabilities, wherein the Bayesian network is obtained through training of historical spilled object types, locations and the spatial displacement of the spilled objects in the short term and the corresponding congestion and collision probabilities.

[0042] The beneficial effects of the present invention are as follows: (1) Compared with the existing technology, the present invention can deeply analyze the characteristics of scattered objects from three dimensions: visual, thermal and spatial, and comprehensively reveal their potential threats to traffic, fire and chain accidents, overcoming the limitations of the traditional single perspective and giving the method better risk insight.

[0043] (2) Compared with the existing technology, the present invention achieves ultra-high-precision capture of the type, location and evolution trend of scattered objects through the integration of cutting-edge image processing, multi-model collaboration and spatiotemporal prediction technology. The detection accuracy exceeds 95%, and the prediction error is compressed to the centimeter level, far ahead of the extensive judgment of traditional methods.

[0044] (3) Compared with the existing technology, the present invention reduces the time consumption of the entire process from data collection to risk warning to milliseconds by virtue of the instantaneous optimization of sensor parameters and the reasoning ability of the detection model, overcoming the bottleneck of traditional manual inspection or single-mode inefficiency and demonstrating the advantage of intelligent response.

[0045] (4) Compared with the existing technology, the present invention endows the method with excellent adaptability under complex conditions such as lighting and smoke through dynamic optimization and distributed model evolution driven by real-time environmental feedback. At the same time, it continuously improves the generalization performance across scenarios, surpassing the rigid design of traditional fixed parameters and becoming a model of intelligent evolution in the field of tunnel safety management.

[0046] In summary, this method first uses multimodal sensors (high-resolution cameras, infrared thermal imaging, and LiDAR) to collaboratively collect data. It then utilizes image processing techniques such as denoising (such as non-local means denoising), illumination compensation (such as the Retinex algorithm), and geometric distortion correction to improve the quality of multi-source data and enhance the recognizability of spilled object features. Furthermore, this method utilizes deep learning models (such as YOLOv11, convolutional neural networks (CNNs), and visual transformers (ViTs)) to efficiently classify and accurately identify spilled objects. This method is capable of distinguishing different types of spilled objects (such as garbage, rocks, and oil) in real time in complex environments, overcoming the limitations of traditional methods in detecting object morphology and dynamic changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 This is a flow chart of a multimodal sensing tunnel pavement spillage warning method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1 As shown, this embodiment provides a tunnel pavement spillage warning method based on multimodal perception, including:

[0052] Acquire multimodal data of the target tunnel pavement, wherein the multimodal data includes visible light images, thermal radiation images, and three-dimensional point cloud data;

[0053] Perform feature extraction on visible light images to obtain characteristic information of the scattered objects;

[0054] Input the 3D point cloud data into the LSTM network to obtain the short-term spatial displacement of the scattered objects;

[0055] An ST-GNN network is constructed based on the characteristic information of the scattered objects and multimodal data. The spatial displacement of the scattered objects in the short term is input into the ST-GNN network to obtain the enhanced spatiotemporal feature vector.

[0056] Input the visible light image and thermal radiation image into the Transformer network to obtain the long-term distribution heat map features;

[0057] The spatiotemporal distribution prediction results of the scattered objects are obtained by combining the enhanced spatiotemporal feature vector and the long-term distribution heat map features;

[0058] Based on multimodal data and the spatiotemporal distribution prediction results of scattered objects, tunnel traffic risk warning results are obtained.

[0059] Furthermore, obtaining multimodal data of the target tunnel pavement includes:

[0060] An initial visible light image is collected using a high-resolution camera, and non-local mean denoising and Retinex illumination compensation are performed on the initial visible light image to obtain a visible light image.

[0061] Collect thermal radiation images through infrared thermal imaging sensors;

[0062] The initial three-dimensional point cloud data is collected by laser radar, and the initial three-dimensional point cloud data is aligned with the visible light image to obtain the three-dimensional point cloud data.

[0063] Specifically, various sensors are deployed at key locations in the tunnel, such as busy traffic areas, tunnel entrances, and areas prone to falling objects. Cameras (HRC) are typically installed on the top or sidewalls of the tunnel to ensure maximum coverage. Infrared sensors are deployed at varying heights to more accurately capture temperature fluctuations. LiDAR sensors are placed at key nodes, using multi-line scanning to enhance the accuracy of 3D point clouds.

[0064] Furthermore, feature extraction is performed on the visible light image to obtain feature information of the scattered objects, including:

[0065] The visible light image is input into the YOLOv11 network, CNN network and ViT network respectively, and the location and category of the spilled objects, local feature map and global feature map are output respectively;

[0066] The intermediate feature map of the YOLOv11 network, the local feature map of the CNN network, and the global feature map of the ViT network are fused to obtain the feature information of the scattered objects.

[0067] Specifically, this embodiment combines local texture, global structural features and deep learning detection models to accurately identify spilled objects in tunnels.

[0068] Furthermore, the method for fusing the intermediate feature map of the YOLOv11 network, the local feature map of the CNN network, and the global feature map of the ViT network is as follows:

[0069] ;

[0070] Among them, Fusion is a comprehensive feature representation, α, β, γ are weight coefficients, 、 、 They are the intermediate feature map of YOLOv11, the local feature map of CNN, and the global feature map of ViT.

[0071] Furthermore, constructing a spatiotemporal graph neural network includes: constructing a spatiotemporal graph neural network with nodes representing the spatial positions of spilled objects and edges representing the spatial relationships between adjacent spilled objects or regions, wherein the node features include coverage area, type of spilled objects, temperature and characteristic information of the spilled objects.

[0072] Specifically, compared to LSTM's temporal modeling and Transformer's global trend analysis, ST-GNN's unique advantage lies in its sensitivity to spatial structure, revealing the mutual influences between spilled objects (such as the chain reaction of diffusion). This is particularly critical in complex tunnel environments where multiple objects coexist. For example, when oil spills spread, ST-GNN can predict their indirect impact on the distribution of nearby accumulated water or gravel, thereby improving the comprehensiveness and accuracy of the prediction.

[0073] Furthermore, the calculation of the spatial displacement of the scattered objects in the short term includes:

[0074] ;

[0075] Among them, ΔA is the spatial displacement of the scattered objects in the short term, and is the coverage area at adjacent time points, is the time interval.

[0076] Furthermore, based on the multimodal data and the spatiotemporal distribution prediction results of the spilled objects, tunnel traffic risk warning results are obtained, including: calculating the current traffic risk index and heat risk index based on the multimodal data, and calculating the future traffic risk index and future heat wind risk index based on the spatiotemporal distribution prediction results of the spilled objects;

[0077] The current traffic risk index is:

[0078] ;

[0079] ;

[0080] in, is the adjusted current traffic risk index, k is the thermal sensitivity coefficient, is the temperature difference, is the adjustment coefficient of the spatial displacement of the scattered objects in the short term, A is the current coverage area, P is the current position weight, R is the current traffic risk index, is the weight coefficient of coverage area, is the weight coefficient of position weight;

[0081] The heat risk index is:

[0082] ;

[0083] in, is the short-term temperature change value predicted by ST-GNN, is the adjustment coefficient of short-term temperature change value, H is the heat risk index;

[0084] The future traffic risk index is:

[0085] ;

[0086] The future heat wave risk index is:

[0087] ;

[0088] in, is the future traffic risk index, is the future heat wave risk index, 、 、 The coverage area, position, and temperature change values for the predicted future time steps.

[0089] Furthermore, obtaining tunnel traffic risk warning results also includes: inputting the type of spilled objects, current location and short-term spatial displacement of the spilled objects into the Bayesian network, and outputting the congestion and collision probabilities, wherein the Bayesian network is obtained through historical spilled object types, locations and short-term spatial displacements of the spilled objects and the corresponding congestion and collision probabilities training.

[0090] The following further describes the method of this implementation:

[0091] The tunnel pavement spillage warning method based on multimodal perception specifically includes the following steps:

[0092] Step 1: Multimodal image acquisition:

[0093] Various sensors are deployed at key locations in tunnels (such as busy traffic areas, tunnel entrances, and areas prone to falling objects). Cameras (HRC) are typically installed on the top or sidewalls of the tunnel to ensure maximum coverage. Infrared sensors are deployed at varying heights to more accurately capture temperature fluctuations. LiDAR sensors are placed at key nodes, using multi-line scanning to enhance the accuracy of 3D point clouds. This collaborative multi-sensor approach to collecting multidimensional data overcomes the blind spots of a single camera in complex environments and improves data comprehensiveness.

[0094] Multimodal sensors (HRC cameras, infrared thermal imaging sensors, LiDAR) can effectively collect various environmental data through precise spatiotemporal synchronization and avoid information loss caused by perspective or local interference of a single sensor.

[0095] Real-time computer-controlled calibration ensures inter-sensor coordinate system alignment, automatically detecting and correcting lens distortion and time synchronization errors. Dedicated calibration scenarios (such as standard calibration targets) are used to verify the error range of each sensor, and data fusion algorithms are used for precise correction.

[0096] According to the ambient light intensity , adjust the exposure time, increase the exposure time in low light environment .

[0097] Automatically adjust the LiDAR scanning frequency based on the ambient smoke concentration or the influence of obstacles to reduce data redundancy and optimize data accuracy.

[0098] HRC collects visible light images ( ), used to detect spilled objects (such as garbage and dropped objects) in the tunnel;

[0099] Infrared sensor acquires thermal radiation image ( ), detect changes in heat sources (such as hot spills or high-temperature objects left behind by vehicles, temperature differences , is the preset threshold);

[0100] LiDAR generates 3D point cloud data ( ), which is used to scan obstacles on the tunnel floor and provide spatial information for subsequent analysis.

[0101] Step 2: Multi-scale image preprocessing:

[0102] This step systematically preprocesses the RGB image and LiDAR point cloud data to extract the visual, thermal, and spatial characteristics of the scattered objects and perform preliminary quantification. The following is the specific implementation process:

[0103] 1. Non-Local Means Denoising:

[0104] This method denoises random noise in RGB images by using similarities between image patches. Specifically, it calculates a weighted average of the area surrounding each pixel. The weights are dynamically and adaptively calculated based on local texture and global structure, ensuring that detailed information about the scattered objects (such as edges and texture) is preserved while suppressing noise.

[0105] 2. Retinex lighting compensation:

[0106] Under complex tunnel lighting conditions (such as strong light or shadows), the features of scattered objects in RGB images are easily obscured by uneven lighting. The Retinex algorithm effectively reduces lighting interference by decomposing the image into reflective and illumination components.

[0107] 3. 3D point cloud registration and distortion correction:

[0108] The point cloud data generated by the LiDAR is aligned with the RGB image through registration to eliminate geometric errors caused by different viewing angles. The specific method is to optimize the transformation matrix T (including rotation and displacement) so that the 3D coordinates in the point cloud correspond to the pixel positions in the image.

[0109] The registered point cloud data not only provides the precise spatial distribution of the spilled objects, but also lays the foundation for subsequent quantitative analysis.

[0110] 4. Multimodal data processing and feature extraction:

[0111] RGB image feature enhancement: RGB images that have undergone denoising and illumination compensation are further used to extract visual features of the scattered objects (such as edges, color, and texture), providing high-quality input for type recognition.

[0112] Infrared image temperature difference analysis: Infrared images are analyzed by detecting temperature differences ( ), using the preset threshold ( ) Identify abnormally high temperature areas (such as hot liquids or burning materials), mark potential heat-sensitive materials, and provide a basis for fire risk assessment.

[0113] Spatial quantification of LiDAR point clouds: The registered point cloud data is segmented and clustered to extract the spatial area of the spilled objects, calculate their geometric parameters (such as coverage area, volume, and thickness), and combine the position and depth information of the RGB image to generate spatial distribution characteristics.

[0114] Step 3: Intelligent detection and identification of spilled objects:

[0115] After image preprocessing in step 2, a high-quality RGB image is received. Multi-level features are extracted using a parallel YOLOv11, CNN, and ViT network. Ultimately, dynamic fusion is used to accurately identify and classify spilled objects. This process includes distinguishing between different types of spilled objects (such as garbage, obstacles, and leaked materials) and determining the size and shape of the objects based on their morphological complexity.

[0116] 1. Multi-feature extraction:

[0117] This example uses three parallel network branches to process the input RGB image, with each network focusing on extracting features at a different level. The following is a detailed description:

[0118] YOLOv11: The YOLOv11 model takes a preprocessed RGB image as input and outputs the location and category of spilled objects (such as garbage, rocks, and dropped tools). The YOLOv11 model offers strong real-time and multi-category recognition capabilities, making it suitable for detecting complex and rapidly changing objects in tunnel environments. It directly extracts spatial features from images to detect and classify various objects, making it particularly suitable for quickly locating spilled objects.

[0119] This example uses the YOLOv11 model for fast and high-precision object detection, making it particularly suitable for detecting different types of spilled objects (such as garbage, rocks, and dropped tools). The YOLOv11 model directly extracts spatial features from images to detect and classify various objects.

[0120] CNN branch: The input of the CNN branch is the preprocessed RGB image, and the output is a local feature map or feature vector that captures the detailed information of the scattered objects (such as edges, textures, etc.).

[0121] Convolutional neural networks (CNNs) specialize in extracting local texture and edge features, making them adept at identifying small or sharply defined objects. By feeding a calibrated image, the CNN's convolutional layers extract local features like edges and textures, capturing the details of the object.

[0122] ViT branch: The input of the ViT branch is the preprocessed RGB image, and the output is the global feature representation to identify large or irregular spills.

[0123] The Visual Transformer (ViT) model uses a self-attention mechanism to capture global structural features in images, making it particularly suitable for identifying large or irregularly shaped spilled objects. ViT can help identify objects with long-range connections or complex topological structures, such as long pieces of trash or large dropped objects. The three networks (YOLOv11, the CNN branch, and the ViT branch) each independently receive the same preprocessed RGB image as input and extract different types of features (spatial, local, and global).

[0124] 2. Dynamic feature fusion:

[0125] Through the dynamic feature fusion mechanism, the intermediate feature map of YOLOv11 (containing spatial and semantic information) is combined with the local feature map / vector of the CNN branch and the global feature representation of the ViT branch to form a comprehensive feature representation.

[0126] The specific formula is as follows:

[0127] ;

[0128] in, is the intermediate feature map of YOLOv11, is the local feature map of CNN, is the global feature map of ViT, α, β, γ are the weight coefficients dynamically optimized by reinforcement learning. If used for supervision tasks, the fused features are optimized by the mean square error loss function L:

[0129] ;

[0130] in, is the fused feature vector, is the true feature label corresponding to the i-th sample, and N is the number of samples.

[0131] The fused features combine the spatial and semantic information of YOLOv11, the local features (texture and edges) of CNN, and the global features (structure and context) of ViT, providing a more comprehensive representation of the spilled object for subsequent tasks, improving prediction accuracy or risk assessment. The detected location and category of the spilled object will serve as direct input for the risk assessment in step 4. This fused feature representation helps subsequent models more accurately analyze the potential threat posed by the spilled object.

[0132] 3. Semi-supervised spill classification:

[0133] A semi-supervised learning strategy is used to train a combination of labeled and unlabeled data. Consistency regularization is performed on unlabeled data, reducing reliance on manually labeled data. This allows the model to learn underlying patterns from unlabeled data and improve generalization. For example, the model can learn more stable feature representations through self-supervision on unlabeled data and adjust YOLOv11 weights based on the results of semi-supervised learning to enhance detection of complex or small scattered objects (such as debris). Among them:

[0134] Labeled data ( ) Calculate the cross entropy loss ( );

[0135] Unlabeled data ( ) Improve generalization through consistency regularization.

[0136] Step 4: Predict and evaluate the impact of spilled materials on tunnel safety:

[0137] 1. Prediction of spatiotemporal evolution of scattered materials:

[0138] Method: Based on spatiotemporal graph neural network (ST-GNN), LSTM, and Transformer models.

[0139] Input data:

[0140] RGB image: visual features of the spilled objects (e.g., type);

[0141] Thermal radiation image: temperature information of the spilled objects (such as temperature difference ΔT);

[0142] LiDAR point cloud: spatial position and geometric parameters (such as coverage area, volume, thickness) of the scattered objects.

[0143] Modeling process:

[0144] (1) Analysis of position and distribution changes (short-term evolution):

[0145] Based on the time series of LiDAR point cloud data, the potential spread of spilled materials within the tunnel is predicted. For example, accumulated water may spread to low-lying areas due to gravity, and gravel may be displaced by passing vehicles.

[0146] Use LSTM to capture the instantaneous changes in the location of the scattered objects. By comparing the point cloud data of consecutive time periods, the spatial displacement of the scattered objects in the short term is calculated:

[0147] ;

[0148] Among them, ΔA is the spatial displacement of the scattered objects in the short term, and is the coverage area at adjacent time points, is the time interval.

[0149] (2) State evolution estimation (long-term diffusion):

[0150] Combining the temperature changes of infrared images with the feature evolution of RGB images, we can predict how the state of the spilled material will change over time. For example, accumulated water may gradually decrease (temperature tends to stabilize), while oil pollution may become more difficult to identify due to diffusion (color features become lighter). The visible light image and thermal radiation image are input into the Transformer network, and the long-term state evolution trend of the spilled material is modeled through the self-attention mechanism to generate a long-term distribution heat map. The heat map reflects the changes in the coverage and temperature distribution of the spill over future time periods, such as the probability distribution of oil spreading or water accumulation decreasing.

[0151] (3) Spatial-temporal graph network enhancement:

[0152] LiDAR point cloud data is constructed as a spatiotemporal graph, where nodes represent the spatial locations of spills (based on the coordinates and geometric parameters from step 1, such as coverage area), and edges represent the spatial relationships between adjacent spills or areas (determined by Euclidean distance or adjacency, with a preset distance threshold). Node features include coverage area from 3D point cloud data, type from visible light imagery (provided by YOLOv11), temperature from thermal radiation imagery, and feature vectors derived from the fusion of YOLOv11, CNN, and ViT networks.

[0153] ST-GNN uses graph convolution operations, incorporating the time dimension, to update the state of each node, capturing the dynamic interactions and spatial dependencies between spills. The fused feature vectors enhance node representations, enabling the model to leverage more comprehensive information about spills (such as local details and global structure) for spatiotemporal distribution predictions. For example, a large spill (such as oil) can cause changes in the distribution of nearby smaller debris (such as gravel), or multiple spills can cluster around curves.

[0154] Compared to LSTM's temporal modeling and Transformer's global trend analysis, ST-GNN's unique advantage lies in its sensitivity to spatial structure, revealing the interactions between spilled objects (such as the chain reaction of diffusion). This is particularly critical in complex tunnel environments where multiple objects coexist. For example, when oil spills spread, ST-GNN can predict their indirect impact on the distribution of nearby water or gravel, thereby improving the comprehensiveness and accuracy of the prediction.

[0155] The short-term spatial displacement (ΔA) of LSTM is used as the temporal input of ST-GNN to drive the temporal evolution of node features. The enhanced spatiotemporal feature representation of ST-GNN output is obtained. , including the predicted feature vector of each spill at the time step (including the updated position, coverage area and distribution probability).

[0156] Dynamic interaction relationships: Spatial impact weights between spills (such as updated values of the adjacency matrix) are used to generate heat maps for subsequent risk assessment.

[0157] Transformer long-term distribution heatmap Provide global constraints for ST-GNN, specifically by embedding heat map features into the graph convolution weights of ST-GNN to limit the spatial range of node updates and ensure that local predictions are consistent with the global trend.

[0158] Output integration:

[0159] Enhanced features of ST-GNN Heatmap with Transformer Forming complete spatiotemporal distribution prediction results through weighted fusion :

[0160] ;

[0161] in, , Weights optimized for reinforcement learning.

[0162] 2. Traffic safety risk assessment:

[0163] Evaluation framework: Based on multimodal data fusion and dynamic risk quantification model.

[0164] Evaluation Process:

[0165] (1) Traffic safety impact assessment (current risk):

[0166] Based on the current coverage area (from LiDAR), position weight (from RGB and LiDAR), and short-term dynamic trend (from LSTM), the current traffic risk index R is calculated:

[0167] ;

[0168] Where: A is the current coverage area (unit: square meters), calculated from LiDAR point cloud data, reflecting the spatial occupancy of the spilled object; P is the current position weight (value range [0, 1]), determined by the RGB image and LiDAR data. For example, the center of the lane has a weight of 1, and the edge has a weight of 0. is the short-term distribution change rate (unit: square meters / second), calculated by LSTM based on the LiDAR point cloud time series, reflecting the immediate diffusion or movement trend of the scattered objects; is the weight coefficient of the coverage area, which indicates the contribution of the coverage area to the current traffic risk. Its value range is [0.5, 2.0]. It is determined through regression analysis of historical traffic data and is used to quantify the direct impact of the spilled area on driving safety. is the weight coefficient of the location weight, which indicates the contribution of the location to the current traffic risk. The value range is [1.0, 3.0]. It is calibrated according to the tunnel traffic volume and regional importance, highlighting the risks of key locations (such as the center of the lane); It is the adjustment coefficient of the spatial displacement of the scattered objects in the short term, indicating the gain effect of the dynamic change of the scattered objects on the current risk. The value range is [0.1, 1.0]. It is optimized through experimental verification to ensure the reasonable weight of the dynamic trend.

[0169] Combined with infrared data, for high-temperature spills (such as burning objects), an additional thermal risk factor is added to adjust the risk index:

[0170] ;

[0171] in, is the adjusted risk index, k is the thermal sensitivity coefficient, is the temperature difference.

[0172] (2) Thermal risk analysis (current risk):

[0173] Based on the current coverage area A, temperature difference and short-term temperature changes (from ), calculate the current thermal risk index H:

[0174] ;

[0175] in, is the short-term temperature change value predicted by ST-GNN, is the adjustment coefficient for short-term temperature changes.

[0176] (3) Secondary risk inference (current risk):

[0177] Using Bayesian networks, based on historical multimodal data (the historical database stored in step 6, such as the correlation between the type of spilled objects and the accident), the current type, location and The input is a Bayesian network with congestion and collision as output nodes, which predicts the probability of spilled objects triggering a chain reaction, such as traffic jam or secondary collision.

[0178] (4) Future risk prediction:

[0179] Based on the spatiotemporal distribution prediction results F final , assess the changes in risk that may be caused by spillage in the future. Specifically, using F final The predicted coverage area change value A pred and position change value P pred , calculate the future traffic risk index:

[0180] ;

[0181] Among them: A pred is the coverage area of the predicted time step (unit: square meters), which is given by F final The ST-GNN in

[15] enhances feature derivation to reflect the possible diffusion or reduction of the scattered objects; pred is the position weight of the predicted time step, ranging from [0,1], and is given by F final For example, the weight of the center of the lane is 1 and the edge is 0.

[0182] Future heat risks: Combined with F final The predicted temperature change , calculate the future heat risk index:

[0183] ;

[0184] Future secondary risks: the type of spilled material, F final The predicted position change values are input into the Bayesian network to infer the probability of future congestion and collision.

[0185] Risk grading standards:

[0186] Low risk: Coverage area < 0.1m², temperature < 50°C, location in marginal area;

[0187] Medium risk: Coverage area 0.1-2m², or temperature 50-100℃, located near the driveway;

[0188] High risk: Covered area > 2m² and located in the center of the lane, or temperature > 100°C.

[0189] Step 5: Adaptive Optimization and Collaborative Learning:

[0190] Execution entities: Federated learning servers and edge computing nodes.

[0191] Function: Dynamically optimize performance and realize multi-node knowledge sharing.

[0192] 1. Dynamic adjustment of reinforcement learning (automatically adjusting parameters based on environmental feedback to balance detection accuracy and efficiency).

[0193] During the spill monitoring process, reinforcement learning algorithms are used to optimize the efficiency and performance of multiple components. By interacting with the environment (such as light, temperature, and humidity), this method can automatically adjust parameters to balance detection accuracy and efficiency. Specific parameters include:

[0194] (1) Sensor layer: camera exposure time. LiDAR scanning frequency ( ,in, is the attenuation coefficient, f base is the base scanning frequency, which means the default frequency without smoke interference. It is usually set to the maximum supported frequency of the LiDAR hardware, such as 10 Hz or 20 Hz, depending on the specific device. 烟雾 The smoke concentration is measured in real time by an infrared sensor or environmental monitoring module. The unit is ppm (parts per million) or percentage, and the value range is usually [0,1000 ppm].

[0195] (2) Computing layer: edge node inference frequency (balancing computing latency and detection accuracy Data upload threshold (transmit only high-risk spill data to reduce bandwidth usage).

[0196] Edge node inference frequency objective function:

[0197] ;

[0198] in, is the trade-off coefficient (known quantity).

[0199] Reinforcement learning optimizes parameter adjustment strategies by continuously interacting with the environment, thereby improving the speed and accuracy of spilled object detection. Through dynamic adjustments enabled by reinforcement learning, this method can adapt to environmental changes (such as lighting, weather, temperature, and humidity) in real time, maintaining high accuracy while improving efficiency.

[0200] 2. Federated model updates (improving model generalization capabilities while protecting data privacy, and optimizing the detection and prediction models in steps 3 and 4).

[0201] Federated Learning integrates model information from each node through distributed training, ensuring that data does not leave the local device. Each edge node (such as a sensor, device, etc.) trains the model using local data (such as an image of a spilled object in a tunnel section) to generate gradients. , and then uploads model gradients instead of raw data, ensuring data privacy. By aggregating model updates from all edge nodes, the central server outputs a global model that generalizes better than the local models of individual nodes, thereby improving the adaptability and accuracy of the spilled object detection process in different environments.

[0202] Specific contents of the global model:

[0203] Each node uploads the model gradient ( ), the server aggregates the global model.

[0204] This method effectively avoids the risk of uploading sensitive data to the cloud, while improving the intelligence and performance of the overall process through joint training.

[0205] Step 6: Visual reporting and decision support:

[0206] Execution body: database and visualization engine.

[0207] Function: Convert test results into actionable maintenance strategies.

[0208] 1. Multidimensional data storage (structured storage of historical data to support long-term trend analysis).

[0209] Storage fields: spilled object location (3D coordinates), geometric parameters, and risk level.

[0210] All detected spillage information (such as location, geometric parameters, temperature data, point cloud data, risk level, etc.) will be stored in the database in real time for subsequent query and analysis.

[0211] The database utilizes an efficient indexing structure, enabling rapid retrieval and multi-dimensional analysis, including filtering and querying based on time, location, and risk level. Historical data can be used for trend analysis, helping to assess the long-term stability of tunnels.

[0212] Through data storage and analysis, this method can provide location-based spillage predictions, facilitating regular inspection and maintenance of specific areas.

[0213] 2. Intelligent report generation (providing intuitive decision-making basis):

[0214] This method provides detailed information on the status of spillage and risk assessments through visualization reports such as heat maps and 3D models. Heat maps highlight high-risk areas, helping maintenance personnel quickly locate critical areas. 3D models visualize the distribution of spillage, helping engineers analyze its impact on the structure from multiple perspectives.

[0215] This invention, based on the organic combination of multimodal sensor data fusion, deep learning technology, and spatiotemporal evolution prediction, successfully addresses the challenges of existing technologies, such as insufficient detection accuracy and lack of dynamic risk prediction capabilities in complex environments. It not only achieves high-precision detection and classification of spilled objects, but also fills the gap in spatiotemporal evolution analysis through advanced predictive models. This technology offers significant technical advantages and broad application prospects, providing a solution for modern tunnel safety management.

[0216] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A tunnel pavement spillage warning method based on multimodal perception, characterized in that: include: Acquiring multimodal data of a target tunnel pavement, wherein the multimodal data includes visible light images, thermal radiation images, and three-dimensional point cloud data; Performing feature extraction on the visible light image to obtain feature information of the scattered objects includes: The visible light image is input into the YOLOv11 network, the CNN network and the ViT network respectively, and the location and category of the spilled object, the local feature map and the global feature map are output respectively; Fusing the intermediate feature map of the YOLOv11 network, the local feature map of the CNN network, and the global feature map of the ViT network to obtain feature information of the scattered object; Input the three-dimensional point cloud data into the LSTM network to obtain the spatial displacement of the scattered object in the short term; Constructing an ST-GNN spatiotemporal graph neural network based on the feature information of the scattered object and the multimodal data, inputting the spatial displacement of the scattered object in a short period of time into the ST-GNN spatiotemporal graph neural network to obtain an enhanced spatiotemporal feature vector; Constructing the ST-GNN spatiotemporal graph neural network includes: constructing the ST-GNN spatiotemporal graph neural network by representing the spatial position of the spilled objects with nodes and the spatial relationship between adjacent spilled objects or regions with edges, wherein the node feature representation includes coverage area, type of spilled objects, temperature, and characteristic information of the spilled objects; Inputting the visible light image and the thermal radiation image into a Transformer network to obtain long-term distribution heat map features; Combining the enhanced spatiotemporal feature vector and the long-term distribution heat map feature to obtain a spatiotemporal distribution prediction result of the scattered objects; Obtaining a tunnel traffic risk warning result based on the multimodal data and the spatiotemporal distribution prediction result of the spilled objects, including: calculating a current traffic risk index and a heat risk index based on the multimodal data, and calculating a future traffic risk index and a future heat risk index based on the spatiotemporal distribution prediction result of the spilled objects; The current traffic risk index is: ； ； in, is the adjusted current traffic risk index, k is the thermal sensitivity coefficient, is the temperature difference, is the adjustment coefficient of the spatial displacement of the scattered objects in the short term, A is the current coverage area, P is the current position weight, R is the current traffic risk index, is the weight coefficient of coverage area, is the weight coefficient of position weight; The heat risk index is: ； in, is the short-term temperature change value predicted by ST-GNN, is the adjustment coefficient of short-term temperature change value, H is the heat risk index; The future traffic risk index is: ； The future heat wave risk index is: ； in, is the future traffic risk index, is the future heat wave risk index, 、 、 The coverage area, position, and temperature change values for the predicted future time steps.

2. The tunnel pavement spillage warning method based on multimodal perception according to claim 1 is characterized in that: Acquiring multimodal data of the target tunnel pavement includes: Acquire an initial visible light image using a high-resolution camera, perform non-local mean denoising and Retinex illumination compensation on the initial visible light image, and obtain the visible light image; Collecting the thermal radiation image by an infrared thermal imaging sensor; Initial three-dimensional point cloud data is collected by laser radar, and the initial three-dimensional point cloud data is aligned with the visible light image to obtain the three-dimensional point cloud data.

3. The tunnel pavement spillage warning method based on multimodal perception according to claim 1 is characterized in that: The method for fusing the intermediate feature map of the YOLOv11 network, the local feature map of the CNN network, and the global feature map of the ViT network is as follows: ； Among them, Fusion is a comprehensive feature representation, α, β, γ are weight coefficients, 、 、 They are the intermediate feature map of YOLOv11, the local feature map of CNN, and the global feature map of ViT.

4. The tunnel pavement spillage warning method based on multimodal perception according to claim 1 is characterized in that: Calculation of short-term spatial displacement of scattered objects includes: ； Among them, ΔA is the spatial displacement of the scattered objects in the short term, and is the coverage area at adjacent time points, is the time interval.

5. The tunnel pavement spillage warning method based on multimodal perception according to claim 1 is characterized in that: Obtaining tunnel traffic risk warning results also includes: inputting the type of spilled objects, current location, and short-term spatial displacement of the spilled objects into the Bayesian network, and outputting the probability of congestion and collision. The Bayesian network is obtained through historical spilled object types, locations, short-term spatial displacement of the spilled objects, and corresponding congestion and collision probabilities training.

Images