Tunnel multimodal information processing and prediction method based on Transformer technology

Abstract

The present invention provides a tunnel multimodal information processing and prediction method based on Transformer technology, the method comprising: step S1: obtaining multiple sensor data of tunnels and vehicles; step S2: constructing vehicle composite features according to the sensor data; step S3: synthesizing the composite features into corresponding one-dimensional data sequences and performing one-dimensional convolution processing, inputting the processing results into corresponding multiple multilayer perceptrons respectively to obtain composite feature weights, obtaining weighted features according to the composite feature weights, adding position information to the convolution processing results of each one-dimensional data sequence and performing cross-modal fusion, performing projection transformation on the cross-modal fusion results to obtain corresponding single-modal features; step S4: using weighted features to weight single-modal features, and inputting the weighted results into the constructed Transformer model to obtain tunnel risk prediction results. The present invention has the advantages of high prediction accuracy and high efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic information processing, and in particular to a tunnel multimodal information processing and prediction method based on Transformer technology. Background Art

[0002] In urban tunnels, information sources are becoming increasingly abundant, mainly including video surveillance data, radar surveillance data, environmental monitoring data and vehicle-mounted sensor data. Video surveillance data can be used to obtain vehicle status and driving behavior characteristics, such as whether a car accident, skidding, or breakdown has occurred, as well as whether there have been any violations such as crossing the solid line or changing lanes across the solid line. Environmental monitoring data provides key information such as temperature, humidity, road slipperiness, and CO2 concentration in the tunnel. Radar surveillance data and vehicle-mounted sensor data can obtain more detailed vehicle information, such as vehicle model, speed, distance between vehicles, battery temperature, and battery life.

[0003] However, due to the large traffic volume in urban tunnels, high frequency of data collection, and a lot of interference information, it is difficult to ensure the accuracy of the prediction when risk prediction is performed through single-source information processing and analysis. A single data source often cannot fully reflect the complex traffic conditions in the tunnel, and it is easy to miss important information, resulting in inaccurate prediction results. Summary of the invention

[0004] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a tunnel multimodal information processing and prediction method based on Transformer technology with high prediction accuracy is provided, which can effectively improve the safety and traffic efficiency of vehicles in tunnels.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A tunnel multimodal information processing and prediction method based on Transformer technology includes the following steps:

[0007] Step S1: Acquire various sensor data of the tunnel and the vehicle;

[0008] Step S2: constructing composite features of vehicles according to the sensor data, wherein the composite features include features of each vehicle and features between vehicles;

[0009] Step S3: synthesizing the composite features into corresponding one-dimensional data sequences, performing one-dimensional convolution processing on each one-dimensional data sequence, and inputting the processing results into corresponding multiple multi-layer perceptrons to obtain the composite feature weights, and obtaining weighted features according to the composite feature weights; adding position information to the convolution processing results of each one-dimensional data sequence, performing cross-modal fusion, and performing projection transformation on the cross-modal fusion results to obtain corresponding unimodal features;

[0010] Step S4: using the weighted features to weight the unimodal features, and inputting the weighted results into the constructed Transformer model to obtain tunnel risk prediction results;

[0011] The composite features include vehicle state features, driving behavior features, vehicle driving stability features, vehicle abnormal risk features and road condition risk features. The vehicle state features, driving behavior features, vehicle driving stability features, vehicle abnormal risk features and road condition risk features are synthesized into corresponding one-dimensional data sequences at the second level, thereby obtaining a vehicle state feature sequence, a driving behavior feature sequence, a vehicle driving stability feature sequence, a vehicle abnormal risk feature sequence and a road condition risk feature sequence. Each feature sequence is input into a Husformer model for one-dimensional convolution processing, and the processing results are input into corresponding multiple multi-layer perceptrons to obtain different outputs of the multiple multi-layer perceptrons, and the different outputs of the multiple multi-layer perceptrons are used as composite feature weights.

[0012] As a further improvement of the method of the present invention: Step S2 is to obtain a feature vector of vehicle information by preprocessing the acquired sensor data and then extracting features; establishing feature engineering of each vehicle based on the feature vector to describe the status of each vehicle; and obtaining a composite feature of the vehicle after combining and transforming the feature engineering.

[0013] As a further improvement of the method of the present invention: the method for establishing the feature engineering of each vehicle includes:

[0014] Use image analysis algorithms to extract vehicle status features and driving behavior features from video surveillance data;

[0015] Using a support vector machine algorithm to extract vehicle driving stability features through one or more of vehicle speed and vehicle distance data;

[0016] Using a logistic regression algorithm to extract abnormal risk features of the vehicle based on one or more of the battery temperature and battery life data;

[0017] A decision tree algorithm is used to extract road condition risk characteristics through one or more of the vehicle type, temperature, humidity, and road slipperiness data.

[0018] As a further improvement of the method of the present invention: the step S3 also includes adjusting the weighted features using preset weights, and using the adjusted weighted features to weight the unimodal features.

[0019] As a further improvement of the method of the present invention: the function expression for weighting the unimodal feature using the adjusted weighted feature is as follows:

[0020] V VSFS= Z VSFS ×M VSFS ×W A(VSFS) ,

[0021] V DBFS= Z DBFS ×M DBFS ×W A(DBFS) ,

[0022] V VDSFS= Z VDSFS ×M VDSFS ×W A(VDSFS) ,

[0023] V VARFS= Z VARFS ×M VARFS ×W A(VARFS) ,

[0024] V RCRFS= Z RCRFS ×M RCRFS ×W A(RCRFS) ,

[0025] in, V VSFS is the weighted feature of vehicle status, V DBFS is the weighted feature of driving behavior, V VDSFS is the weighted feature of vehicle driving stability, V VARFS is the vehicle abnormal risk weighted feature, V RCRFS is the road condition risk weighted feature, M VSFS is the vehicle state feature weight, M DBFS is the driving behavior feature weight, M VDSFS is the characteristic weight of vehicle driving stability, M VARFS is the vehicle abnormal risk feature weight,M RCRFS is the road risk feature weight, W A(VSFS) is the vehicle state weight, W A(DBFS) is the driving behavior weight, W A(VDSFS) is the vehicle driving stability weight, W A(VARFS) is the vehicle abnormal risk weight, W A(RCRFS) is the road condition risk weight, Z VSFS is the vehicle state feature sequence, Z DBFS is the driving behavior feature sequence, Z VDSFS is the vehicle driving stability characteristic sequence, Z VARFS is the vehicle abnormal risk feature sequence, Z RCRFS is the road condition risk feature sequence.

[0026] As a further improvement of the method of the present invention: in step S4, the Transformer model adopts a multi-head attention mechanism to achieve attention to different parts of the data; and the Transformer model is trained and cross-validated through historical tunnel accident data and normal driving data to obtain a trained Transformer model.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention obtains multiple sensor data information in the tunnel, constructs composite features of the vehicle itself and between vehicles, realizes multi-source data fusion, effectively avoids the one-sidedness and limitations of a single data source, and lays a solid data foundation for accurate prediction. The constructed composite features can not only depict the details of each vehicle in detail, but also show the interaction and connection between vehicles, making the description of the traffic scene more three-dimensional and comprehensive; and then after synthesizing the composite features into a one-dimensional data sequence, it is input into the Husformer model for one-dimensional convolution processing to extract local features and time series information, and capture the changing trends and laws of traffic data in the time dimension; and through a multi-layer perceptron, nonlinear transformation is performed to deeply explore the complex relationship between features, and different feature weights are reasonably assigned, which further improves the efficiency and accuracy of feature processing, and ensures that the features input into the Transformer model have stronger representativeness and effectiveness. With its powerful self-attention mechanism, the Transformer model can keenly capture the long-distance dependency between features in a complex and changeable tunnel traffic environment, thereby accurately predicting tunnel risks, which helps to improve the safety and traffic efficiency of tunnel traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of a tunnel multimodal information processing and prediction method based on Transformer technology in an embodiment of the present invention.

[0030] Figure 2 It is a schematic diagram of a tunnel multimodal information processing and prediction method based on Transformer technology in a specific application embodiment of the present invention.

[0031] Figure 3 The present invention is a flowchart of a tunnel multimodal information processing and prediction method based on Transformer technology in a specific application embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] like Figure 1 As shown, the tunnel multimodal information processing and prediction method based on the Transformer technology in this embodiment includes the following steps:

[0034] Step S1: Acquire various sensor data of the tunnel and the vehicle.

[0035] In this embodiment, the data collected from the built-in sensors in the tunnel covers the following aspects:

[0036] Video Monitoring (VM) data collects video image information in the tunnel in real time through cameras and other devices, and is used to analyze vehicle driving trajectories, traffic flow, vehicle type identification, and driver behavior.

[0037] Radar Monitoring (RM) uses radar technology to obtain information such as vehicle position, speed, and distance, which is used to analyze and monitor the distance between vehicles and the distance between vehicles and tunnel structures.

[0038] Environmental monitoring data include temperature (T), humidity (H), slipperiness of road (SR), and CO2 concentration. Among them, the measurement of temperature in the tunnel helps to evaluate vehicle performance, road conditions, and the impact on pedestrians; humidity measurement can reflect the humidity level in the tunnel, and the slipperiness of the road directly affects the friction of the road and the braking performance of the vehicle, which is crucial to the safety of vehicle driving.

[0039] The on-board sensor data include: vehicle model (VM), vehicle speed (VS), vehicle distance (VD), radar perception (RP), battery temperature (BT), battery health (BH) and on-board sensor data information. These data play a key role in vehicle performance evaluation, safety assurance and vehicle behavior analysis. They are an important source of information to ensure the safe and efficient operation of vehicles in tunnels. At the same time, they also provide rich data support for traffic management and intelligent transportation systems, which helps to improve the safety and smoothness of tunnel traffic.

[0040] Step S2: constructing composite features of the vehicle based on the sensor data, where the composite features include features of each vehicle and features between vehicles.

[0041] In this embodiment, the acquired sensor data is preprocessed and then feature extracted to obtain a feature vector of the vehicle information; feature engineering of each vehicle is established based on the feature vector to describe the status of each vehicle; and each feature engineering is combined and transformed to obtain a composite feature of the vehicle.

[0042] Specifically, since the raw sensor data contains multiple sources, in the data preprocessing stage, data cleaning is first performed to remove noise, outliers and erroneous data in the raw data. For example, if the radar sensor has an erroneous reading for some reason, it may cause abnormal values ​​of vehicle speed or distance, and these abnormal data points need to be corrected or deleted. Then the data is normalized or standardized. The data range and magnitude of different sensors may be different. In order to make data from different sources comparable in subsequent processing, the data needs to be normalized to a unified range (usually between 0 and 1) or standardized (so that the data conforms to the standard normal distribution). For example, the speed data is normalized from 0 to 200 kilometers per hour to the range of 0 to 1, and the ambient temperature range from -20 to 40 degrees Celsius is also normalized accordingly. Then, information or features useful for the final prediction result are extracted from the preprocessed data. For example: extract the vehicle's outline, license plate number, vehicle location and motion trajectory from video surveillance data as vehicle status features. The vehicle's driving stability characteristics are extracted from the continuous speed and acceleration data of the on-board sensors. For example, by analyzing the rate of change of speed and the fluctuation of acceleration, it is determined whether the vehicle is driving stably or frequently accelerating and decelerating.

[0043] In this embodiment, the method for establishing feature engineering for each vehicle includes:

[0044] Use the image analysis algorithm Vision Transformer (VIT) to extract vehicle status features and driving behavior features from video surveillance data;

[0045] Use the Support Vector Machine (SVM) algorithm to extract vehicle driving stability characteristics through one or more of the vehicle speed and vehicle distance data;

[0046] Use the Logistic Regression (LR) algorithm to extract vehicle abnormal risk features based on one or more of the battery temperature and battery life data;

[0047] A decision tree (DT) algorithm is used to extract road condition risk features through one or more of the vehicle type, temperature, humidity, and road slipperiness data.

[0048] In this embodiment, by extracting features from a variety of different types of data, the information obtained covers multiple dimensions such as vehicle status, driving behavior, vehicle performance, and external road conditions, avoiding the limitations of a single data source and providing a sufficient information basis for a comprehensive and accurate description of vehicle and traffic conditions. At the same time, adaptive algorithms are used for different data types and feature extraction requirements, and the accuracy and effectiveness of feature extraction are improved through the reasonable use of the above algorithms.

[0049] Step S3: synthesize the composite features into corresponding one-dimensional data sequences, perform one-dimensional convolution processing on each one-dimensional data sequence, and input the processing results into corresponding multiple multi-layer perceptrons to obtain composite feature weights, and obtain weighted features according to the composite feature weights; add position information to the convolution processing results of each one-dimensional data sequence and perform cross-modal fusion, and perform projection transformation on the cross-modal fusion results to obtain corresponding unimodal features.

[0050] like Figure 2 As shown, in a specific application embodiment, the vehicle state characteristics, driving behavior characteristics, vehicle driving stability characteristics, vehicle abnormal risk characteristics, and road condition risk characteristics are respectively synthesized into corresponding one-dimensional data sequences at the second level, thereby obtaining a vehicle state feature sequence (Vehicle State Feature Sequence, VSFS) X VSFS , Driving Behavior Feature Sequence (DBFS) X DBFS , Vehicle Driving Stability Feature Sequence (VDSFS) X VDSFS , Vehicle Abnormal Risk Feature Sequence (VARFS) X VARFS , Road Condition Risk Feature Sequence (RCRFS) X RCRFS , each feature sequence is input into the Husformer model for one-dimensional convolution processing Conv1, and the processing results are input into multiple multi-layer perceptrons respectively to obtain different outputs of multiple multi-layer perceptrons, and the different outputs of multiple multi-layer perceptrons are used as composite feature weights.

[0051] This embodiment can continuously record and integrate various types of feature information in the time dimension by synthesizing different types of features into corresponding one-dimensional data sequences at the second level, and capture the dynamic changes of vehicles and traffic conditions over time. For example, through the vehicle state feature sequence, the state change trend of the vehicle at different times can be observed, providing data support for analyzing the dynamic characteristics of traffic flow. The feature sequence processed by one-dimensional convolution is input into multiple multi-layer perceptrons to obtain different outputs as composite feature weights. Among them, the multi-layer perceptron has a powerful nonlinear mapping ability, and can perform complex transformations and processing on the input features, so as to learn the importance of different features in describing traffic conditions and vehicle status, and provide a more reasonable weight distribution for subsequent weighting and model training, which helps to improve the accuracy of the model's prediction of traffic conditions.

[0052] In this embodiment, the method of combining and transforming vehicle-related feature engineering to obtain composite features includes but is not limited to the following methods:

[0053] 1. Simple mathematical operation combination

[0054] Additive combination: two or more features can be added together to obtain a new composite feature. For example, the engine power and torque of a vehicle can be added together to obtain a feature that comprehensively reflects the vehicle's dynamic performance. This method is suitable for situations where the features are additivity and the result after addition can represent some meaningful physical or performance concept.

[0055] Multiplication combination: In some cases, the product of features may have a specific meaning. For example, multiplying the vehicle's speed by the driving time gives a composite feature called mileage. Such a composite feature may be useful when analyzing issues such as the relationship between a vehicle's energy consumption and driving distance.

[0056] Dividing two features can also create valuable composite features. For example, by dividing the fuel consumption of a vehicle by the mileage, you can get the important composite feature of fuel consumption, which is used to evaluate the fuel economy of the vehicle.

[0057] 2. Combination based on statistical methods

[0058] Mean combination: For features with multiple dimensions or multiple similar properties, their means can be calculated to form a composite feature. For example, a vehicle has pressure sensor data for multiple tires. Calculating the mean of these tire pressures can obtain a composite feature representing the overall tire pressure status of the vehicle, which helps to quickly determine the general status of the vehicle's tires.

[0059] Variance combination: The variance of the calculated feature can reflect the stability or discreteness of the feature. Taking the acceleration data of a vehicle at different times as an example, the variance of the calculated acceleration can be used as a composite feature to measure the stability of the vehicle acceleration process.

[0060] 3. Encoding-based combination

[0061] One-hot encoding combination: If there are some categorical features in the vehicle data, such as vehicle color, model, etc., you can use one-hot encoding to convert these categorical features into vector form, and then combine them with other numerical features. Assuming that the vehicle color has three values: red, blue, and black, after one-hot encoding, you can get a three-dimensional vector, which is then spliced ​​together with other numerical features of the vehicle to form a composite feature vector containing more information, which helps the model better process categorical information.

[0062] Feature hashing combination: For some high-dimensional or sparse features, feature hashing can be used to map them to a low-dimensional space and then combine them with other features. This method can reduce the dimension of the feature while retaining some important information of the feature, improving the training efficiency and generalization ability of the model.

[0063] 4. Combination based on machine learning algorithms

[0064] Principal Component Analysis (PCA): PCA can perform linear transformation on multiple features of a vehicle and convert them into a set of new unrelated principal component features. These principal component features can retain most of the variance information of the original features and achieve dimensionality reduction and combination of features. For example, for multiple sensor data of a vehicle, using PCA can obtain several comprehensive principal component features, which can represent the main change direction of the original data, help simplify the data structure and extract key information.

[0065] Autoencoder: Using the unsupervised learning algorithm of autoencoder, low-dimensional representation of vehicle features can be learned. The autoencoder consists of an encoder and a decoder. The encoder maps the original features to a low-dimensional latent space to obtain a set of new composite feature representations, while the decoder attempts to reconstruct the original features from these composite features. In this way, the potential structure and pattern in the data can be discovered and more representative composite features can be generated.

[0066] This embodiment also includes adjusting the weighted features using preset weights, and using the adjusted weighted features to weight the unimodal features.

[0067] The function expression for weighting the unimodal features using the adjusted weighted features is as follows:

[0068] V VSFS=Z VSFS ×M VSFS ×W A(VSFS)

[0069] V DBFS= Z DBFS ×M DBFS ×W A(DBFS)

[0070] V VDSFS= Z VDSFS ×M VDSFS ×W A(VDSFS)

[0071] V VARFS= Z VARFS ×M VARFS ×W A(VARFS)

[0072] V RCRFS= Z RCRFS ×M RCRFS ×W A(RCRFS)

[0073] in, V VSFS is the weighted feature of vehicle status, V DBFS is the weighted feature of driving behavior, V VDSFS is the weighted feature of vehicle driving stability, V VARFS is the vehicle abnormal risk weighted feature, V RCRFS is the road condition risk weighted feature, M VSFS is the vehicle state feature weight, M DBFS is the driving behavior feature weight, M VDSFS is the vehicle driving stability characteristic weight, M VARFS is the vehicle abnormal risk feature weight, M RCRFS is the road risk feature weight,W A(VSFS) is the vehicle state weight, W A(DBFS) is the driving behavior weight, W A(VDSFS) is the vehicle driving stability weight, W A(VARFS) is the vehicle abnormal risk weight, W A(RCRFS) is the road condition risk weight, Z VSFS is the vehicle state feature sequence, Z DBFS is the driving behavior feature sequence, Z VDSFS is the vehicle driving stability characteristic sequence, Z VARFS is the vehicle abnormal risk feature sequence, Z RCRFS is the road condition risk feature sequence.

[0074] This embodiment adjusts the weights of composite features by using preset weights, allowing the system to flexibly change the importance of each composite feature according to different scenarios, environments or specific task requirements. This makes the method of the present invention highly adjustable and can better adapt to different traffic conditions and prediction needs. For example, under different weather conditions, the importance of road risk features may be different. By adjusting the corresponding preset weights, the model can focus more on this feature, improving the adaptability and prediction accuracy of the model in different environments. At the same time, multiple important composite features (vehicle status, driving behavior, vehicle driving stability, vehicle abnormality risk, road condition risk) are weighted, taking into account multiple key aspects of traffic condition prediction, avoiding focusing on a single feature and ignoring other important information. This comprehensive consideration realizes a multi-dimensional evaluation of traffic status and improves the accuracy and reliability of the overall prediction of the system. In addition, different composite features reflect the status of the traffic system from different perspectives. By weighting and combining them organically, they can more comprehensively reflect the operating status of the vehicle and the complexity of the traffic environment, such as V VSFS Comprehensive information reflecting the vehicle status, V DBFS Comprehensive information reflecting driving behavior, etc., which together provide richer and more valuable data for prediction.

[0075] Step S4: Input the weighted feature sequence into the constructed Transformer model to obtain the tunnel risk prediction result.

[0076] In this embodiment, the Transformer model uses a multi-head attention mechanism to focus on different parts of the data; and the Transformer model is trained and cross-validated through historical tunnel accident data and normal driving data to obtain a trained Transformer model.

[0077] This embodiment uses a Transformer model with a multi-head attention mechanism to focus on different parts of the data from multiple different perspectives. In tunnel traffic condition prediction, this means that it can simultaneously focus on different aspects of the vehicle (such as vehicle status, driving behavior, road condition risk, etc.) and the relationship between different vehicles. For example, the multi-head attention mechanism can simultaneously consider information such as the vehicle's speed, acceleration, position, and the relative position and speed difference between vehicles without being limited to local information, thereby more comprehensively understanding the structure and information of the data.

[0078] For multimodal data, the multi-head attention mechanism can focus on and process data of different modalities separately, effectively fusing different types of information. For example, for features extracted from video data, sensor data, and environmental data, the multi-head attention mechanism can process these data in parallel and deeply explore the relationship between them, thereby achieving better information fusion and feature representation.

[0079] In a specific application embodiment, by adding the weighted features V VSFS , V DBFS , V VDSFS , V VARFS , V RCRFS Input the Self-Attention Transformer to enhance the weights of the three features that have a strong correlation with traffic accidents: vehicle status features, driving behavior features, and road condition risk features. This improves the model’s accuracy in predicting vehicle accidents and the efficiency of real-time response to tunnel traffic conditions. Finally, the Self-Attention Transformer output y is the tunnel risk prediction result.

[0080] The present invention is further described below by taking the above method for tunnel multimodal information processing and prediction based on Transformer technology in a specific application embodiment as an example. Figure 3 As shown, the detailed steps of this embodiment are:

[0081] Step 1: Collect tunnel built-in sensor data, including video surveillance data, radar monitoring data, environmental monitoring data and vehicle-mounted sensor data;

[0082] Step 2: Data preprocessing and feature extraction, establish feature engineering for each vehicle, and construct composite features based on raw sensor data;

[0083] Step 3: Build a Transformer-based model with a multi-head attention mechanism that allows the model to focus on different parts of the data.

[0084] Step 4: Train and cross-validate the model using historical tunnel accident data and normal driving data;

[0085] Step 5: Train the model.

[0086] Step 6: Use the trained model to perform real-time data analysis to predict the risk of a possible accident involving the vehicle;

[0087] Step 7: Implement a full-process tracking and monitoring mode for predicted high-risk vehicles in the tunnel management system.

[0088] In this embodiment, in order to verify the practical application effect of the present invention, a series of typical tunnel traffic scenarios were selected for testing. As shown in Table 1, these scenarios not only cover different environmental conditions (such as temperature, humidity, etc.), but also include a variety of vehicle behavior modes (such as normal driving, abnormal parking, solid line lane change, etc.), and accurate risk assessment results are obtained through comprehensive analysis of multiple factors. For example, vehicles 2 to 5 all experienced abnormal parking, while vehicles 6 and 7 performed solid line lane changes. These behaviors may increase the risk of accidents in actual driving. Among them, the speed of vehicle 2 is 58.03km / h, and the distance between vehicles is only 10.32 meters. In a dry tunnel environment, its accident risk probability is as high as 45%. This shows that even in the case of close to normal driving conditions, when certain key parameters are at critical values, the system can still accurately identify and warn of potential high-risk situations.

[0089] In addition, vehicle 3 has a speed of 50.08 km / h, a headway of 8.13 meters, a high humidity in the tunnel (72.85%), and a slippery road surface, and its accident risk probability is 53%. This further proves that the present invention can comprehensively consider environmental factors and vehicle status to provide more accurate risk assessment results. Vehicles 4 and 5 also show similar high risk probabilities due to their shorter headway, which are 58% and 56% respectively.

[0090] On the other hand, although vehicles 6 and 7 changed lanes on a solid line, their accident risk probabilities were 48% and 45% respectively due to their relatively high speeds and moderate distances between vehicles, demonstrating the system's ability to flexibly respond to different driving behaviors. In particular, despite changing lanes on a solid line, vehicle 7 had a speed of 69.13 km / h, a distance of 10.25 meters, low tunnel humidity (61.31%), and dry road surface, so its accident risk probability was relatively low.

[0091] In summary, the present invention can not only conduct a meticulous analysis of the status of a single vehicle, but also comprehensively consider environmental factors to provide more accurate risk assessment results. Through effective multimodal information fusion, different types of sensor data are integrated to form a comprehensive feature engineering result. This effective fusion of multimodal information enables the model to comprehensively evaluate the risk status of each vehicle from multiple dimensions, thereby improving the accuracy of the prediction. More importantly, the present invention can immediately issue an early warning when a vehicle enters a high-risk state, and implement a full-process tracking and monitoring mode for predicted high-risk vehicles through the tunnel management system to ensure that all necessary preventive measures are implemented. This method not only improves the safety of individual vehicles, but also optimizes traffic flow management in the entire tunnel, preventing potential accidents in advance, thereby greatly improving the overall traffic safety level of the tunnel.

[0092] Table 1 Data of different application scenarios

[0093]

[0094] This embodiment also provides a tunnel multimodal information processing and prediction device based on Transformer technology, including a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to execute the tunnel multimodal information processing and prediction method based on Transformer technology.

[0095] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices in cooperation with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other applications. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0096] This embodiment also provides a computer-readable storage medium storing a tunnel multimodal information processing and prediction program based on Transformer technology. When the computer program is executed, a tunnel multimodal information processing and prediction method based on Transformer technology is implemented.

[0097] Those skilled in the art should understand that the above-mentioned embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The above are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

[0098] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A tunnel multimodal information processing and prediction method based on Transformer technology, characterized in that: The following steps are involved: Step S1: Acquire various sensor data of the tunnel and the vehicle; Step S2: constructing composite features of vehicles according to the sensor data, wherein the composite features include features of each vehicle and features between vehicles; Step S3: synthesizing the composite features into corresponding one-dimensional data sequences, performing one-dimensional convolution processing on each one-dimensional data sequence, and inputting the processing results into corresponding multiple multi-layer perceptrons to obtain composite feature weights, and obtaining weighted features according to the composite feature weights; adding position information to the convolution processing results of each one-dimensional data sequence, performing cross-modal fusion, and performing projection transformation on the cross-modal fusion results to obtain corresponding unimodal features; Step S4: using the weighted features to weight the unimodal features, and inputting the weighted results into the constructed Transformer model to obtain tunnel risk prediction results; The composite features include vehicle state features, driving behavior features, vehicle driving stability features, vehicle abnormal risk features and road condition risk features. The vehicle state features, driving behavior features, vehicle driving stability features, vehicle abnormal risk features and road condition risk features are synthesized into corresponding one-dimensional data sequences at the second level, thereby obtaining a vehicle state feature sequence, a driving behavior feature sequence, a vehicle driving stability feature sequence, a vehicle abnormal risk feature sequence and a road condition risk feature sequence. Each feature sequence is input into a Husformer model for one-dimensional convolution processing, and the processing results are input into corresponding multiple multi-layer perceptrons to obtain different outputs of the multiple multi-layer perceptrons, and the different outputs of the multiple multi-layer perceptrons are used as composite feature weights.

2. According to claim 1, the tunnel multimodal information processing and prediction method based on Transformer technology is characterized in that: The step S2 is to extract features after preprocessing the acquired sensor data to obtain feature vectors of vehicle information; establish feature engineering of each vehicle based on the feature vector to describe the status of each vehicle; and obtain composite features of the vehicle after combining and transforming the feature engineering.

3. The tunnel multimodal information processing and prediction method based on Transformer technology according to claim 2 is characterized in that: The method for establishing feature engineering for each vehicle includes: Use image analysis algorithms to extract vehicle status features and driving behavior features from video surveillance data; Using a support vector machine algorithm to extract vehicle driving stability features through one or more of vehicle speed and vehicle distance data; Using a logistic regression algorithm to extract abnormal risk features of the vehicle based on one or more of the battery temperature and battery life data; A decision tree algorithm is used to extract road condition risk characteristics through one or more of the vehicle type, temperature, humidity, and road slipperiness data.

4. The tunnel multimodal information processing and prediction method based on Transformer technology according to claim 1 is characterized in that: The step S3 also includes adjusting the weighted features using preset weights, and using the adjusted weighted features to weight the unimodal features.

5. The tunnel multimodal information processing and prediction method based on Transformer technology according to claim 4 is characterized in that: The function expression for weighting the unimodal feature using the adjusted weighted feature is as follows: V VSFS= Z VSFS ×M VSFS ×W A(VSFS) V DBFS= Z DBFS ×M DBFS ×W A(DBFS) V VDSFS= Z VDSFS ×M VDSFS ×W A(VDSFS) V VARFS= Z VARFS ×M VARFS ×W A(VARFS) V RCRFS= Z RCRFS ×M RCRFS ×W A(RCRFS) in, V VSFS is the weighted feature of vehicle status, V DBFS is the weighted feature of driving behavior, V VDSFS is the weighted feature of vehicle driving stability, V VARFS is the vehicle abnormal risk weighted feature, V RCRFS is the road condition risk weighted feature, M VSFS is the vehicle state feature weight, M DBFS is the driving behavior feature weight, M VDSFS is the vehicle driving stability characteristic weight, M VARFS is the vehicle abnormal risk feature weight, M RCRFS is the road risk feature weight, W A(VSFS) is the vehicle state weight, W A(DBFS) is the driving behavior weight, W A(VDSFS) is the vehicle driving stability weight, W A(VARFS) is the vehicle abnormal risk weight, W A(RCRFS) is the road condition risk weight, Z VSFS is the vehicle state feature sequence, Z DBFS is the driving behavior feature sequence, Z VDSFS is the vehicle driving stability characteristic sequence, Z VARFS is the vehicle abnormal risk feature sequence, Z RCRFS is the road condition risk feature sequence.

6. The tunnel multimodal information processing and prediction method based on Transformer technology according to claim 1 is characterized in that: In step S4, the Transformer model adopts a multi-head attention mechanism to achieve attention to different parts of the data; and the Transformer model is trained and cross-validated through historical tunnel accident data and normal driving data to obtain a trained Transformer model.

Images