Intelligent Decision-Making System and Method for Highway Emergencies Based on Multimodal Fusion

Abstract

This invention relates to the field of traffic management technology, specifically disclosing an intelligent decision-making system for highway emergencies based on multimodal fusion. The system includes a data acquisition module, a data processing module, a multimodal data fusion analysis module, and an intelligent decision-making module. The system collects three types of parameters—event entity, traffic flow impact, and environment—through cameras and sensors along the route. After data preprocessing, quantitative modeling and the vector space method are used to calculate the cosine similarity between the event entity state vector, traffic flow impact state vector, and environmental parameter state vector and their corresponding reference vectors. Then, intelligent decision-making indicators are constructed and input into a pre-trained decision neural network model, outputting an optimized decision-making policy composed of multiple emergency actions. This achieves full-process automation from perception and analysis to intelligent response, significantly improving the efficiency and safety of handling highway emergencies.

Description

Technical Field

[0001] This invention relates to the field of traffic management technology, specifically to an intelligent decision-making system and method for highway emergencies based on multimodal fusion. Background Technology

[0002] Traditional highway incident handling relies heavily on manual patrols, telephone alerts, and manual analysis at monitoring centers. This approach suffers from inherent flaws such as delayed information acquisition, strong subjectivity, and limited analytical dimensions. Managers struggle to integrate heterogeneous data from multiple sources, including video, sensors, and weather stations, within a short timeframe. This hinders rapid and comprehensive assessments of the incident's type, severity, traffic impact, and environmental risks, leading to data-driven decision-making. Consequently, mismatched response measures, inefficient resource allocation, and untimely coordinated actions are common problems. This not only prolongs incident response time and exacerbates traffic congestion but also risks secondary accidents, posing a serious threat to road efficiency and public safety.

[0003] Therefore, this invention proposes an intelligent decision-making system and method for highway emergencies based on multimodal fusion, which can realize automatic fusion, intelligent analysis and rapid decision-making of multi-source information, so as to improve the intelligence level and handling efficiency of highway emergency management. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent decision-making system and method for highway emergencies based on multimodal fusion, thereby solving the above-mentioned technical problems: The objective of this invention can be achieved through the following technical solutions: A highway emergency intelligent decision-making system based on multimodal fusion is characterized in that the system includes a data acquisition module, a data processing module, a multimodal data fusion analysis module, and an intelligent decision-making module; The data acquisition module is used to collect various data on the highway, including cameras and various sensors installed along the highway, to collect highway event-related parameters. The data processing module is used to preprocess the various collected data to improve the quality and usability of the data. The multimodal data fusion and analysis module is used to fuse data from different modalities and extract different data features; The multimodal data fusion analysis module calculates the event ontology state vector using vector space modeling and cosine similarity calculation methods. Reference vector of event ontology cosine similarity Traffic flow influence state vector Traffic flow influence state reference vector cosine similarity and environmental parameter state vector With environmental parameter state reference vector cosine similarity ; The intelligent decision-making module constructs intelligent decision-making indicators based on the aforementioned three cosine similarities. The index is then input into a pre-trained decision neural network model, which outputs an optimized decision policy consisting of multiple emergency actions.

[0005] As a further description of the technical solution of the present invention, the highway event-related parameters include: event ontology parameters, traffic flow impact parameters, and environmental parameters; The event ontology parameters include event type parameters, event location parameters, and event severity parameters; The traffic flow impact parameters include congestion delay parameters and impact range parameters; The environmental parameters include weather parameters.

[0006] As a further description of the technical solution of the present invention, the working process of the data processing module includes: for video image data, performing noise reduction processing to remove noise interference in the image; image enhancement to enhance the visual effect of the image; For sensor data, data cleaning is performed to remove outliers and erroneous data; data calibration is performed to calibrate the measurement errors of the sensors and improve the accuracy of the data; data normalization is performed to unify the data from different sensors to the same scale range, which facilitates subsequent fusion processing.

[0007] As a further description of the technical solution of the present invention, the working process of the multimodal data fusion analysis module includes: The types of emergencies are quantified and coded as follows: 1-Traffic accident, 2-Vehicle breakdown, 3-Fire, 4-Hazardous materials leak, 5-Severe weather, 6-Road facility damage; The system retrieves the event type parameter, i.e., the current emergency event type is 'i'. Based on the emergency event type 'i', the system assigns event type index values ​​to different event types. Where i is the event type code; Obtain event location parameters, namely the number of lanes currently occupied by the emergency and the specific lane, and construct a mathematical model for event location indicators. In the formula, n is the number of lanes occupied, and Q is the lane occupancy characteristic coefficient; Obtain the event severity parameter, i.e. whether there are casualties, and assign values ​​to the event severity assessment index based on the casualty situation.

[0008] As a further description of the technical solution of the present invention, if the current emergency event type is vehicle malfunction... =0.2; If the current emergency type is severe weather or damage to road facilities, =0.5; If the current emergency type is a traffic accident, =0.8; if the current event is a fire or a hazardous materials leak, =1; If the current emergency occupies the slow lane, Q=0.5; if the current emergency occupies both the fast lane and the emergency lane, Q=1. If there are no casualties in the current emergency, the severity assessment index S=0; if there are no casualties in the current emergency, the severity assessment index S=1. Define the event entity state vector as V. Obtain the system-defined event ontology reference vector. Calculate the event ontology state vector V and the event ontology reference vector. cosine similarity .

[0009] As a further description of the technical solution of the present invention, the working process of the multimodal data fusion analysis module also includes: Obtain the length L of the road segment currently affected by the accident, which is the distance from the point of the incident upstream along the direction of travel to the location of the end of the queue of vehicles; Get the average speed of vehicles within the affected road section of the current accident. Meanwhile, the average free-flow speed was obtained within the affected road section. Construct a calculation model for current accident congestion and delay indicators. Where N is the number of vehicles affected; Define the traffic flow influence state vector as follows: , Obtain the traffic flow impact state reference vector set by the system. The traffic flow influence state vector is calculated as follows: and traffic flow influence state reference vector cosine similarity .

[0010] As a further description of the technical solution of the present invention, the working process of the multimodal data fusion analysis module also includes: The working process of the multimodal data fusion and analysis module also includes: Construct an environmental parameter vector from real-time weather and visibility parameters. ,in, , , and These are visibility, precipitation intensity, road surface adhesion coefficient, and crosswind intensity, respectively. Obtain weight coefficients , , and , , , and These are the weighting coefficients for visibility, precipitation intensity, road surface adhesion coefficient, and crosswind intensity, respectively. Therefore, the environmental parameter state vector is: Obtain the system-defined environmental parameter status reference vector. Calculate the cosine similarity between the environmental parameter state vector and the environmental parameter state reference vector. .

[0011] As a further description of the technical solution of the present invention, the working process of the intelligent decision-making module includes: Constructing a mathematical model for intelligent decision-making indicators ,in, and These are the weighting coefficients; Obtain the set of executable actions P defined by the system. ; Obtain the retrained decision neural network model from the system, with the intelligent decision-making index for the current event as the input. The output label is the effective decision-making plan (Policy) adopted at that time, where Policy is a combination of multiple actions.

[0012] A multimodal fusion-based intelligent decision-making method for highway emergencies, comprising the following steps: Step S1: Collect data through cameras and various sensors along the highway to collect various parameters related to emergencies in real time; Step S2: Process the parameters. Video image processing: noise reduction and image enhancement; sensor data processing: data cleaning, calibration, normalization, and standardization of data scale. Step S3: Perform multimodal data fusion analysis on the emergency, including: event ontology analysis, traffic flow impact analysis, and environmental parameter analysis; Step S4: Construct intelligent decision indicators, input the indicators into the pre-trained decision neural network model, and output the decision policy, which is a series of action combinations.

[0013] The beneficial effects of this invention are: This invention, by constructing an intelligent decision-making system based on multimodal data fusion, achieves precise perception, quantitative assessment, and scientific decision-making for highway emergencies, bringing significant benefits. The system integrates multi-source data from cameras, sensors, and other sources, and employs quantitative modeling and vector space analysis methods to uniformly transform the event ontology, traffic flow impact, and environmental parameters into a calculable cosine similarity index, thereby enabling a comprehensive, objective, and real-time assessment of the event situation. Based on this, the system innovatively constructs an intelligent decision-making index model that integrates multi-dimensional features and introduces a pre-trained neural network for decision mapping. This model can automatically generate an optimal response plan that highly matches the overall event situation and is a coordinated combination of multiple emergency actions. This closed-loop process not only greatly improves the speed and efficiency of emergency response and reduces the risk of secondary accidents and traffic paralysis caused by improper handling, but also significantly reduces management costs, enhances the intelligence level and proactive safety assurance capabilities of highway management, and provides strong technical support for building an efficient and safe modern traffic management system. Attached Figure Description

[0014] The invention will now be further described with reference to the accompanying drawings.

[0015] Figure 1 This is a partial structural diagram of the intelligent decision-making system for highway emergencies based on multimodal fusion, as described in this invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 As shown, the present invention provides an intelligent decision-making system for highway emergencies based on multimodal fusion, characterized in that the system includes a data acquisition module, a data processing module, a multimodal data fusion analysis module, and an intelligent decision-making module; The data acquisition module is used to collect various data on the highway, including cameras and various sensors installed along the highway, to collect highway event-related parameters. The data processing module is used to preprocess the various collected data to improve the quality and usability of the data. The multimodal data fusion and analysis module is used to fuse data from different modalities and extract different data features; The multimodal data fusion analysis module calculates the event ontology state vector using vector space modeling and cosine similarity calculation methods. Reference vector of event ontology cosine similarity Traffic flow influence state vector Traffic flow influence state reference vector cosine similarity and environmental parameter state vector With environmental parameter state reference vector cosine similarity ; The intelligent decision-making module constructs intelligent decision-making indicators based on the aforementioned three cosine similarities. The index is then input into a pre-trained decision neural network model, which outputs an optimized decision policy consisting of multiple emergency actions.

[0018] Through the above technical solution, this invention provides an intelligent decision-making system and method for highway emergencies based on multimodal fusion. Its core working principle lies in achieving rapid and accurate response to highway emergencies through a closed-loop process integrating perception, fusion analysis, and intelligent decision-making. The system first collects comprehensive data from multi-source sensing devices (such as cameras, weather sensors, and traffic flow detectors) deployed along the highway, acquiring multimodal information including event-specific parameters (such as event type, location, and severity), traffic flow impact parameters (such as congestion delays and impact range), and environmental parameters (such as visibility, precipitation intensity, road surface conditions, and crosswind intensity). This raw data then enters the data processing module for standardized preprocessing: video images undergo denoising and enhancement to improve quality, while sensor data undergoes cleaning, calibration, and normalization to eliminate outliers and dimensional differences, providing a high-quality, standardized data foundation for subsequent analysis.

[0019] In the crucial multimodal data fusion and analysis phase, the system employs quantitative modeling and vector space analysis methods to deeply integrate and extract features from three core parameter categories. For the event ontology, the system quantifies and encodes event types (such as traffic accidents, vehicle malfunctions, and fires) and assigns them index values. This is combined with location indicators calculated based on the number of occupied lanes and lane characteristic coefficients, as well as severity indicators based on casualties, to construct an event ontology state vector. The system then quantifies the emergency state of the event ontology by calculating its cosine similarity to a preset reference vector. For traffic flow impact, the system constructs a congestion delay model by real-time monitoring parameters such as the length of affected road segments, the difference between average vehicle speed and free-flow speed, forming a traffic flow impact state vector and calculating its deviation from the standard state. For environmental parameters, the system constructs an environmental parameter vector from visibility, precipitation intensity, road surface adhesion coefficient, and crosswind intensity, and introduces a weighted influence matrix for weighted fusion. Finally, the system assesses the hazard of environmental conditions by calculating the cosine similarity between the environmental state vector and the reference vector.

[0020] Finally, in the intelligent decision-making stage, the system integrates the analysis results of the above three dimensions (i.e. , and The three cosine similarities are integrated through a comprehensive decision index mathematical model, which uses environmental similarity ΔC as a moderating factor to adjust the event ontology similarity. Similarity to traffic flow impact The system uses a weighted sum and modulation to generate an intelligent decision-making index that comprehensively reflects the overall situation of an event. This index is input into a pre-trained decision neural network model, which maps out the optimal emergency response plan. The output plan is a dynamically combined set of executable actions, which may include issuing variable information alerts, implementing tiered speed limits, closing the nearest entrance lane, dispatching patrol cars, activating remote diversion plans, requesting coordination with fire and medical departments, and even activating a full-line closure plan. The entire system, through an automated process from data acquisition to preprocessing to multimodal fusion analysis to intelligent decision output, achieves real-time perception, accurate assessment, and scientific decision-making for highway emergencies, significantly improving the speed and efficiency of emergency response and providing strong intelligent technical support for ensuring the safety and smooth flow of highways.

[0021] The highway event-related parameters include: event ontology parameters, traffic flow impact parameters, and environmental parameters; The event ontology parameters include event type parameters, event location parameters, and event severity parameters; The traffic flow impact parameters include congestion delay parameters and impact range parameters; The environmental parameters include weather parameters.

[0022] The data processing module works by: performing noise reduction on video image data to remove noise interference from the image; and enhancing the image visual effect. For sensor data, data cleaning is performed to remove outliers and erroneous data; data calibration is performed to calibrate the measurement errors of the sensors and improve the accuracy of the data; data normalization is performed to unify the data from different sensors to the same scale range, which facilitates subsequent fusion processing.

[0023] The working process of the multimodal data fusion and analysis module includes: The types of emergencies are quantified and coded as follows: 1-Traffic accident, 2-Vehicle breakdown, 3-Fire, 4-Hazardous materials leak, 5-Severe weather, 6-Road facility damage; The system retrieves the event type parameter, i.e., the current emergency event type is 'i'. Based on the emergency event type 'i', the system assigns event type index values ​​to different event types. Where i is the event type code; Obtain event location parameters, namely the number of lanes currently occupied by the emergency and the specific lane, and construct a mathematical model for event location indicators. In the formula, n is the number of lanes occupied, and Q is the lane occupancy characteristic coefficient; Obtain the event severity parameter, i.e. whether there are casualties, and assign values ​​to the event severity assessment index based on the casualty situation.

[0024] When acquiring the severity parameter of an event, the system needs to determine whether the current emergency has caused casualties. The methods for acquiring casualty information include, but are not limited to, one or more combinations of the following: Image recognition and analysis: By collecting images or videos of the accident scene through cameras deployed along the highway, deep learning models (such as object detection and human posture recognition algorithms) are used to automatically detect features such as whether there are fallen people, the posture of the injured, or the arrival of rescuers, and output a preliminary judgment on whether there are casualties. Multi-source data fusion: The system connects to external platforms such as traffic police, road administration, and 120 emergency centers, and obtains accident alarm records, on-site handling feedback, ambulance dispatch information, etc. in real time through API or data interface. After data cleaning and structuring, personnel injury and death tags are extracted. Vehicle system reporting: Supports communication with intelligent connected vehicles (such as C-V2X) to receive emergency calls (eCall) automatically triggered by accident vehicles or personnel injury estimation information reported by vehicle sensors (such as airbag deployment and seat occupancy detection); Manual input assistance: In cases where the system does not obtain automatic recognition results or there is uncertainty, personnel in the monitoring center are allowed to manually confirm or correct the status of personnel injuries or fatalities through a graphical interface, serving as a backup input source for system decision-making.

[0025] The system selects the best from the above multi-source information and finally outputs a binary event severity assessment index S (S=0 for no casualties, S=1 for casualties), which is used for the subsequent construction of the event ontology state vector and cosine similarity calculation.

[0026] If the current emergency type is vehicle malfunction =0.2; If the current emergency type is severe weather or damage to road facilities, =0.5; If the current emergency type is a traffic accident, =0.8; if the current event is a fire or a hazardous materials leak, =1; If the current emergency occupies the slow lane, Q=0.5; if the current emergency occupies both the fast lane and the emergency lane, Q=1. If there are no casualties in the current emergency, the severity assessment index S=0; if there are no casualties in the current emergency, the severity assessment index S=1. Define the event entity state vector as V. Obtain the system-defined event ontology reference vector. Calculate the event ontology state vector V and the event ontology reference vector. cosine similarity .

[0027] Through the above technical solution, this embodiment constitutes the core link of the multimodal data fusion analysis module in the system to quantitatively evaluate the event ontology. Its core lies in transforming the essential attributes of the sudden event into measurable and standardized decision-making basis through a set of refined quantitative index system and vector space similarity calculation.

[0028] Specifically, the system first deconstructs the key dimension of the event ontology, breaking it down into three quantifiable core parameters: event type, event location, and event severity. It then details the quantification methods for these parameters: the system digitally encodes abstract emergency event types (such as traffic accidents, vehicle breakdowns, fires, hazardous material leaks, severe weather, and road infrastructure damage) and assigns a weighted index value to each type. (For example, the least dangerous vehicle malfunctions) =0.2, the most dangerous fire and hazardous material leak. =1), thus transforming qualitative type judgment into quantitative numerical input. For event location, the system not only focuses on the number of lanes occupied, n, but also introduces a lane characteristic coefficient Q to distinguish the importance of different lane functions (e.g., Q=0.5 for occupying the slow lane, Q=1 for occupying the fast lane and emergency lane), and uses a mathematical model The system calculates location indicators, which reflect the actual physical encroachment of the event on road capacity. For the severity of the event, the system uses a crucial binary judgment—whether there are casualties—to directly assign a severity index S (S=0 for no casualties, S=1 for casualties), which is directly related to the priority of life-saving efforts in the emergency response.

[0029] After quantifying the three basic parameters mentioned above, a mathematical model was further defined to integrate these dispersed indicators and perform advanced analysis. The system incorporates event type indicators. The location index W and the severity index S are combined to form a comprehensive event ontology state vector. This vector V constitutes a specific coordinate point of the current event in the multidimensional feature space. To assess the urgency or severity of the event, the system compares it with a pre-defined event ontology reference vector representing a baseline or worst-case scenario. A comparison is made. This comparison is not a simple numerical difference, but rather uses a cosine similarity algorithm to calculate the similarity between vector V and V. The cosine value ΔV of the angle between the vectors. Cosine similarity effectively measures the similarity of two vectors in direction. It is not sensitive to the absolute magnitude of the vectors, but focuses more on the consistency of the relative proportions of various dimensions, thus providing a more robust assessment of the direction and degree of deviation of the current event state from the standard state. The final calculated value is... , is a scalar value between -1 and 1 (usually taken as an absolute value or processed for positive correlation comparison), which quantifies the overall urgency of the current event ontology relative to the standard state.

[0030] In summary, this embodiment, through a coherent technical path from parameter quantization to vector construction to similarity calculation, achieves in-depth mining and fusion of event ontology information, providing a solid data foundation for the intelligent decision-making of the entire system.

[0031] The working process of the multimodal data fusion and analysis module also includes: Obtain the length L of the road segment currently affected by the accident, which is the distance from the point of the incident upstream along the direction of travel to the location of the end of the queue of vehicles; Get the average speed of vehicles within the affected road section of the current accident. Meanwhile, the average free-flow speed was obtained within the affected road section. Construct a calculation model for current accident congestion and delay indicators. Where N is the number of vehicles affected; Define the traffic flow influence state vector as follows: , Obtain the traffic flow impact state reference vector set by the system. The traffic flow influence state vector is calculated as follows: and traffic flow influence state reference vector cosine similarity .

[0032] Through the above technical solution, this embodiment constitutes the core link of the multimodal data fusion analysis module in the system to accurately quantify the impact of traffic flow. Its core idea is to transform the impact of sudden events on the dynamic traffic capacity of highways into calculable indicators through mathematical models, and to use vector space similarity for standardized evaluation.

[0033] Specifically, this technical solution focuses on the traffic congestion effect caused by sudden events and quantifies it from both spatial and temporal dimensions. First, it acquires a key spatial parameter—the length L of the road segment currently affected by the accident. This length is defined as the distance from the point of occurrence of the event, upstream, to the end of the queue where vehicles have dissipated. This directly reflects the spatial extent of traffic disorder. Second, it uses a sophisticated mathematical model to quantify the time-dimensional delay loss, namely, constructing a congestion delay index calculation model. The model compares the current average speed of vehicles within the affected road segment. Compared to the ideal free-flow vehicle speed The time lost by a single vehicle passing through the section of road is calculated, and then multiplied by the total number of affected vehicles N, thus obtaining the total time delay caused by the entire event. This profoundly reveals the actual damage the event caused to traffic efficiency.

[0034] After obtaining these two basic indicators, the system integrates them into a comprehensive traffic flow impact state vector. This vector defines the current traffic flow state based on two orthogonal features: the scope of influence and the degree of delay. To determine the severity of this state, the system compares it with a pre-defined traffic flow influence state reference vector representing a certain critical or typical congestion state. The comparison is not a simple numerical comparison, but rather uses a cosine similarity algorithm to calculate the vector similarity. and cosine value of the angle between Cosine similarity can effectively measure the similarity of two vectors in direction, that is, assess the degree of fit between the current traffic flow impact pattern (the ratio of L to D) and the standard pattern, thus obtaining a relative and robust severity score. Ultimately, this The value serves as a key input for measuring the degree of traffic paralysis. It is then fed into the subsequent intelligent decision-making module and integrated with the evaluation results of other dimensions such as event ontology and environmental parameters to jointly drive the generation of the final emergency decision-making plan.

[0035] The working process of the multimodal data fusion and analysis module also includes: Construct an environmental parameter vector from real-time weather and visibility parameters. ,in, , , and These are visibility, precipitation intensity, road surface adhesion coefficient, and crosswind intensity, respectively. Among them, the road surface adhesion coefficient The coefficient of friction is a key parameter characterizing the friction between the tire and the road surface, directly affecting the vehicle's braking performance and handling stability under sudden events. The specific methods by which the system obtains the coefficient of friction include, but are not limited to, one or more combinations of the following: Direct detection by road surface sensors: Road surface condition sensors (such as friction coefficient detectors, water film thickness sensors, and icing sensors) are buried or installed in key sections of highways (such as long downhill slopes, curves, and accident-prone areas) to collect raw data on the road surface adhesion coefficient in real time. After calibration, this data is directly used as the data. enter; Meteorological model calculation: The system accesses data from meteorological stations along the route (temperature, humidity, precipitation type and intensity, road surface temperature), and dynamically estimates the current road surface adhesion coefficient by combining empirical models or physical formulas (such as road surface slip coefficient model, freezing point prediction model). For example, the coefficient is 0.7–0.9 for dry roads, 0.4–0.6 for wet roads, and 0.1–0.3 for snow-covered or icy roads. Vehicle data inversion: By communicating with intelligent connected vehicles or roadside radar, information such as wheel slip ratio, ABS (anti-lock braking system) trigger status, and ESC (electronic stability control) intervention status of passing vehicles is obtained. The equivalent adhesion coefficient of the current road surface is inverted using the vehicle dynamics model, and the accuracy is improved by multi-vehicle data fusion. Historical data and image analysis: Based on road surface images captured by cameras, the road surface condition (dry, wet, waterlogged, snow-covered, icy) is identified through semantic segmentation or texture analysis models. Combined with historical meteorological and traffic data from the same period, an empirical estimate of the road surface adhesion coefficient is output. System default and manual correction: When the above methods are unavailable or data is missing, the system adopts the empirical default value based on the season and weather warning level (such as 0.8 for dry roads on sunny days), and allows monitoring center personnel to make manual corrections based on on-site videos or inspection reports.

[0036] The system selects the best value from the above multi-source information and finally outputs a standardized road adhesion coefficient. The values ​​typically range from 0 to 1, and together with other environmental parameters, they form an environmental parameter vector. .

[0037] Obtain weight coefficients , , and , , , and These are the weighting coefficients for visibility, precipitation intensity, road surface adhesion coefficient, and crosswind intensity, respectively. Therefore, the environmental parameter state vector is: Obtain the system-defined environmental parameter status reference vector. Calculate the cosine similarity between the environmental parameter state vector and the environmental parameter state reference vector. .

[0038] Through the above technical solution, this embodiment constitutes the core link of the multimodal data fusion analysis module in the system to perform weighted fusion and quantitative evaluation of environmental parameters. Its core lies in transforming complex and ever-changing environmental conditions into a unified and quantifiable decision reference index by constructing a weighted environmental parameter vector and calculating its similarity to the standard state.

[0039] Specifically, this embodiment first structures the key environmental factors affecting highway safety and traffic efficiency. It constructs a four-dimensional environmental parameter vector from real-time weather and visibility parameters. ,in, , , and Visibility, precipitation intensity, road adhesion coefficient, and crosswind intensity are represented by these parameters, respectively. This step integrates environmental data with different physical meanings and dimensions into a unified mathematical framework. Subsequently, weighting coefficients are obtained. , , and , , , and These are the weighting coefficients for visibility, precipitation intensity, road surface adhesion coefficient, and crosswind intensity, respectively, corresponding to the degree of influence of the corresponding environmental parameters (from visibility to crosswind intensity) on the safe operation of the highway. The system transforms the original environmental parameter vector X into a weighted environmental parameter state vector C. The essence of this operation is to sum the environmental parameters according to their importance, thereby generating a scalar value that can comprehensively reflect the overall environmental risk level.

[0040] To assess the severity of the current comprehensive environmental risk, the system compares the weighted state vector C with a preset environmental parameter state reference vector representing a certain baseline or high-risk environmental condition. The comparison is performed. The method for comparison is also to calculate the cosine similarity between the two. This similarity value quantifies the degree of deviation between the current overall environmental state and the standard state in terms of direction, thus effectively integrating complex and multifaceted environmental conditions into a single, standardized risk assessment value. Ultimately, this environmental risk similarity... The data is fed into the intelligent decision-making module, which dynamically amplifies or reduces the urgency level determined by the event itself and traffic flow based on the severity of the environment, so that the final decision can respond more comprehensively and adaptively to the overall situation, including weather conditions.

[0041] The working process of the intelligent decision-making module includes: Constructing a mathematical model for intelligent decision-making indicators ,in, and These are the weighting coefficients; Get the set of executable actions defined by the system. , ,in, : Publish warning information on variable message signs, Implement tiered speed limits (e.g., 100-80-60 km / h). Close the nearest entrance ramp. Dispatch patrol cars, : Activate the remote traffic diversion plan Requesting coordination between fire and medical departments and : Activate the full-line closure contingency plan, etc.; Obtain the retrained decision neural network model from the system, with the intelligent decision-making index for the current event as the input. The output label represents the effective decision-making option (Policy) adopted at that time, where a Policy is a combination of multiple actions. For example, .

[0042] The aforementioned decision neural network model employs a multi-layer fully connected feedforward neural network structure to map the intelligent decision index I into the optimal emergency action combination scheme Policy.

[0043] Detailed definition of the network structure of the decision neural network model: Input layer Input dimension: 1; Input content: Intelligent decision index I∈[0,1] (scalar value after normalization); Preprocessing: The input I is standardized to have a mean of 0 and a variance of 1 to accelerate network convergence.

[0044] Hidden layer The network employs a three-layer hidden layer structure, with batch normalization and dropout regularization introduced in each layer to prevent overfitting.

[0045] Hidden layer Number of neurons Activation function Dropout ratio Batch normalization First layer 64 ReLU 0.2 yes Second floor 128 ReLU 0.3 yes Third layer 64 ReLU 0.2 yes Hidden layer dimensionality expansion explanation: Although the input is only 1-dimensional, it is expanded to 64 dimensions through the first hidden layer, enabling the network to learn the high-order non-linear feature representation of I, thereby capturing the mapping relationship between decision indicators and complex action combinations.

[0046] Output layer Output dimension: m (where m is a predefined set of executable actions) (size) Activation function: Sigmoid (each output neuron independently outputs a probability value between 0 and 1); Output meaning: The j-th output ∈[0,1] indicates that an action is performed. The probability of [the outcome]. Binarization is performed by setting a threshold to obtain the final decision scheme, Policy={ | ≥θ}.

[0047] The training data comes from the following sources: Historical event database contains historical emergency data recorded by highway operation and management units, including event parameters, traffic flow data, environmental data, and actual emergency response plans. Expert annotation: Traffic management experts are invited to conduct manual analysis of typical event scenarios and annotate the optimal decision-making solutions.

[0048] Total sample size: no less than 10,000.

[0049] Training set: 70% (7,000 records).

[0050] Validation set: 15% (1,500 records).

[0051] Test set: 15% (1,500 records).

[0052] Through the above technical solution, this embodiment constitutes the core of the final decision generation of the intelligent decision-making system. Its core lies in integrating multi-dimensional analysis results through a comprehensive mathematical model and using a pre-trained neural network model to map complex emergency situations into a series of specific and executable emergency response actions.

[0053] Specifically, the implementation example first constructs a mathematical model of intelligent decision-making indicators. The model integrates the three core outputs of the multimodal data fusion analysis module—representing the urgency of the event ontology— , representing the severity of the impact on traffic flow and representing the level of environmental risk. —To achieve organic integration. In this model, and The weighted sum (by weight coefficients) and The adjustment reflects the combined urgency of the event's inherent attributes and its traffic impact, while This is then multiplied as a global modulation factor. This design has profound physical implications: it means that even if the event itself and the traffic impact are the same, in adverse environments (…), Under abnormal values, the overall decision-making urgency index This will also be significantly amplified, thereby driving the system to take higher-level countermeasures. (Generate decision indicators) Then, this intelligent decision-making indicator As input to the neural network, the network learns from a large amount of historical emergency response data, enabling it to capture the highly nonlinear mapping relationship between complex situations and optimal response plans. The output of the neural network is a decision policy, which is not a single action but rather a set of predefined executable actions from the system. A combination of multiple actions selected from the given list. For example, the output might be... This means simultaneously issuing warnings, closing the nearest entrance ramp, and activating remote diversion plans. This multi-action combined output mechanism based on neural networks enables the system to simulate expert decision-making, generating dynamic, collaborative, and precisely matched optimal emergency response plans that align with the overall situation of the event, thus achieving a closed loop from perception and analysis to intelligent and automated decision-making.

[0054] A multimodal fusion-based intelligent decision-making method for highway emergencies, comprising the following steps: Step S1: Collect data through cameras and various sensors along the highway to collect various parameters related to emergencies in real time; Step S2: Process the parameters. Video image processing: noise reduction and image enhancement; sensor data processing: data cleaning, calibration, normalization, and standardization of data scale. Step S3: Perform multimodal data fusion analysis on the emergency, including: event ontology analysis, traffic flow impact analysis, and environmental parameter analysis; Step S4: Construct intelligent decision indicators, input the indicators into the pre-trained decision neural network model, and output the decision policy, which is a series of action combinations.

[0055] Calculation Example A traffic accident occurred on a highway, and the specific details are as follows: Incident type: Rear-end collision between two cars (classified as "traffic accident").

[0056] Location of the incident: The accident occupies the fast lane (therefore, the number of lanes occupied is 1, and the lane characteristic coefficient Q=1).

[0057] Casualties: Preliminary assessment indicates one person was injured (therefore the severity index S=1).

[0058] Traffic impact: The accident caused a traffic jam, affecting a section of road with a length of L=3 kilometers.

[0059] The average vehicle speed on the affected road section was 20 km / h.

[0060] The free-flow speed limit on this section of road is 100 km / h.

[0061] The estimated number of affected vehicles is 150.

[0062] Environmental parameters: Visibility 5 km (light fog) Rainfall intensity 2 mm / h (light rain) Road surface adhesion coefficient: 0.6 (wet and slippery road surface) Crosswind intensity 3 m / s (light breeze) The environmental weight matrix is ​​A=[0.3,0.2,0.4,0.1] (system preset, emphasizing road surface adhesion and visibility), therefore .

[0063] Therefore, The system-defined event ontology reference vector = This represents the most serious situation. =0.995; = The system sets the traffic flow reference vector. = This represents the standard for severe congestion: a 5-kilometer queue and a total delay of 40 hours. =0.999; in, =150*(3 / 20-3 / 100)=18; = , = cosine similarity =0.999; This is a system-preset reference vector representing the most ideal (safest) environmental conditions. Under ideal conditions: extremely high visibility (e.g., clear weather, visibility 20 km), no precipitation (precipitation intensity 0 mm / h), dry road surface (adhesion coefficient 1.0), and crosswind (crosswind intensity 0 m / s). All these conditions are weighted using the same weight matrix A. Therefore, Calculate cosine similarity =0.946; =0.946*(0.4*0.995+0.6*0.999)=0.9435. The calculated decision index 0.9435 is input into a pre-trained decision neural network. Based on patterns learned from historical data, the neural network determines a high-urgency decision scheme corresponding to this index. The final output is the decision scheme Policy: Policy={ } Right now : Issue variable information alerts, Implement tiered speed limits. Dispatch patrol cars (to handle accidents and injured persons). Requesting coordination between fire and medical departments.

[0064] It should be noted that the formulas in this application are all dimensionless and numerical calculations. The formulas are obtained by software simulation based on a large amount of data and are the closest to the real situation. The thresholds, threshold ranges and coefficients involved in this application are all empirical values ​​and are selected by those skilled in the art according to the actual situation.

[0065] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A highway emergency response intelligent decision-making system based on multimodal fusion, characterized in that, The system includes a data acquisition module, a data processing module, a multimodal data fusion and analysis module, and an intelligent decision-making module; The data acquisition module is used to collect various data on the highway, including cameras and various sensors installed along the highway, to collect highway event-related parameters. The data processing module is used to preprocess the various collected data to improve the quality and usability of the data. The multimodal data fusion and analysis module is used to fuse data from different modalities and extract different data features; The multimodal data fusion analysis module calculates the event ontology state vector using vector space modeling and cosine similarity calculation methods. Reference vector of event ontology cosine similarity Traffic flow influence state vector Traffic flow influence state reference vector cosine similarity and environmental parameter state vector With environmental parameter state reference vector cosine similarity ; The intelligent decision-making module constructs intelligent decision-making indicators based on the aforementioned three cosine similarities. The index is then input into a pre-trained decision neural network model, which outputs an optimized decision policy consisting of multiple emergency actions.

2. The intelligent decision-making system for highway emergencies based on multimodal fusion according to claim 1, characterized in that, The highway event-related parameters include: event ontology parameters, traffic flow impact parameters, and environmental parameters; The event ontology parameters include event type parameters, event location parameters, and event severity parameters; The traffic flow impact parameters include congestion delay parameters and impact range parameters; The environmental parameters include weather parameters.

3. The intelligent decision-making system for highway emergencies based on multimodal fusion according to claim 1, characterized in that, The data processing module works by: performing noise reduction on video image data to remove noise interference from the image; and enhancing the image visual effect. For sensor data, data cleaning is performed to remove outliers and erroneous data; data calibration is performed to calibrate the measurement errors of the sensors and improve the accuracy of the data; data normalization is performed to unify the data from different sensors to the same scale range, which facilitates subsequent fusion processing.

4. The intelligent decision-making system for highway emergencies based on multimodal fusion according to claim 2, characterized in that, The working process of the multimodal data fusion and analysis module includes: The types of emergencies are quantified and coded as follows: 1-Traffic accident, 2-Vehicle breakdown, 3-Fire, 4-Hazardous materials leak, 5-Severe weather, 6-Road facility damage; The system retrieves the event type parameter, i.e., the current emergency event type is 'i'. Based on the emergency event type 'i', the system assigns event type index values ​​to different event types. Where i is the event type code; Obtain event location parameters, namely the number of lanes currently occupied by the emergency and the specific lane, and construct a mathematical model for event location indicators. In the formula, n is the number of lanes occupied, and Q is the lane occupancy characteristic coefficient; Obtain the event severity parameter, i.e. whether there are casualties, and assign values ​​to the event severity assessment index based on the casualty situation.

5. The intelligent decision-making system for highway emergencies based on multimodal fusion according to claim 4, characterized in that, If the current emergency type is vehicle malfunction =0.2; If the current emergency type is severe weather or damage to road facilities, =0.5; If the current emergency type is a traffic accident, =0.8; if the current event is a fire or a hazardous materials leak, =1; If the current emergency occupies the slow lane, Q=0.5; if the current emergency occupies both the fast lane and the emergency lane, Q=1. If there are no casualties in the current emergency, the severity assessment index S=0; if there are no casualties in the current emergency, the severity assessment index S=1. Define the event entity state vector as V. Obtain the system-defined event ontology reference vector. Calculate the event ontology state vector V and the event ontology reference vector. cosine similarity .

6. The intelligent decision-making system for highway emergencies based on multimodal fusion according to claim 2, characterized in that, The working process of the multimodal data fusion and analysis module also includes: Obtain the length L of the road segment currently affected by the accident, which is the distance from the point of the incident upstream along the direction of travel to the location of the end of the queue of vehicles; Get the average speed of vehicles within the affected road section of the current accident. Meanwhile, the average free-flow speed was obtained within the affected road section. Construct a calculation model for current accident congestion and delay indicators. Where N is the number of vehicles affected; Define the traffic flow influence state vector as follows: , Obtain the traffic flow impact state reference vector set by the system. The traffic flow influence state vector is calculated as follows: and traffic flow influence state reference vector cosine similarity .

7. The intelligent decision-making system for highway emergencies based on multimodal fusion according to claim 2, characterized in that, The working process of the multimodal data fusion and analysis module also includes: Construct an environmental parameter vector from real-time weather and visibility parameters. ,in, , , and These are visibility, precipitation intensity, road surface adhesion coefficient, and crosswind intensity, respectively. Obtain weight coefficients , , and , , , and These are the weighting coefficients for visibility, precipitation intensity, road surface adhesion coefficient, and crosswind intensity, respectively. Therefore, the environmental parameter state vector is Obtain the system-defined environmental parameter status reference vector. Calculate the cosine similarity between the environmental parameter state vector and the environmental parameter state reference vector. .

8. The intelligent decision-making system for highway emergencies based on multimodal fusion according to claim 1, characterized in that, The working process of the intelligent decision-making module includes: Constructing a mathematical model for intelligent decision-making indicators ,in, and These are the weighting coefficients; Get the set of executable actions P defined by the system. ; Obtain the retrained decision neural network model from the system, with the intelligent decision-making index for the current event as the input. The output label is the effective decision-making plan (Policy) adopted at that time, where Policy is a combination of multiple actions.

9. A smart decision-making method for highway emergencies based on multimodal fusion, characterized in that, The method is implemented based on the intelligent decision-making system for highway emergencies based on multimodal fusion as described in any one of claims 1-8, and the method includes the following steps: Step S1: Collect data through cameras and various sensors along the highway to collect various parameters related to emergencies in real time; Step S2: Process the parameters; video image processing: noise reduction and image enhancement. Sensor data processing: data cleaning, calibration, normalization, and standardization of data scale; Step S3: Perform multimodal data fusion analysis on the emergency, including: event ontology analysis, traffic flow impact analysis, and environmental parameter analysis; Step S4: Construct intelligent decision indicators, input the indicators into the pre-trained decision neural network model, and output the decision policy, which is a series of action combinations.

Images