Traffic alarm automatic processing system and method based on multi-modal large model

Abstract

The invention discloses a traffic alarm automatic processing method and system based on a multi-modal large model. The method comprises the following steps: step 1, multi-source heterogeneous data acquisition and real-time access; 2, carrying out multi-modal space-time alignment and fusion preprocessing; 3, event recognition and semantic reasoning based on a multi-modal large model; 4, generating a disposal strategy and arranging an instruction; 5, safe and controllable instruction execution and linkage control are carried out; step 6, performing treatment effect feedback and closed-loop optimization; and step 7, full-link auditing and visual redisk are carried out. According to the invention, the event detection accuracy in a complex environment is improved, and automation of alarm generation, risk grading, strategy generation and facility linkage is realized; the adaptability and expansibility of the system in different traffic environments are improved, the real-time response requirements of typical application scenes such as expressways and tunnels are met, and a reliable basis is provided for subsequent disposal strategies.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of traffic warning based on multi-modal large models, and in particular to a traffic warning automatic processing system and method based on multi-modal large models. BACKGROUND

[0002] Traffic warning automatic processing technology is a key link in intelligent transportation systems (ITS) and smart highway construction, and its core goal is to quickly and accurately respond to traffic abnormal events (such as accidents, congestion, bad weather, road obstacles, etc.) through perception, analysis and decision-making, thereby reducing traffic delays, reducing the risk of secondary accidents, and improving road safety and traffic efficiency. Existing research and applications mainly focus on the following categories:

[0003] 1) Video-based detection and warning method: Chinese patent CN120318778A discloses a traffic accident identification method based on computer vision, which proposes to use deep learning and image processing algorithms to detect targets in highway videos, identify abnormal parking, collisions and other events, and generate alarms through threshold and scene analysis. This kind of technology has high accuracy in good lighting and normal weather conditions, but its performance significantly decreases in rain, fog, tunnel and night scenes, and has high false alarm and missed alarm rates.

[0004] 2) Radar and vehicle networking-based warning method: Chinese patent CN118311535A discloses a multi-laser radar fusion event detection system and method, which proposes to collect vehicle flow speed, vehicle spacing and position information through multiple laser radars, and combine with emergency braking signals reported by vehicle-mounted units to realize traffic event detection and warning. This scheme can maintain good detection capability in low visibility and complex weather environment, but lacks visual semantic evidence, making it difficult to perform fine classification of accidents and explainable analysis.

[0005] 3) Rule-based and expert system-based warning method: The system mentioned in the "Highway Intelligent Transportation System Intelligent Monitoring and Emergency Disposal Application Report" uses fixed thresholds (such as speed drop > 30%, flow reduction > 50%) to trigger alarms. This kind of method is simple to implement, but it relies too much on empirical parameters and is prone to false triggering in complex environments, requiring a lot of manual involvement for confirmation and processing.

[0006] As can be seen, the existing technology still has the following outstanding problems and defects:

[0007] Severe single-mode dependence: Based on a single video or a single radar data source, the accuracy significantly decreases in the face of tunnel, curve, bad weather and other scenes;

[0008] Lack of semantic understanding: Most methods can only recognize "stop / congestion" phenomena, lack the ability to analyze complex semantics, and are difficult to distinguish between "minor scratches" and "multiple car pileups" and other different severity events;

[0009] Response lag: Existing warning systems mostly generate prompts, and actual handling (such as lane closure, diversion, and dispatch of rescue) requires manual intervention, resulting in an average response time of 3-10 minutes, delaying rescue;

[0010] Difficulty in cross-domain linkage: Traffic signals, mechanical and electrical facilities (fans, broadcasts, lighting), and emergency dispatch systems have inconsistent interface standards and insufficient automation, resulting in a fragmented "warning-handling" link.

[0011] The reasons for these problems include:

[0012] a) Video detection is severely affected by occlusion, lighting, and weather in complex scenes;

[0013] b) Radar and V2X lack visual interpretability, making it difficult to determine the severity of accidents;

[0014] c) Multi-source data standards are not unified, and there are engineering difficulties in spatio-temporal alignment and fusion;

[0015] d) Rule-driven methods are difficult to adapt to dynamic and complex traffic environments, lacking self-learning and generalization capabilities.

[0016] Among them, Chinese patent CN119293551A "Traffic incident detection system based on multi-modal data" proposes a traffic incident detection method that integrates video, radar, and vehicle-to-everything (V2X) multi-source sensor data. The core idea of this scheme is to deploy multi-modal sensors along the road to obtain traffic running information in different dimensions, and use fusion algorithms for incident detection:

[0017] 1) Data acquisition layer: The system deploys video monitoring cameras, radar speed measurement devices, and vehicle-to-everything (V2X) roadside units at road monitoring points to collect multi-dimensional information such as vehicle images, speed, acceleration, and vehicle spacing.

[0018] 2) Data preprocessing: Video data is processed through image processing and deep learning models to extract the motion trajectory and posture of the target vehicle; radar and V2X data are filtered and normalized to form structured inputs.

[0019] 3) Multi-modal fusion: Time synchronization and spatial registration techniques are used to fuse video trajectories with radar speed and V2X sudden braking signals to form unified traffic flow data. This fusion process uses weighted averaging and probability models to improve the accuracy of abnormal event detection.

[0020] 4) Event identification: By setting an anomaly detection model (such as a rule-based threshold judgment + machine learning classifier), events including emergency braking, vehicle collision, abnormal parking, traffic congestion, etc. are identified, and alarm information is generated after judgment.

[0021] 5) Result output: Alarm information is pushed to the monitoring terminal by the center platform for traffic management personnel to confirm and handle.

[0022] The implementation steps of the patent scheme are clear, especially in multi-source information fusion and abnormal event identification, which provides an engineering implementation path for improving the robustness of traffic event detection and provides a useful reference for improving the robustness of traffic event detection.

[0023] However, it still has the following shortcomings:

[0024] 1) Single fusion strategy: mainly relying on weighted average or simple probability model for multi-modal fusion, lacking deep semantic level unified modeling, leading to detection performance decline in complex scenes (such as tunnel entrance, strong / weak light alternation, rain and fog weather).

[0025] 2) Lack of semantic understanding: although the scheme can detect "emergency braking" and "congestion" events, it lacks the ability to classify and distinguish complex semantic events (such as light scratches and multiple car rear-end collisions).

[0026] 3) Incomplete alarm handling link: the patent mainly focuses on event detection and alarm, and does not involve subsequent automatic handling links such as lane closure, VMS induction, emergency dispatch, etc. Linkage functions, leading to the need for manual intervention in response.

[0027] 4) Limited scalability: the system needs to be equipped with multiple types of sensors on the road side, with high construction and maintenance costs, and there are economic and engineering challenges in promoting it in large-scale highway scenarios.

[0028] Therefore, it is a technical problem that needs to be solved by those skilled in the art to provide a multi-modal traffic event detection technology that further improves the depth of fusion, semantic understanding, and automatic handling capability. SUMMARY

[0029] Therefore, the present application provides a traffic alarm automatic handling system and method based on a multi-modal large model.

[0030] To solve the above technical problems, the present application adopts the following technical solutions:

[0031] A traffic alarm automatic handling method based on a multi-modal large model, comprising the following steps:

[0032] Step 1: Multi-source heterogeneous data acquisition and real-time access

[0033] Real-time traffic operation data is collected by sensor devices, and the collected data is accessed through multiple protocols, analyzed and cleaned in real time to provide high-quality input for subsequent processing.

[0034] Step two: multi-modal spatio-temporal alignment and fusion preprocessing

[0035] Time synchronization and spatial registration are performed on heterogeneous data from different sources and different sampling rates. Meanwhile, lane-level topology is constructed based on high-precision maps or real-time mapping results, realizing deep fusion and context association of multi-source data in the spatio-temporal dimension.

[0036] Step three: event recognition and semantic reasoning based on multi-modal large model

[0037] The pre-trained multi-modal large model MLLM is used to analyze and reason the fused data in depth.

[0038] Step four: disposition strategy generation and instruction arrangement

[0039] According to the event recognition and risk assessment results, the matching disposition plan is retrieved from the knowledge graph, and the executable atomic instruction sequence is generated combined with the real-time traffic state.

[0040] According to the event recognition results and risk assessment, the disposition plan is retrieved from the knowledge graph.

[0041] Conflict detection and resource optimization allocation are performed through rule engine and reinforcement learning strategy, and rollback conditions are preset to ensure operation safety.

[0042] Step five: safe and controllable instruction execution and linkage control

[0043] Through standardized interface and roadside equipment, instruction issuing and state monitoring are performed, and device feedback and traffic state changes are monitored in real time during execution.

[0044] Step six: disposition effect feedback and closed-loop optimization

[0045] Real-time traffic operation data after disposition is collected through multi-source sensors, disposition effect is evaluated, feedback data is used for model fine-tuning and strategy optimization, loss function is continuously minimized, and closed-loop learning mechanism is formed.

[0046] Step seven: full-link audit and visualized review

[0047] Record the full-process log from data access to instruction execution Provide visual interface to support real-time monitoring and historical backtracking, and generate explainable report.

[0048] Preferably, in the first step, the sensor device includes a CCTV camera, a millimeter wave radar, a laser radar, and a V2X roadside unit, and the collected data types include RGB video stream, radar point cloud, and vehicle state message multi-modal data.

[0049] Preferably, in the first step, the high-quality input is specifically:

[0050] Let the original data collected by the s-th sensor at time t be d s (t), the data stream accessed by the system can be represented as:

[0051]

[0052] where S is the number of sensors, and after protocol analysis and data cleaning, high-quality input satisfying the integrity constraint is obtained

[0053] Preferably, in the second step, time synchronization uses a protocol combined with a hardware timestamp to unify the data of each sensor to the same time reference, and each modality timestamp is unified to the global time base t * :

[0054]

[0055] Spatial registration maps each sensor data to a unified lane coordinate system through external parameter calibration and coordinate transformation. If the position of a modality point in the local coordinate system is p s , then the mapping to the lane unified coordinate system p is:

[0056] p=T s p s ,T s ∈SE(3)

[0057] where T s is an external parameter calibration transformation matrix;

[0058] The unified spatio-temporal data set after fusion is represented as:

[0059]

[0060] Preferably, in the third step, the model extracts features of images, radars, and V2X data through a visual encoder, a point cloud processing network, and a text embedding module, respectively, and realizes feature interaction and enhancement through a cross-modal attention mechanism.

[0061] where the image, radar, and V2X features are represented as

[0062] Feature interaction is realized through a cross-modal attention mechanism:

[0063]

[0064] Where Q is the query matrix, K is the key matrix, V is the value matrix, d is the scaling factor, and z is the fused cross-modal attention output representation;

[0065] Event category prediction uses cross-entropy loss:

[0066]

[0067] Where, p c Let y be the probability of the event category. c This is a real label;

[0068] The output includes event type identification, risk level assessment (L1-L3), and semantic description generation, and combines knowledge graphs for rule constraints and interpretability enhancement.

[0069] Preferably, in step four, a reinforcement learning strategy is used to optimize the control gains:

[0070]

[0071] Where π is the strategy, r t For immediate returns, γ is the discount factor; the generated atomic instruction sequence is denoted as {a1,…,a...} n};

[0072] In step 4, the instructions include lane control, speed limit announcement, guidance information dissemination, and emergency resource dispatch.

[0073] Preferably, in step five, a two-phase commit mechanism is used to ensure the atomicity and consistency of instructions, and to support automatic rollback and manual intervention;

[0074] The two-phase commit mechanism can be formalized as follows:

[0075]

[0076] Real-time monitoring and feedback ensure consistent and secure execution.

[0077] Preferably, in step six, the evaluation of the treatment effect includes, for example, the queue dissipation time τ. q With the improvement rate of traffic efficiency η:

[0078]

[0079] Among them, Q after Q represents the traffic flow per unit time after processing. beforeη is the traffic efficiency improvement rate (in percentage form) for the traffic flow per unit time before disposal;

[0080] Loss function:

[0081]

[0082] wherein, is the event classification loss, is the strategy optimization loss, and λ is the weight coefficient of the two losses, is the overall optimization objective function.

[0083] Preferably, in step seven, the full-process log includes data snapshots, inference results, instruction content, and execution state key information.

[0084] A traffic warning automatic processing system based on a multi-modal large model, the system is composed of six parts of a data acquisition layer, a data access and governance layer, a space-time fusion layer, a multi-modal understanding and reasoning layer, an execution and arrangement layer, and a security and audit layer; wherein,

[0085] The data acquisition layer is used to acquire multi-source heterogeneous traffic data.

[0086] The data access and governance layer adopts a streaming media server, a message queue and a time series database, and is used to realize the access, cleaning, synchronization and storage of multi-source data.

[0087] The space-time fusion layer is used to synchronize the clock, calibrate the space and model the road topology of different modal data, and form a unified traffic flow data set.

[0088] The multi-modal understanding and reasoning layer is based on a multi-modal large model MLLM and a knowledge graph, and is used to identify and semantically reason the fusion data, output the event category and risk level.

[0089] The execution and arrangement layer converts the reasoning result into an executable strategy, and through the interface linkage of a VMS bulletin board, a traffic signal machine, a tunnel fan, a broadcast system and an emergency dispatching system, is used to realize automatic disposal.

[0090] The security and audit layer is used to be responsible for system permission control, log recording, rollback mechanism and explainability audit of the disposal process.

[0091] The data acquisition layer includes a road camera CCTV, a radar speedometer, a multi-laser radar, a V2X roadside unit and a meteorological sensor.

[0092] The present application has the following technical effects relative to the prior art:

[0093] 1) The present application significantly improves the detection accuracy and robustness in complex environments: through the spatio-temporal alignment and deep fusion of multi-source heterogeneous data (video, radar, V2X), the limitations of single modal sensors in rain, fog, night, tunnel and other complex scenes are overcome; the cross-modal attention mechanism of MLLM can comprehensively utilize visual texture features, radar penetration ability and V2X vehicle state information, mutually confirm and complement each other to greatly reduce the false positive rate (FPR) and the false negative rate, and improve the recall rate (Recall) and the precision rate (Precision);

[0094] 2) The present application realizes the deep semantic understanding and accurate grading of traffic events: traditional methods can only identify simple events such as "stopping" and "congestion", while the present application uses the powerful semantic reasoning ability of MLLM to understand the context semantics of complex events such as "multiple vehicles in a row rear-end" and "goods thrown and cause congestion", and accurately grade the severity of the event (L1-L3) by combining the traffic rules and historical plans in the knowledge graph, providing reliable and explainable decision basis for subsequent differentiated disposal strategies;

[0095] 3) The present application realizes the "perception-decision-execution" second-level automatic closed-loop response: it breaks through the complete link from event identification to disposal execution, automatically generates control instructions through the execution and arrangement layer, and automatically executes VMS, signal machine, broadcast and other roadside facilities, changes the traditional response mode relying on manual confirmation and operation to second-level automatic closed loop, greatly shortens the event response time (from an average of 3-10 minutes to seconds), effectively avoids the occurrence of secondary accidents, and improves road safety and traffic efficiency;

[0096] 4) The present application has excellent system scalability and scene adaptability: the system adopts a modular and interfaced design concept, the data access layer supports multiple protocols, the reasoning layer can adapt to different scenes by replacing models, and the execution layer supports standardized device control interfaces, so that the system can be flexibly deployed in edge, cloud or hybrid cloud environment, easily integrated with existing ITS system, and continuously optimized through online learning mechanism to adapt to the needs of different road sections and different traffic flow characteristics, reducing the cost and complexity of large-scale deployment;

[0097] (5) The present application ensures the safety, controllability and auditability of the disposal process: by introducing two-phase commit, conflict detection, operation white list and rollback mechanism, the risk of misoperation and unauthorized instructions is effectively eliminated; the full-link log recording and explainable report generation function make every automatic disposal behavior traceable, reviewable and auditable, meeting the stringent requirements of system safety and reliability in actual operation, and providing technical support for "unmanned" value. BRIEF DESCRIPTION OF DRAWINGS

[0098] Figure 1A system overall architecture diagram of the present application;

[0099] Figure 2 A multi-modal data diagram of the present application;

[0100] Figure 3 An event identification diagram of the present application;

[0101] Figure 4 A technical flow diagram of the present application;

[0102] Figure 5 A scatter case diagram of the present application;

[0103] Figure 6 A fire case diagram of the present application. DETAILED DESCRIPTION

[0104] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0105] As shown in Figure 1 , the present application discloses a traffic warning automatic processing system based on a multi-modal large model, which is composed of six parts of a data acquisition layer, a data access and management layer, a space-time fusion layer, a multi-modal understanding and reasoning layer, an execution and arrangement layer, and a security and audit layer.

[0106] Among them, the data acquisition layer includes road cameras (CCTV), radar speedometers, multi-laser radars, V2X roadside units, weather sensors, etc., for acquiring multi-source heterogeneous traffic data.

[0107] The data access and management layer adopts a streaming media server, a message queue and a time series database to realize the access, cleaning, synchronization and storage of multi-source data.

[0108] The space-time fusion layer performs clock synchronization, space calibration and road topology modeling on different modal data to form a unified traffic flow data set.

[0109] The multi-modal understanding and reasoning layer performs traffic event identification and semantic reasoning on the fusion data based on a multi-modal large model (MLLM) and a knowledge graph, and outputs event categories and risk levels.

[0110] The execution and arrangement layer converts the reasoning results into executable strategies, and through the interface linkage of VMS information boards, traffic signal machines, tunnel fans, broadcast systems and emergency dispatching systems, realizes automatic disposal.

[0111] Security and audit layer: responsible for system permission control, log recording, rollback mechanism and explainable audit of disposal process.

[0112] As Figure 4 shown, the application also discloses a traffic warning automatic processing method based on a multi-modal large model, comprising the following steps:

[0113] Step one: multi-source heterogeneous data acquisition and real-time access

[0114] Real-time acquisition of traffic operation data through various sensor devices (including CCTV cameras, millimeter wave radars, laser radars, and V2X roadside units) arranged along the road;

[0115] The types of collected data include multi-modal data such as RGB video streams, radar point clouds, and vehicle state messages; as Figure 2 shown;

[0116] The data is accessed through multiple protocols, and real-time analysis and cleaning are performed to ensure the reliability and integrity of the data stream, providing high-quality input for subsequent processing.

[0117] Let the original data collected by the s-th sensor at time t be d s (t), the data stream accessed by the system can be represented as:

[0118]

[0119] Where S is the number of sensors, and after protocol analysis and data cleaning, high-quality input

[0120] Step two: multi-modal spatio-temporal alignment and fusion preprocessing

[0121] Time synchronization and spatial registration are performed on heterogeneous data from different sources and different sampling rates;

[0122] Time synchronization uses protocols combined with hardware timestamps to unify the data of each sensor to the same time reference, and each modal timestamp is unified to the global time base t * :

[0123]

[0124] Spatial registration through external parameter calibration and coordinate transformation maps each sensor data to a unified lane coordinate system If a modal point has a position p s in the local coordinate system, then the mapping to the lane unified coordinate system p:

[0125] p=T s p s ,T s∈ SE(3).

[0126] where T s is the extrinsic calibration transformation matrix. The fused unified spatio-temporal dataset is represented as:

[0127]

[0128] Meanwhile, based on the high-precision map or real-time mapping results, a lane-level topology structure is constructed to realize the deep fusion and context association of multi-source data in the spatio-temporal dimension.

[0129] Step three: event recognition and semantic reasoning based on multi-modal large model

[0130] As shown in Figure 3 , a pre-trained multi-modal large model (MLLM) is used to perform deep analysis and reasoning on the fused data. The model extracts features of images, radar and V2X data through a visual encoder, a point cloud processing network and a text embedding module, respectively, and realizes feature interaction and enhancement through a cross-modal attention mechanism.

[0131] The image, radar and V2X features are represented as Feature interaction is realized through a cross-modal attention mechanism:

[0132]

[0133] where Q is the Query matrix, K is the Key matrix, V is the Value matrix, d is the scaling factor, and z is the fused cross-modal attention output representation.

[0134] The event category prediction uses cross-entropy loss:

[0135]

[0136] where p c is the event category probability, and y c is the true label.

[0137] The output includes event type identification (such as accidents, congestion, litter, etc.), risk level assessment (L1-L3) and semantic description generation, and combines knowledge graph for rule constraint and explainability enhancement.

[0138] Step four: disposition strategy generation and instruction arrangement

[0139] According to the event recognition and risk assessment results, the matching disposition plan is retrieved from the knowledge graph, and the executable atomic instruction sequence is generated combined with the real-time traffic state. According to the event recognition results and risk assessment, the disposition scheme is retrieved from the knowledge graph.

[0140] Optimize control gains using reinforcement learning strategies:

[0141]

[0142] Where π is the strategy, r t For immediate returns, γ is the discount factor. The generated atomic instruction sequence is denoted as {a1,…,a...} n}

[0143] The instructions include lane control, speed limit issuance, guidance information dissemination, and emergency resource allocation. The system uses a rule engine and reinforcement learning strategies for conflict detection and optimized resource allocation, and preset rollback conditions to ensure operational safety.

[0144] Step 5: Safe and controllable command execution and linkage control

[0145] It uses standardized interfaces to issue commands and monitor the status of roadside equipment.

[0146] A two-phase commit mechanism ensures the atomicity and consistency of instructions, supporting automatic rollback and manual intervention. The two-phase commit mechanism can be formalized as follows:

[0147]

[0148] Real-time monitoring and feedback ensure consistency and safety in execution. During execution, real-time monitoring of equipment feedback and changes in traffic conditions ensures the effective implementation of response measures.

[0149] Step Six: Feedback on Treatment Results and Closed-Loop Optimization

[0150] Real-time traffic operation data after the response is collected using multi-source sensors to assess the effectiveness of the response, such as the queue dissipation time τ. q With the improvement rate of traffic efficiency η:

[0151]

[0152] Among them, Q after Q represents the traffic flow per unit time after processing. before The traffic flow per unit time before processing is η, which is the traffic efficiency improvement rate (in percentage form).

[0153] Feedback data is used for model fine-tuning, adjusting the loss function.

[0154]

[0155] Continuously minimize, where, Classify the losses for the event. The loss is used for strategy optimization, where λ is the weighting coefficient of the two losses. The overall optimization objective function is defined.

[0156] Feedback data is used for model fine-tuning and strategy optimization, forming a closed-loop learning mechanism to continuously improve the system's adaptability and decision-making accuracy in complex environments.

[0157] Step Seven: Full-Link Audit and Visualized Review

[0158] This step records the full-process log from data access to instruction execution including data snapshots, inference results, instruction content, execution status, and other key information. The system provides a visual interface for real-time monitoring and historical review, and generates explainability reports for manual review, responsibility tracing, and continuous improvement.

[0159] Example 1: Debris Case

[0160] As Figure 5 shown;

[0161] Scenario Overview: One-way driving lane inside the tunnel, good lighting conditions.

[0162] Results:

[0163] Event Type: Debris.

[0164] Risk Level Assessment: L1 (Low Risk), Small and Medium-sized Debris, Single Lane Partially Occupied (10%-20% of the lane), No Complete Lane Blockage, No Sudden Speed Drop in Current Traffic Flow.

[0165] Semantic Description: Two ground foreign objects (possibly dropped boxes or scattered objects) appear, partially occupying the driving lane, posing a risk of sudden braking, scraping, or secondary rear-end collisions for vehicles. It is necessary to alert vehicles behind and promptly handle the debris.

[0166] Disposal Strategy: Based on the matched disposal plan retrieved from the knowledge graph, the following atomic instruction sequence is generated:

[0167] a) VMS Information Board publishes prompt information: "There is an obstacle in the front lane, please drive carefully";

[0168] b) Traffic signal machine controls the tunnel lane indicator light to flash yellow;

[0169] c) Dynamic navigation system updates road conditions and suggests detours;

[0170] d) Automatically generate a clearance task sheet and assign it to the nearest clearance unit;

[0171] e) After disposal, VMS and navigation system return to normal.

[0172] Instruction Execution:

[0173] First stage (pre-commit) : send the proposed instructions to each device controller, wait for device ACK & device state check (timeout threshold 2s).

[0174] Second stage (formal commit) : all devices return to an executable state, formally commit the instruction execution.

[0175] Closed-loop feedback and audit:

[0176] During the disposal process, the system continuously monitors traffic flow changes and the position of the wrecker vehicle through sensors, and the disposal effect is good. The whole process log (including data snapshot, inference result, instruction content, execution state) is recorded in the audit layer, and an explainable report is generated.

[0177] Example 2: fire case

[0178] As shown in Figure 6 ;

[0179] Scenario overview: one-way 4-lane tunnel, night / low light conditions.

[0180] Results:

[0181] Event type: tunnel fire.

[0182] Risk level assessment: L3 (extremely high risk).

[0183] The fire point is located on the right wall of the middle section of the tunnel, accompanied by obvious smoke, which has triggered the heat-sensitive and smoke-sensitive dual-channel alarms; the smoke spreads quickly along the longitudinal direction, and the visible distance is <50m; multiple vehicles are suddenly stopped, and there is a risk of suffocation, chain collision and structure damage.

[0184] Semantic description: there is an open fire in the tunnel and a large amount of toxic smoke is generated, which has caused traffic interruption, and the fire emergency mode needs to be started immediately to prioritize personnel evacuation and fire rescue.

[0185] Disposal strategy: based on the matched disposal plan retrieved from the knowledge graph, the following atomic instruction sequence is generated:

[0186] a) VMS information board immediately issues a red warning: "tunnel fire, immediately leave the exit, do not enter!" ;

[0187] b) Traffic signal machine switches all lane indicator lights in the tunnel to red constant, and closes the tunnel entrance signal light;

[0188] c) Tunnel broadcast and vehicle-road cooperation terminal synchronous voice broadcast: "fire ahead, please maintain a distance and orderly leave the nearest exit" ;

[0189] d) Start the longitudinal smoke exhaust fan (reverse air supply 2.5m / s), and open the cross-sectional smoke exhaust valve to ensure that the smoke is discharged downstream of the fire;

[0190] e) The dynamic navigation system immediately seals the entrances at both ends of the tunnel and induces vehicles to take the nearest national road line;

[0191] f) Automatically generate a fire rescue task sheet and synchronize it to the fire department 119, traffic police, road administration, and emergency center, with the tunnel BIM fire point positioning;

[0192] g) The tunnel lighting system switches to the emergency high-light mode (≥80lx) to guide personnel along the evacuation channel to evacuate;

[0193] h) After the disposal is completed, the safety is confirmed by multiple departments, and the fans, lighting, signals, and navigation are restored to normal in stages.

[0194] Command execution:

[0195] First stage (pre-submission): Send the proposed command to all subsystems (VMS, signal, fan, lighting, broadcast, navigation, rescue platform), wait for device ACK & state self-check (timeout threshold 1.5s).

[0196] Second stage (official submission): All devices return to executable, officially submit the command and lock the fire mode until manually released.

[0197] Closed-loop feedback and audit:

[0198] The disposal is real-time returned by smoke, temperature, CO / VI detector, video AI, and vehicle flow detector; the system evaluates the risk level every 30s, dynamically adjusts the fan speed and evacuation strategy. The trajectory of the rescue vehicle and the number of personnel evacuation are real-time plotted. The whole process log (data snapshot, reasoning result, command content, execution status, department receipt) is written into the audit layer, and the explainable report and improvement suggestions are automatically generated.

[0199] The above is only the preferred embodiment of the present application, and does not limit the technical scope of the present application. Any slight modification, equivalent change and modification made according to the technical essence of the present application to the above embodiment are still within the scope of the technical solution of the present application.

Claims

1. An automatic traffic alarm processing method based on a multimodal large model, characterized in that, Includes the following steps: Step 1: Multi-source heterogeneous data acquisition and real-time access Traffic operation data is collected in real time by sensor devices, and the collected traffic operation data is connected to the system through multiple protocols, and then analyzed and cleaned in real time to provide high-quality input for subsequent processing. Step 2: Multimodal Spatiotemporal Alignment and Fusion Preprocessing Time synchronization and spatial registration are performed on heterogeneous data from different sources and with different sampling rates; at the same time, a lane-level topology structure is constructed based on high-precision maps or real-time mapping results to achieve deep fusion and contextual association of multi-source data in the spatiotemporal dimensions. Step 3: Event Recognition and Semantic Reasoning Based on Multimodal Large Models The pre-trained multimodal large model (MLLM) is used to perform deep analysis and inference on the fused data; Step 4: Generating Disposal Strategies and Arranging Instructions Based on the event identification and risk assessment results, matching contingency plans are retrieved from the knowledge graph, and executable atomic instruction sequences are generated by combining real-time traffic conditions. Based on the event identification results and risk assessment, retrieve the response plan from the knowledge graph; Conflict detection and resource optimization are performed through a rule engine and reinforcement learning strategy, and rollback conditions are preset to ensure operational safety. Step 5: Safe and controllable command execution and linkage control The system uses standardized interfaces to issue commands and monitor the status of roadside equipment, and monitors equipment feedback and changes in traffic conditions in real time during execution. Step Six: Feedback on Treatment Results and Closed-Loop Optimization By collecting traffic operation data after the treatment in real time through multi-source sensors, the effectiveness of the treatment is evaluated, and the feedback data is used for model fine-tuning and strategy optimization to continuously minimize the loss function and form a closed-loop learning mechanism. Step 7: End-to-End Audit and Visualized Retrospective Record the entire process log from data access to command execution. It provides a visual interface to support real-time monitoring and historical review, and generates interpretable reports.

2. The automatic traffic alarm processing method based on a multimodal large model according to claim 1, characterized in that, In step one, the sensor devices include CCTV cameras, millimeter-wave radar, lidar, and V2X roadside units, and the data types collected cover RGB video streams, radar point clouds, and multimodal data of vehicle status messages.

3. The automatic traffic alarm processing method based on a multimodal large model according to claim 1, characterized in that, In step one, high-quality input specifically refers to: Let d be the raw data collected by the s-th sensor at time t. s The data stream accessed by the system (t) can be represented as: Where S represents the number of sensors, and after protocol parsing and data cleaning, high-quality inputs that satisfy integrity constraints are obtained.

4. The automatic traffic alarm processing method based on a multimodal large model according to claim 1, characterized in that, In step two, time synchronization employs a combination of protocol and hardware timestamps to unify the data from all sensors to the same time base, with each modal timestamp recorded as follows: Unified to global time base t * : Spatial registration maps sensor data to a unified lane coordinate system through extrinsic parameter calibration and coordinate transformation. If a modal point is located at position p in the local coordinate system... s Then it is mapped to the lane unified coordinate system p: p=T s p s ,T s ∈SE(3) Among them, T s Transformation matrix for extrinsic parameter calibration; The unified spatiotemporal dataset after fusion is represented as follows:

5. The automatic traffic alarm processing method based on a multimodal large model according to claim 1, characterized in that, In step three, the model extracts features from image, radar, and V2X data through a visual encoder, a point cloud processing network, and a text embedding module, respectively, and uses a cross-modal attention mechanism to achieve feature interaction and enhancement. Among them, image, radar, and V2X features are respectively represented as: Feature interaction is achieved through a cross-modal attention mechanism: Where Q is the query matrix, K is the key matrix, V is the value matrix, d is the scaling factor, and z is the fused cross-modal attention output representation; Event category prediction uses cross-entropy loss: Where, p c Let y be the probability of the event category. c This is a real label; The output includes event type identification, risk level assessment (L1-L3), and semantic description generation, and combines knowledge graphs for rule constraints and interpretability enhancement.

6. The automatic traffic alarm processing method based on a multimodal large model according to claim 1, characterized in that, In step four, a reinforcement learning strategy is used to optimize and control the returns. Where π is the strategy, r t For immediate returns, γ is the discount factor; the generated atomic instruction sequence is denoted as {a1,…,a...} n }; In step 4, the instructions include lane control, speed limit announcement, guidance information dissemination, and emergency resource dispatch.

7. The automatic traffic alarm processing method based on a multimodal large model according to claim 1, characterized in that, In step five, a two-phase commit mechanism is used to ensure the atomicity and consistency of instructions, and supports automatic rollback and manual intervention. The two-phase commit mechanism can be formalized as follows: Real-time monitoring and feedback ensure consistent and secure execution.

8. The automatic traffic alarm processing method based on a multimodal large model according to claim 1, characterized in that, In step six, the effectiveness of the treatment is evaluated, such as the queue dissipation time τ. q With the improvement rate of traffic efficiency η: Among them, Q after Q represents the traffic flow per unit time after processing. before The traffic flow per unit time before processing is η, which is the traffic efficiency improvement rate (in percentage form). Loss function: in, Classify the losses for the event. The loss is used for strategy optimization, where λ is the weighting coefficient of the two losses. The overall optimization objective function is defined.

9. The automatic traffic alarm processing method based on a multimodal large model according to claim 1, characterized in that, In step seven, the full-process log includes data snapshots, inference results, instruction content, and key information on execution status.

10. An automatic traffic alarm processing system based on a multimodal large model, characterized in that, The system consists of six parts: a data acquisition layer, a data access and governance layer, a spatiotemporal fusion layer, a multimodal understanding and reasoning layer, an execution and orchestration layer, and a security and auditing layer; among which... The data acquisition layer is used to acquire multi-source heterogeneous traffic data. The data access and governance layer employs streaming media servers, message queues, and time-series databases to enable the access, cleaning, synchronization, and storage of multi-source data. The spatiotemporal fusion layer is used to perform clock synchronization, spatial calibration, and road topology modeling on data from different modalities, forming a unified traffic flow dataset. The multimodal understanding and reasoning layer, based on the multimodal large model MLLM and knowledge graph, is used to identify traffic incidents and perform semantic reasoning on fused data, outputting incident categories and risk levels. The execution and orchestration layer transforms the inference results into executable strategies, which are then linked to VMS information boards, traffic signals, tunnel fans, broadcasting systems, and emergency dispatch systems via interfaces to achieve automated handling. The security and audit layer is responsible for auditing the interpretability of system access control, logging, rollback mechanisms, and handling processes. The data acquisition layer includes road cameras (CCTV), radar speedometers, multiple lidar systems, V2X roadside units, and meteorological sensors.

Images