Toll station special situation emergency processing system integrated with localized AI large model

Abstract

The invention discloses a toll station special situation emergency processing system integrated with a localized AI large model, and belongs to the technical field of toll station management and control. Comprising an edge computing device, a multi-element sensing module, a localization AI large model engine, a dynamic rule engine and an execution control interface. The edge computing device is responsible for being deployed in a toll station site and is embedded hardware with AI acceleration computing power, and the multi-source sensing module is responsible for being connected with a camera, a millimeter wave radar and an ETC-RSU device. According to the invention, by deploying the lightweight large model in the edge server, millisecond-level identification and automatic disposal of special situations such as vehicle card punching and dangerous goods leakage are realized. The system adopts a multi-modal data fusion technology, and is combined with a dynamic plan engine, so that the operation of core functions can still be ensured in an off-network environment, and the emergency response efficiency and safety of the toll station are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of toll station control, and particularly relates to a special situation emergency handling system for toll stations integrating a localized AI large model. Background Art

[0002] As a key node of the highway network, toll stations face increasingly complex emergency scenarios (such as vehicle ramming through toll barriers, hazardous chemical leakage, abnormal parking, etc.). There are three core defects in the current mainstream processing solutions: High latency in response: Solutions relying on cloud AI analysis need to upload video streams to the cloud server (average time-consuming ≥ 2 seconds), resulting in the ramming-through vehicle escaping the scene before the system responds. Fragmentation of multi-modal data: Video surveillance, radar trajectories, and ETC transaction data are processed independently, lacking the ability of spatio-temporal alignment. When the radar detects a speeding vehicle, it is unable to associate with the ETC arrears information, and the missed detection rate is as high as 15%. Failure in the offline scenario: The network stability of toll stations in remote areas is poor, and the traditional cloud architecture completely collapses in the offline state. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a special situation emergency handling system for toll stations integrating a localized AI large model.

[0004] The technical solution adopted to solve the above technical problem is: A special situation emergency handling system for toll stations integrating a localized AI large model, including five subsystems: an edge computing device, a multi-source perception module, a localized AI large model engine, a dynamic rule engine, and an execution control interface; The edge computing device is responsible for being deployed at the toll station site, and is an embedded hardware with AI acceleration computing power. The multi-source perception module is responsible for connecting cameras, millimeter-wave radars, and ETC-RSU devices to collect vehicle video streams, trajectory point clouds, and transaction status data in real time; The localized AI large model engine is responsible for loading a lightweight multi-modal Transformer neural network, integrating a spatio-temporal alignment unit, and performing cross-modal synchronization on video frames, radar scan cycles, and ETC transaction timestamps.

[0005] Through the above technical solution, the lightweight large model is deployed on the edge server to achieve millisecond-level recognition and automated disposal of special situations such as vehicle ramming through toll barriers and hazardous material leakage. The system adopts multi-modal data fusion technology, combined with a dynamic pre-plan engine, to still ensure the operation of core functions in the offline environment, significantly improving the emergency response efficiency and safety of toll stations.

[0006] Furthermore, the dynamic rule engine is responsible for automatically matching the emergency plan library according to the special situation type and confidence level output by the AI large model; the execution control interface is responsible for controlling lane barrier machines, signal lights, and emergency broadcast devices through the industrial bus protocol.

[0007] Furthermore, the lightweight multi-modal Transformer neural network is generated through knowledge distillation technology. The lightweight multi-modal Transformer neural network uses a vision-language large model with tens of billions of parameters as the teacher model, adopts hierarchical structured pruning to remove redundant attention heads, retains key weight layers, embeds an adaptive quantization module, and dynamically switches between FP32 and FP16 precisions.

[0008] Furthermore, the spatio-temporal alignment unit performs operations including synchronizing the data acquisition times of each sensor based on the hardware clock, and then establishing a 3D-2D projection matrix from the radar coordinate system to the video pixel coordinate system: where is the camera internal parameter matrix, which projects 3D points in the camera coordinate system to 2D points in the image pixels, is the rotation matrix, which describes the rotation transformation from the radar coordinate system to the camera coordinate system, is the translation vector, which describes the spatial offset from the radar origin to the camera origin, is the lateral distance of the target in the radar coordinate system, is the longitudinal distance of the target in the radar coordinate system, is the height of the target in the radar coordinate system.

[0009] Through the above technical solutions, the vehicle position detected by the radar can be mapped to the corresponding pixel points in the video frame, enabling the AI large model to simultaneously analyze "radar trajectory + visual features + ETC data".

[0010] Furthermore, the emergency plan library contains a hierarchical response mechanism. The hierarchical response mechanism includes level 1 special situations and level 2 special situations. The level 1 special situation triggers lane closure, audible and visual alarms, and linkage with the traffic police platform. The level 2 special situation triggers broadcast warnings and dispatching instructions for patrolmen. The response strategy of the response mechanism is dynamically optimized based on a reinforcement learning model, and the input parameters include lane position, real-time traffic flow, and weather conditions.

[0011] Through the above technical solutions, corresponding operations can be carried out according to different levels of special situations, greatly improving the emergency response speed.

[0012] Furthermore, the edge computing device works in coordination with the cloud management platform. The edge-side computing device stores 7 days of historical data and performs real-time inference. The cloud management platform receives encrypted event logs, updates the model weights through incremental learning, and transmits them back to the edge-side computing device.

[0013] Further, when the network is disconnected, the edge computing device enables the local cache model to continue reasoning. The control instruction is directly connected to the actuator through the RS485 bus, and the data during the network disconnection is automatically synchronized to the cloud after the network is restored.

[0014] Through the above technical solution, it is possible to prevent the edge computing device from losing its function after the network is disconnected.

[0015] Further, the execution control interface supports controlling the lane barrier machine through the Modbus / TCP protocol, publishing signal light instructions through the MQTT protocol, and linking the surveillance dome camera to track the target vehicle through the GB / T 28181 protocol.

[0016] Through the above technical solution, each execution end of the toll station can respond quickly.

[0017] The beneficial effects of the present invention are as follows: By deploying the lightweight large model on the edge server, the present invention realizes millisecond-level identification and automated disposal of special situations such as vehicle ramming through the toll gate and dangerous goods leakage. The system adopts multi-modal data fusion technology and combines a dynamic pre-plan engine to ensure the operation of core functions even in a network-disconnected environment, significantly improving the emergency response efficiency and safety of the toll station. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is the system framework flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] As Figure 1 shown, a toll station special situation emergency handling system integrating a local AI large model in this embodiment includes five subsystems: an edge computing device, a multi-source perception module, a local AI large model engine, a dynamic rule engine, and an execution control interface; The edge computing device is responsible for being deployed on the toll station site and is an embedded hardware with AI acceleration computing power. The multi-source perception module is responsible for connecting cameras, millimeter-wave radars, and ETC-RSU devices to collect vehicle video streams, trajectory point clouds, and transaction status data in real time; The localized AI large model engine is responsible for loading the lightweight multi-modal Transformer neural network, integrating the spatio-temporal alignment unit, performing cross-modal synchronization on video frames, radar scan cycles, and ETC transaction timestamps, deploying the lightweight large model on the edge server, and achieving millisecond-level recognition and automated disposal of special situations such as vehicle ramming through toll gates and dangerous goods leakage. The system adopts multi-modal data fusion technology and combines a dynamic pre-plan engine to ensure the operation of core functions even in a network-disconnected environment, significantly improving the emergency response efficiency and safety of toll stations.

[0021] The dynamic rule engine is responsible for automatically matching the emergency pre-plan library according to the special situation type and confidence level output by the AI large model; the execution control interface is responsible for controlling lane barrier machines, signal lights, and emergency broadcast devices through the industrial bus protocol.

[0022] The lightweight multi-modal Transformer neural network is compressed and generated through knowledge distillation technology. The lightweight multi-modal Transformer neural network uses a vision-language large model with tens of billions of parameters as the teacher model, adopts hierarchical structured pruning to remove redundant attention heads, retains key weight layers, embeds an adaptive quantization module, and dynamically switches between FP32 and FP16 precisions.

[0023] The spatio-temporal alignment unit executes operations including synchronizing the data acquisition times of each sensor based on the hardware clock, and then establishing a 3D-2D projection matrix from the radar coordinate system to the video pixel coordinate system: where is the camera internal parameter matrix, which projects 3D points in the camera coordinate system to 2D points in the image pixels, is the rotation matrix, which describes the rotational transformation from the radar coordinate system to the camera coordinate system, is the translation vector, which describes the spatial offset from the radar origin to the camera origin, is the lateral distance of the target in the radar coordinate system, is the longitudinal distance of the target in the radar coordinate system, is the height of the target in the radar coordinate system. It can map the vehicle position detected by the radar to the corresponding pixel points in the video frame, enabling the AI large model to simultaneously analyze "radar trajectory + visual features + ETC data".

[0024] The emergency plan library contains a hierarchical response mechanism, which includes level 1 special situations and level 2 special situations. The level 1 special situation triggers lane closure, audible and visual alarms, and linkage with the traffic police platform. The level 2 special situation triggers broadcast warnings and dispatching instructions for patrolmen. The response strategy of the response mechanism is dynamically optimized based on a reinforcement learning model. The input parameters include lane position, real-time traffic flow, and weather conditions, and corresponding operations can be performed according to different levels of special situations, greatly improving the emergency response speed.

[0025] The edge computing device works in coordination with the cloud management platform. The edge-side computing device stores 7-day historical data and performs real-time inference. The cloud management platform receives encrypted event logs, updates the model weights through incremental learning, and transmits them back to the edge-side computing device.

[0026] When the network is disconnected, the edge computing device enables the local cache model to continue inference. The control instructions are directly connected to the actuator through the RS485 bus. After the network is restored, the data during the network disconnection period is automatically synchronized to the cloud, which can prevent the edge computing device from losing its function after the network is disconnected.

[0027] The execution control interface supports controlling the lane barrier machine through the Modbus / TCP protocol, publishing signal light instructions through the MQTT protocol, and linking the surveillance dome camera to track the target vehicle through the GB / T 28181 protocol, which can enable each execution end of the toll station to respond quickly.

[0028] It includes the following specific steps: Step 1: Receive multi-source sensor data streams in real time; Step 2: Invoke the localized AI large model for spatio-temporal fusion analysis, and output the type of special situation and confidence level; Step 3: When the confidence level ≥ 95%, the dynamic rule engine automatically triggers the emergency plan; Step 4: Send control instructions to the execution device through the industrial protocol; Step 5: Encrypt and record the event log and upload it to the cloud for auditing.

[0029] The fusion analysis in Step 2 specifically includes: extracting vehicle appearance, license plate, and behavior features from the video stream; calculating speed, acceleration, and heading angle from the radar point cloud; verifying the matching of the payment status and vehicle position with the ETC data; and inputting multi-modal features into the Transformer encoder to generate joint embedding vectors.

[0030] The above is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. An integrated toll station special situation emergency handling system with a localized AI large model, characterized in that, It includes five subsystems: an edge computing device, a multi-source perception module, a local AI large model engine, a dynamic rule engine, and an execution control interface; The edge computing device is responsible for being deployed at the toll station site and is an embedded hardware with AI acceleration computing power. The multi-source perception module is responsible for connecting cameras, millimeter-wave radars, and ETC-RSU devices to collect vehicle video streams, trajectory point clouds, and transaction status data in real time; The local AI large model engine is responsible for loading a lightweight multi-modal Transformer neural network and integrating a spatio-temporal alignment unit to perform cross-modal synchronization of video frames, radar scan cycles, and ETC transaction timestamps.

2. The toll station special situation emergency handling system integrating a local AI large model according to claim 1, wherein The dynamic rule engine is responsible for automatically matching the emergency plan library according to the special situation type and confidence level output by the AI large model; the execution control interface is responsible for controlling the lane barrier machine, signal lights, and emergency broadcast devices through the industrial bus protocol.

3. The toll station special situation emergency handling system integrating a local AI large model according to claim 1, characterized in that, The lightweight multi-modal Transformer neural network is generated by compressing through knowledge distillation technology. The lightweight multi-modal Transformer neural network uses a vision-language large model with tens of billions of parameters as the teacher model, adopts hierarchical structured pruning to remove redundant attention heads, retains key weight layers, embeds an adaptive quantization module, and dynamically switches between FP32 and FP16 precisions.

4. The toll station special situation emergency handling system integrating a local AI large model according to claim 1, characterized in that, The spatio-temporal alignment unit performs operations including synchronizing the data acquisition times of each sensor based on the hardware clock, and then establishing a 3D-2D projection matrix from the radar coordinate system to the video pixel coordinate system: where is the camera intrinsic matrix, which projects the 3D points in the camera coordinate system to the 2D points of the image pixels, is the rotation matrix, which describes the rotation transformation from the radar coordinate system to the camera coordinate system, is the translation vector, which describes the spatial offset from the radar origin to the camera origin, is the lateral distance of the target in the radar coordinate system, is the longitudinal distance of the target in the radar coordinate system, is the height of the target in the radar coordinate system.

5. The toll station special situation emergency handling system integrating a localized AI large model according to claim 1, characterized in that, The emergency plan library contains a hierarchical response mechanism. The hierarchical response mechanism includes level 1 special situations and level 2 special situations. The level 1 special situation triggers lane closure, sound and light alarms, and linkage with the traffic police platform. The level 2 special situation triggers broadcast warnings and dispatching instructions for patrolmen. The response strategy of the response mechanism is dynamically optimized based on a reinforcement learning model, and the input parameters include lane position, real-time traffic flow, and weather conditions.

6. The toll station special situation emergency handling system integrating a localized AI large model according to claim 1, characterized in that, The edge computing device works in coordination with the cloud management platform. The edge-side computing device stores 7 days of historical data and performs real-time inference. The cloud management platform receives encrypted event logs, updates the model weights through incremental learning, and sends them back to the edge-side computing device.

7. The toll station special situation emergency handling system integrating a local AI large model according to claim 6, characterized in that, The edge computing device enables local cached models to continue inference when the network is disconnected. Control instructions are directly connected to the actuator through the RS485 bus, and data during the network disconnection period is automatically synchronized to the cloud after the network is restored.

8. The toll station special situation emergency handling system integrating a local AI large model according to claim 1, wherein, The execution control interface supports controlling the lane barrier machine through the Modbus / TCP protocol, publishing signal light instructions through the MQTT protocol, and linking the surveillance dome camera to track the target vehicle through the GB / T 28181 protocol.

Images