Traffic control system based on multi-modal perception

Abstract

The application relates to a traffic management and control system based on multi-modal perception, which comprises a support arranged at the top of a road intersection vertical rod, an installation platform is arranged on the support, a radar monitoring module is arranged on the installation platform, the radar monitoring module comprises a millimeter wave radar and a laser radar, an angle adjusting assembly is arranged on the installation platform, the angle adjusting assembly is used for adjusting the included angle between the main shaft line of the laser radar and the main shaft line of the millimeter wave radar, a meteorological monitoring module is arranged on the installation platform, the meteorological monitoring module comprises at least one of a temperature sensor, a humidity sensor, a visibility sensor and a barometric pressure sensor, and a visual monitoring module is arranged on the outer side wall of the road intersection vertical rod, the visual monitoring module comprises a camera. The system can improve the accuracy of traffic information perception.

Description

Technical Field

[0001] This application relates to the field of traffic control technology, and in particular to a traffic control system based on multimodal perception. Background Technology

[0002] With the accelerating pace of global urbanization and the dramatic increase in motor vehicle ownership, urban transportation systems are facing unprecedented challenges. Traffic congestion is becoming increasingly common, and traffic accidents are frequent, severely impacting not only the travel efficiency and quality of life of urban residents but also significantly hindering socio-economic development. Against this backdrop, Intelligent Transportation Systems (ITS) have emerged and continue to evolve. ITS aims to comprehensively, in real-time, and efficiently manage and control transportation systems through advanced information technology, data communication and transmission technology, electronic control technology, and computer processing technology, in order to alleviate traffic pressure and improve traffic safety.

[0003] However, current systems often rely too heavily on a single sensor for traffic information perception and acquisition, such as using only cameras or radar. While a single camera can provide rich visual information under normal weather conditions, its recognition accuracy will drop significantly in adverse weather environments such as rain, fog, and sandstorms. Utility Model Content

[0004] Therefore, it is necessary to provide a traffic control system based on multimodal perception that can improve the accuracy of traffic information perception.

[0005] A traffic control system based on multimodal perception. The traffic control system includes:

[0006] A bracket is installed on top of the intersection pole, and the bracket is equipped with an installation platform;

[0007] A radar monitoring module is installed on the mounting platform, and the radar monitoring module includes millimeter-wave radar and lidar.

[0008] An angle adjustment component is disposed on the mounting platform, and the angle adjustment component is used to adjust the angle between the main axis of the lidar and the main axis of the millimeter-wave radar.

[0009] A meteorological monitoring module is installed on the mounting platform, and the meteorological monitoring module includes at least one of a temperature sensor, a humidity sensor, a visibility sensor, and a barometric pressure sensor;

[0010] A visual monitoring module is installed on the outer wall of the intersection pole, and the visual monitoring module includes a camera.

[0011] In one embodiment, the traffic control system further includes a switch and an edge computing device communicatively connected to the switch;

[0012] The millimeter-wave radar is communicatively connected to the switch via a first data transmission cable;

[0013] The lidar is connected to the switch via a second data transmission cable.

[0014] The meteorological monitoring module is connected to the switch via a third data transmission cable.

[0015] The camera is connected to the switch via a fourth data transmission cable.

[0016] In one embodiment, the edge computing device is provided with a hardware trigger synchronization module, which includes a synchronization pulse generator and multiple synchronization signal output ports connected to the synchronization pulse generator; the millimeter-wave radar, the lidar and the camera are all provided with synchronization trigger pins, and the multiple synchronization trigger pins are connected to the multiple synchronization signal output ports through multiple signal cables;

[0017] The synchronization pulse generator is configured to simultaneously send trigger signals to the millimeter-wave radar, the lidar, and the camera through multiple synchronization signal output ports, and the millimeter-wave radar, lidar, and camera are configured to collect traffic data in response to the trigger.

[0018] In one embodiment, the switch and the edge computing device are housed in an intersection cabinet adjacent to the intersection pole.

[0019] In one embodiment, the traffic control system further includes a cloud-based decision-making module and an execution control module;

[0020] The cloud-based decision-making module is connected to the edge computing device and is configured to receive meteorological anomaly signals from the edge computing device, triggering the emergency equipment activation of the execution control module.

[0021] The emergency equipment of the execution control module includes a holographic projection module and a directional speaker. The intersection pole where the holographic projection module is located and the intersection pole where the directional speaker is located are set at intervals. The holographic projection module is configured to project virtual arrows, and the directional speaker is configured to play voice warnings.

[0022] In one embodiment, the execution control module further includes an LED information screen, which is located at the intersection. The LED information screen is communicatively connected to the edge computing device via a network interface and is configured to receive intersection traffic data from the edge computing device to display intersection traffic information.

[0023] In one embodiment, the angle between the main axis of the lidar and the main axis of the millimeter-wave radar is 5 degrees to 25 degrees.

[0024] In one embodiment, the main axis of the lidar is aligned with the main axis of the camera.

[0025] In one embodiment, the installation platform includes a first platform and a second platform, with the first platform located above the second platform; the meteorological monitoring module is mounted on the first platform, and the radar monitoring module is mounted on the second platform.

[0026] In one embodiment, the lidar is further provided with a connecting block, and the angle adjustment component includes a drive motor, which is mounted on the second platform, with the motor shaft of the drive motor extending out of the second platform and fixedly connected to the connecting block.

[0027] The aforementioned traffic control system based on multimodal perception includes: a bracket mounted on top of a road intersection pole, with an installation platform on the bracket; a radar monitoring module mounted on the installation platform, comprising millimeter-wave radar and lidar; an angle adjustment component mounted on the installation platform, used to adjust the angle between the main axis of the lidar and the main axis of the millimeter-wave radar; a meteorological monitoring module mounted on the installation platform, comprising at least one of a temperature sensor, a humidity sensor, a visibility sensor, and a barometric pressure sensor; and a visual monitoring module mounted on the outer wall of the road intersection pole, comprising a camera. This traffic control system integrates millimeter-wave radar, lidar, a camera, and multi-dimensional meteorological monitoring sensors, significantly improving the system's target detection, recognition, and tracking capabilities under various complex weather conditions, ensuring the continuity and accuracy of traffic information. Meanwhile, the angle adjustment component allows the main axis of the lidar to be adjusted to the angle between the main axis of the millimeter-wave radar, which can avoid excessive overlap of the field of view, thereby more effectively covering different areas of the intersection, reducing blind spots, and allowing the system to be optimized on-site according to the size, shape and traffic flow characteristics of different intersections to achieve the best area coverage and perception performance. Attached Figure Description

[0028] Figure 1This is a schematic diagram of the architecture of a multimodal perception traffic control system in one embodiment. Detailed Implementation

[0029] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0030] In the description of this application, it should be understood that if terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" appear, these terms indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0031] Furthermore, where the terms "first" and "second" appear, these terms are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, where the term "multiple" appears, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0032] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0033] In this application, unless otherwise expressly specified and limited, the use of descriptions such as "above" or "below" the second feature indicates that the first and second features are in direct contact or indirect contact via an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. Similarly, "below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0034] It should be noted that if an element is referred to as being "fixed to" or "set on" another element, it can be directly on the other element or there may be an intervening element. If an element is considered to be "connected to" another element, it can be directly connected to the other element or there may be an intervening element. If so, the terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used in this application are for illustrative purposes only and do not represent the only possible implementation.

[0035] One embodiment of this application provides a traffic control system based on multimodal perception, including a support frame, a radar monitoring module, an angle adjustment component, a meteorological monitoring module, and a visual monitoring module.

[0036] The bracket in this embodiment is installed on the top of the intersection pole, and the bracket is provided with an installation platform.

[0037] For example, the bracket may include a metal rod and a fixed base, the metal rod being mounted on top of the intersection pole via the fixed base, and the mounting platform being mounted on the metal rod.

[0038] The radar monitoring module of this application embodiment is mounted on the installation platform, and the radar monitoring module includes millimeter-wave radar and lidar.

[0039] The angle adjustment component of this application embodiment is disposed on the mounting platform, and the angle adjustment component is used to adjust the angle between the main axis of the lidar and the main axis of the millimeter-wave radar.

[0040] The main axis refers to the center line of the field of view of both the lidar and millimeter-wave radar. By adjusting the angle between the main axes of the lidar and the millimeter-wave radar using the angle adjustment component, the intersection area can be covered more comprehensively, allowing for more accurate perception of traffic conditions.

[0041] The meteorological monitoring module of this application embodiment is installed on the installation platform, and the meteorological monitoring module includes at least one of a temperature sensor, a humidity sensor, a visibility sensor and a barometric pressure sensor.

[0042] By centrally mounting the radar monitoring module and the meteorological monitoring module on the installation platform at the top of the intersection pole, it is beneficial to simplify on-site wiring, reduce installation difficulty, and take advantage of the height of the top of the pole to obtain a wider field of vision.

[0043] The visual monitoring module of this application embodiment is disposed on the outer wall of the intersection pole, and the visual monitoring module includes a camera.

[0044] The visual monitoring module is mounted on the outer wall of the intersection pole. Its layered deployment with radar and weather sensors avoids physical obstruction between different types of sensors and allows the camera to be independently adjusted according to its optimal viewing angle, ensuring effective capture of traffic details and overall road conditions. Placing it at a lower position on the outer wall better aligns with the height of vehicles and pedestrians, providing a more comprehensive horizontal field of view and preventing a downward viewing angle from hindering the capture of details at the far end of the intersection or the overall traffic flow.

[0045] The aforementioned traffic control system based on multimodal perception includes: a bracket mounted on top of a road intersection pole, with an installation platform on the bracket; a radar monitoring module mounted on the installation platform, comprising millimeter-wave radar and lidar; an angle adjustment component mounted on the installation platform, used to adjust the angle between the main axis of the lidar and the main axis of the millimeter-wave radar; a meteorological monitoring module mounted on the installation platform, comprising at least one of a temperature sensor, a humidity sensor, a visibility sensor, and a barometric pressure sensor; and a visual monitoring module mounted on the outer wall of the road intersection pole, comprising a camera. The traffic control system of this application integrates millimeter-wave radar, lidar, a camera, and multi-dimensional meteorological monitoring sensors, greatly improving the system's target detection, recognition, and tracking capabilities under various complex weather conditions, ensuring the continuity and accuracy of traffic information. Meanwhile, the angle adjustment component allows the main axis of the lidar to be adjusted to the angle between the main axis of the millimeter-wave radar, which can avoid excessive overlap of the field of view, thereby more effectively covering different areas of the intersection, reducing blind spots, and allowing the system to be optimized on-site according to the size, shape and traffic flow characteristics of different intersections to achieve the best area coverage and perception performance.

[0046] In one exemplary embodiment, the angle between the main axis of the lidar and the main axis of the millimeter-wave radar is 5 degrees to 25 degrees, and the main axis of the lidar is aligned with the main axis of the camera.

[0047] Both millimeter-wave radar and lidar have specific field of view angles, i.e., the area they can perceive. By setting the angle between their main axes to 5 degrees to 25 degrees, they can work together from different horizontal perspectives to achieve more comprehensive and robust coverage of intersections. Aligning the camera lens with the lidar's center detection direction on the horizontal plane helps ensure that the two sensors work together to perceive the same area or target.

[0048] In one exemplary embodiment, the mounting platform includes a first platform and a second platform, with the first platform positioned above the second platform. The meteorological monitoring module is mounted on the first platform, and the radar monitoring module is mounted on the second platform. By mounting the meteorological monitoring module on the first platform of the support structure, it can be ensured that the meteorological sensors are not easily interfered with by other equipment or vehicles below, thus obtaining the most accurate meteorological data.

[0049] The first platform and the second platform can both be disposed on the outer side wall of the bracket; the bracket has a circular cross-section, and the first platform and the second platform are annularly fitted onto the bracket; the first platform, the second platform, and the bracket can be integrally formed. Alternatively, the first platform and the second platform can be welded to the bracket.

[0050] In an exemplary embodiment, the lidar is further provided with a connecting block, and the angle adjustment component includes a drive motor, which is mounted on the second platform, with the motor shaft of the drive motor extending out of the second platform and fixedly connected to the connecting block.

[0051] In this embodiment, the orientation of the lidar is adjusted by a drive motor, while the millimeter-wave radar can remain stationary.

[0052] In other embodiments, the lidar may be equipped with a first connecting block, and the angle adjustment component includes a first drive motor, which is mounted on the second platform. The motor shaft of the first drive motor extends out of the second platform and is fixedly connected to the first connecting block. The millimeter-wave radar may be equipped with a second connecting block, and the angle adjustment component includes a second drive motor, which is mounted on the second platform. The motor shaft of the second drive motor extends out of the second platform and is fixedly connected to the second connecting block. By using dual drive motors, the pointing of the millimeter-wave radar and lidar can be optimized separately for different intersection sizes, traffic flow directions, and specific sensing needs, thereby maximizing their detection performance and coverage.

[0053] The aforementioned drive motor can be a stepper motor or a servo motor and is connected to an edge computing device, enabling remote and automated angle adjustment without the need for manual pole climbing, thus greatly improving operation and maintenance efficiency.

[0054] For example, see Figure 1 , Figure 1 The diagram shows a system architecture of a traffic control system based on multimodal perception according to one embodiment of this application. The front-end perception layer of this embodiment includes millimeter-wave radar, a 4K starlight camera, a meteorological sensor, and lidar.

[0055] Millimeter-wave radar is primarily used for real-time detection of vehicle speed, distance, and azimuth, supporting multi-target classification (cars / trucks / bicycles). It outperforms cameras, especially in rainy or foggy weather, providing reliable obstacle detection. It connects to an edge computing unit via gigabit Ethernet, transmitting point cloud data, which is then spatiotemporally aligned with lidar data before being output to an AI model for trajectory prediction. The millimeter-wave radar is mounted on a pole at the intersection, 5.5 meters high, at a 15° angle to the lidar to cover the entire intersection area. Basic parameters include a 77GHz operating frequency, a 0-200 meter detection range, ±5cm ranging accuracy, and the ability to simultaneously track 64 targets.

[0056] The 4K starlight-level camera is responsible for capturing license plate recognition and detecting pedestrians or non-motorized vehicles (supporting 14 types of violations), and provides RGB-Depth data to assist LiDAR 3D modeling. It streams data to the edge computing unit via the RTSP protocol, triggering violation detection and behavior analysis, and cross-verifies target attributes (such as distinguishing stationary vehicles from pedestrians) in collaboration with millimeter-wave radar. The 4K starlight-level camera is installed on a pole at the intersection, 30 meters behind the stop line, aligned with the LiDAR's optical axis. The basic parameters of the 4K starlight-level camera include a resolution of 3840×2160, a minimum illumination of 0.005 lux, and a wide dynamic range of 120dB. Its built-in GPU supports YOLOv7 real-time inference (30FPS).

[0057] The meteorological sensor is used to monitor environmental parameters in real time and dynamically adjust detection thresholds (e.g., increasing radar weight in foggy weather). The meteorological sensor can send data to the edge computing unit every 10 seconds, triggering adaptive algorithms (e.g., enabling rain and fog enhancement mode). The device is installed on top of a traffic light pole, 2 meters away from the camera. Basic parameters include temperature and humidity accuracy of ±2%RH / ±0.3℃, visibility measurement of 10m-10km (±5%), and RS485 / Modbus output protocol.

[0058] The lidar can be a 16-line Lidar-X16 lidar with a ranging accuracy of ±2cm and a horizontal field of view of 120°. Working in conjunction with millimeter-wave radar, the lidar provides the system with more accurate 3D point cloud data, enhancing the ability to identify and track traffic targets in complex scenarios.

[0059] In one exemplary embodiment, the traffic control system further includes a switch and an edge computing device communicatively connected to the switch.

[0060] The millimeter-wave radar is connected to the switch via a first data transmission cable.

[0061] The lidar is connected to the switch via a second data transmission cable.

[0062] The camera is connected to the switch via a third data transmission cable.

[0063] The meteorological monitoring module is connected to the switch via a fourth data transmission cable.

[0064] The millimeter-wave radar, the lidar, the camera, and the meteorological monitoring module communicate with the edge computing device through a switch. The switch can forward the data collected by the millimeter-wave radar, the lidar, the camera, and the meteorological monitoring module to the edge computing device.

[0065] For example, the switch is configured with multiple Ethernet interfaces, which can be RJ45 interfaces. The millimeter-wave radar is also configured with an Ethernet interface, which can be an RJ45 interface. The Ethernet interface of the millimeter-wave radar is connected to the Ethernet interface of the switch via a first data transmission cable. The first data transmission cable can be a Cat5e or Cat6 Ethernet cable, with four pairs of twisted wires inside for high-speed data transmission, and the data transmission supports TCP / UDP protocols.

[0066] For example, the LiDAR also has an Ethernet interface, which can be an RJ45 interface. The LiDAR's Ethernet interface is connected to the switch's Ethernet interface via a second data transmission cable. The second data transmission cable can be a Cat5e or Cat6 Ethernet cable, with four pairs of twisted wires inside for high-speed data transmission, supporting TCP / UDP protocols.

[0067] For example, the camera can also be configured with an Ethernet interface, which can be an RJ45 interface. The camera's Ethernet interface is connected to the switch's Ethernet interface via a third data transmission cable. The third data transmission cable can be a Cat5e or Cat6 Ethernet cable, with four pairs of twisted wires inside for high-speed data transmission, and the data transmission supports the RTSP (Real-Time Streaming Protocol).

[0068] For example, each sensor in the meteorological monitoring module is equipped with an RS485 interface, and the switch is also equipped with an RS485 interface. The fourth data transmission line includes an RS485 bus, and each sensor and the switch are connected in parallel to the RS485 bus via the RS485 interface. The switch communicates with multiple meteorological sensors on the bus via the Modbus protocol.

[0069] Different sensors exhibit temporal and spatial deviations in their data due to variations in sampling frequency, installation location, and data processing procedures. For example, the data acquisition time of cameras and radar may have an error of 0.3 seconds or even longer. This leads to significant deviations in subsequent trajectory prediction and target recognition, making it difficult to accurately track and analyze traffic targets in real time.

[0070] In an exemplary embodiment, to further accurately track and analyze traffic targets in real time, the edge computing device is equipped with a hardware trigger synchronization module. The hardware trigger synchronization module includes a synchronization pulse generator and multiple synchronization signal output ports connected to the synchronization pulse generator. The millimeter-wave radar, the lidar, and the camera are all equipped with synchronization trigger pins, and the multiple synchronization trigger pins are connected to the multiple synchronization signal output ports through multiple signal cables.

[0071] The synchronization trigger pin can be an SMA interface, a BNC interface, or a general-purpose I / O pin. The signal cable can be a single-core shielded cable (like a coaxial cable) or a multi-core signal cable.

[0072] The synchronization pulse generator is configured to simultaneously send trigger signals to the millimeter-wave radar, the lidar, and the camera via a switch through multiple synchronization signal output ports. The millimeter-wave radar, lidar, and camera are configured to collect traffic data in response to the triggers. By simultaneously sending trigger signals to the millimeter-wave radar, lidar, and camera through the synchronization pulse generator, data acquisition can begin at the same nanosecond level or even less, thereby resolving the time deviation problem caused by differences in sampling periods, internal processing delays, or transmission delays of different sensors.

[0073] In an exemplary embodiment, to further accurately track and analyze traffic targets in real time, the edge computing device is equipped with a PPS distribution module. The PPS distribution module includes a PPS signal generation circuit, a buffer amplifier, multiple output drive circuits, and multiple PPS signal output ports corresponding to the multiple output drive circuits. The PPS signal generation circuit is configured to generate a high-precision pulses per second (PPS) synchronization signal. The millimeter-wave radar, the lidar, and the camera are all equipped with PPS input interfaces. The multiple PPS input interfaces are connected to the multiple PPS signal output ports through multiple independent signal cables and a switch. The PPS distribution module is configured to simultaneously send the PPS synchronization signal to the millimeter-wave radar, the lidar, and the camera through the multiple PPS signal output ports. The millimeter-wave radar, the lidar, and the camera are configured to trigger a preset internal clock calibration in response to the PPS synchronization signal.

[0074] In one exemplary embodiment, the switch and the edge computing device may be housed in an intersection cabinet adjacent to the intersection pole.

[0075] In an exemplary embodiment, the switch and the edge computing device may also be mounted on the intersection pole. The intersection pole has a receiving groove inside, and the switch and the edge computing device are mounted inside the groove. The outer wall of the intersection pole has an opening communicating with the receiving groove. The intersection pole has a cable receiving cavity communicating with the receiving groove. The top of the intersection pole has a cable receiving port communicating with the cable receiving cavity. The bracket is located near the cable receiving port. The first data transmission cable, the second data transmission cable, and the third data transmission cable are all stored in the receiving cavity through the cable receiving port.

[0076] For example, the aforementioned signal cable can also be housed within the housing cavity.

[0077] For example, a door is provided on the outer side wall of the intersection pole near the opening, the door is provided with several heat dissipation holes, and a rain cover extends from the outer side wall at the top of the door.

[0078] For example, please refer to Figure 1The edge computing device in this embodiment can employ an AI acceleration chip, primarily used to accelerate YOLOv7 target detection (120FPS) and ST-GCN trajectory prediction (latency <10ms). The edge computing device can also be embedded in an intersection control cabinet and connected to an edge server via PCIe. After receiving and processing multi-sensor data in parallel, the edge computing device outputs a risk level to the decision-making module and collaborates with the edge server to dynamically load and update the model. The basic parameters of the edge computing device include 128 TOPS computing power (INT8), 15W power consumption, and a PCIe 4.0 × 16 interface.

[0079] The edge server is equipped with a dynamic timing algorithm that optimizes traffic light timing based on reinforcement learning (PPO algorithm), reducing average delays by 25%-30%. The edge server communicates with the traffic lights via UDP, receiving global traffic predictions (such as data from major events) from the cloud-based traffic management system. It then combines this data with local real-time data to generate timing plans and works in conjunction with the V2X module to dynamically plan green light routes for emergency vehicles. It supports 9-phase cycles dynamically adjustable from 30-180 seconds, with a minimum green light duration of 4 seconds.

[0080] The fault diagnosis system uses an LSTM model to identify sensor faults (such as lens obstruction or abnormal radar signals). Integrated with an edge server, it monitors device status via a heartbeat mechanism. Upon detecting a fault, it automatically switches to a backup sensor and triggers a drone inspection (via the MQTT protocol). Its basic parameters include a detection accuracy of 99.2% and a response time of <2 seconds.

[0081] The edge computing device in this embodiment uses an AI chip to achieve low-power (no more than 15W) and high-concurrency (120FPS target detection) edge computing, thereby improving real-time decision-making capabilities and reducing traffic signal timing delays.

[0082] In one exemplary embodiment, the traffic control system further includes a cloud-based decision-making module and an execution control module.

[0083] The cloud-based decision-making module is connected to the edge computing device and is configured to receive meteorological anomaly signals from the edge computing device, triggering the activation of the emergency equipment in the execution control module.

[0084] The emergency equipment of the execution control module includes a holographic projection module and a directional speaker. The intersection pole where the holographic projection module is located and the intersection pole where the directional speaker is located are set at intervals. The holographic projection module is configured to project virtual arrows, and the directional speaker is configured to play voice warnings.

[0085] In one exemplary embodiment, the execution control module further includes an LED information screen, which is disposed at the intersection; the LED information screen is communicatively connected to the edge computing device via a network interface, and the LED information screen is configured to receive intersection traffic data from the edge computing device to display intersection traffic information.

[0086] For example, please refer to Figure 1 In this embodiment, the cloud-based decision-making module is a traffic brain platform deployed on a government cloud, ensuring a network bandwidth of ≥100Mbps. The cloud-based decision-making module deploys a city-level traffic digital twin model with an accuracy of less than 0.5 meters. The traffic digital twin model supports federated learning for over 1000 intersections, generating global optimization strategies based on real-time and historical data from each intersection, such as increasing warning frequency by 20% during peak traffic hours. It also integrates with the emergency management system, automatically activating flood warning plans for waterlogged sections during extreme weather events such as heavy rain, and coordinating the collaborative work of holographic projection and directional sound columns.

[0087] Digital twin models can simulate the impact of extreme weather (such as heavy rain, heavy snow, and strong winds) on urban road networks. Through big data analysis and model prediction, they can predict potential traffic congestion hotspots and time points, plan and optimize emergency response routes in advance, and provide decision support for traffic management departments.

[0088] The execution control module may include a holographic projection module, which projects virtual arrows (error <10cm) and warning halos (5-meter radius), supporting AR ground guidance. It receives commands from edge computing devices via an HDMI 2.1 interface, triggers warnings synchronously with the sound column (response time <50ms), and works with the V2X module to provide real-time route guidance for autonomous vehicles. The holographic projection module can be installed on a central pole at intersections, 4 meters high, with a projection angle covering the entire lane. Its basic parameters include 1920×1080 resolution, 7500 lumens brightness, and a projection distance of 5-15 meters.

[0089] The execution control module may also include omnidirectional speakers, which play customized voice warnings in real time and support multiple languages ​​(Chinese / English / sign language). It receives real-time voice commands via GPIO signals and connects with edge computing devices (response time <50ms), and plays reassuring voice messages when it detects pedestrian anxiety in conjunction with an emotion perception system. It is installed on poles on both sides of the zebra crossing, symmetrically arranged with the holographic projector. Its basic parameters include 60W power and a 50-meter radius coverage area (120° directional angle).

[0090] The execution control module may also include an LED variable information screen, which is installed in a prominent position at the intersection and connected to the edge computing unit through a network interface to display real-time traffic information (such as congestion level, accident location, etc.), traffic regulations, and real-time traffic guidance information, providing intuitive and timely information for traffic participants.

[0091] This embodiment constructs an "edge-cloud" collaborative architecture to achieve efficient linkage between roadside equipment, cloud platforms, and autonomous vehicles, supporting dynamic optimization of city-level traffic and reducing signal timing delays by more than 30%. It also integrates holographic projection and directional sound columns to achieve multimodal traffic warnings and route guidance, increasing the success rate of violation intervention to over 98%.

[0092] The following examples from different scenarios will further illustrate this application.

[0093] Example 1: Normal weather scenario.

[0094] Front-end perception layer: Millimeter-wave radar, 4K starlight-level camera and lidar collect traffic data in real time, and meteorological sensors detect that the current weather conditions are normal and send environmental parameters to the edge computing unit.

[0095] Edge computing layer: Domestically produced AI acceleration chips process data from multiple sensors in parallel; the YOLOv7 algorithm performs real-time detection of vehicles and pedestrians; and the ST-GCN algorithm predicts target trajectories. A dynamic timing algorithm sets the traffic light cycle to 60 seconds during off-peak hours based on real-time traffic data. A fault diagnosis system monitors the operating status of each sensor in real time to ensure normal system operation.

[0096] Cloud-based decision-making layer: The traffic brain platform receives data from various intersections, analyzes and processes it to generate global optimization strategies, reducing the frequency of warnings. The digital twin module simulates and predicts current traffic conditions, providing a reference for subsequent decision-making.

[0097] Execution control layer: A holographic projection system projects virtual arrows to guide vehicles along designated routes. Directional speakers play standard voice prompts, such as "Please obey traffic rules." LED variable message displays "Current road conditions are good," providing clear information to road users.

[0098] Example 2: Heavy rain weather scenario.

[0099] Front-end perception layer: When weather sensors detect heavy rain, they immediately send environmental parameters such as temperature, humidity, and visibility to the edge computing unit. Based on preset rules, the system automatically increases the weight of the millimeter-wave radar by 30% to enhance target detection capabilities.

[0100] Edge computing layer: The AI ​​acceleration chip prioritizes processing data from millimeter-wave radar, while simultaneously combining data from cameras and LiDAR for comprehensive analysis. Dynamic timing algorithms shorten green light duration to 15 seconds, reducing vehicle congestion time in flooded areas. The fault diagnosis system closely monitors the operational status of each sensor to ensure system stability under adverse weather conditions.

[0101] Cloud-based decision-making: Upon receiving information about heavy rain, the traffic brain platform immediately coordinates with the emergency management system to activate the flooded road section warning plan. The digital twin module simulates the impact of heavy rain on traffic, predicts potential congestion areas, and provides a basis for developing detour routes.

[0102] Execution Control Layer: A holographic projection system enhances the warning aura, flashing red to alert vehicles and pedestrians to safety. Directional speakers broadcast rainstorm safety warnings, such as "Flood ahead, please detour." LED variable message displays show detailed detour routes to guide vehicles safely.

[0103] Example 3: Peak hour scenario.

[0104] Front-end perception layer: During peak hours, each sensor continuously and in real time collects traffic data, including information such as traffic flow, vehicle speed, and queue length, and transmits this data to the edge computing unit in a timely manner.

[0105] Edge computing layer: The dynamic timing algorithm receives global traffic prediction information from the cloud-based traffic brain, combines it with local real-time data, extends the traffic light cycle to 180 seconds, and adds green light phases to improve the intersection's throughput. AI acceleration chips rapidly process large amounts of data to ensure real-time system response.

[0106] Cloud-based decision-making layer: The traffic brain platform analyzes traffic congestion hotspots and trends based on real-time and historical data from various intersections, generating more accurate global optimization strategies. The digital twin module simulates and predicts traffic conditions during peak hours in real time, providing decision support for traffic management departments.

[0107] Execution Control Layer: A holographic projection system displays a real-time traffic heatmap, intuitively showing the congestion situation on each road segment. Directional speaker columns announce "Currently peak hour, it is recommended to travel during off-peak hours," guiding road users to plan their travel time accordingly. LED variable message signs display a real-time congestion index, providing drivers with a reference.

[0108] Example 4: Emergency vehicle priority scenario.

[0109] Front-end perception layer: The V2X communication module receives request information from emergency vehicles such as ambulances and fire trucks. Millimeter-wave radar and cameras quickly track the emergency vehicles in real time, determining their location and direction of travel, and sending the relevant information to the edge computing unit. Simultaneously, lidar assists in precise positioning, providing more accurate 3D location information to ensure accurate tracking of emergency vehicles. Weather sensors continuously monitor environmental conditions, and if severe weather occurs, this information is transmitted for subsequent system consideration.

[0110] Edge computing layer: The dynamic timing algorithm dynamically plans green light routes for emergency vehicles based on their location and speed, reducing response time to less than 3 minutes. Domestically produced AI acceleration chips process multi-sensor data at high speed, ensuring not only real-time tracking of emergency vehicles but also consideration of surrounding normal vehicles and pedestrians. It combines historical traffic data and current real-time road conditions to predict the arrival time of emergency vehicles at various intersections and adjust traffic light timings in advance. For example, if there are many vehicles queuing at an intersection ahead of the emergency vehicle, the algorithm will appropriately extend the green light time for the emergency vehicle's direction at that intersection while shortening the green light times for other directions, ensuring the emergency vehicle can pass quickly. The fault diagnosis system will then strengthen the monitoring of various sensors and communication modules to prevent equipment failures from affecting dispatching in emergency situations.

[0111] Cloud-based decision-making layer: After receiving emergency vehicle information from the edge computing units, the traffic brain platform quickly analyzes the traffic conditions of the entire city's road network. It considers factors such as the capacity of surrounding roads and the presence of other traffic incidents, globally optimizing the routes planned by the edge computing layer. If it detects that severe congestion is imminent on the emergency vehicle's original planned route, the traffic brain platform will promptly adjust the route, sending the new route information to the emergency vehicle and the edge computing units at relevant intersections via the V2X communication module. The digital twin module simulates the emergency vehicle's driving on the adjusted route in real time, predicts potential problems, and feeds the simulation results back to the traffic brain platform for further decision-making.

[0112] Execution Control Layer: The holographic projection system projects dedicated lane arrows to create a special passage for emergency vehicles, with errors controlled to a minimum to ensure emergency vehicle drivers can clearly see the guidance. These arrows dynamically adjust as emergency vehicles move, always pointing in the correct direction. Directional speakers broadcast clear and unambiguous voice prompts such as "Please give way to ambulances," using high volume and directional propagation to ensure surrounding vehicles and pedestrians can hear them clearly. Simultaneously, LED variable message displays show the location and estimated arrival time of emergency vehicles, reminding surrounding vehicles to prepare to give way. After the emergency vehicle has passed, the holographic projection system stops projecting the dedicated lane arrows, the directional speakers stop playing the prompts, the LED variable message displays resume normal traffic information, and the entire traffic system returns to normal operation.

[0113] The dynamic timing algorithm of the aforementioned edge computing layer and the adjustment of the millimeter-wave radar weights according to weather scenarios can be processed and analyzed using domestically produced AI chips, such as GPUs and CPUs.

[0114] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0115] (1) Improved accuracy of multimodal perception: Through the deep fusion of millimeter-wave radar, 4K starlight-level camera and lidar, the target recognition accuracy is improved from below 70% of the traditional system to above 99.5% under adverse weather conditions such as rain and fog, which is an improvement of more than 20%. The spatiotemporal alignment error is reduced from more than 0.3 seconds to ≤10ms, the trajectory prediction accuracy is improved by 35%, and the traffic accident rate is effectively reduced by 15%.

[0116] (2) Real-time decision-making and low power consumption: The domestically produced AI acceleration chip achieves low-power operation, reducing power consumption from 25W of traditional GPUs to below 15W. At the same time, the edge computing latency is shortened from more than 150ms to less than 10ms, and the target detection speed reaches 120FPS, significantly improving the system's real-time decision-making capability. The application of dynamic timing algorithm reduces the average delay time by more than 30%, and improves the passage efficiency by 3 times compared with traditional systems.

[0117] (3) City-level collaborative optimization: The "edge-cloud" collaborative architecture enables efficient linkage between roadside equipment, cloud platforms, and autonomous vehicles. The traffic brain platform achieves accurate prediction of city-level traffic flow through federated learning, reducing emergency response time from more than 3 seconds to less than 50ms. In extreme weather conditions such as heavy rain, the system automatically activates warning plans for flooded road sections, improving the traffic efficiency of relevant road sections by 40%.

[0118] (4) Multimodal warnings and guidance: The integration of holographic projection and directional sound columns enables multimodal traffic warnings and route guidance. The success rate of violation intervention has increased from about 60% with traditional electronic police to over 98%, an increase of 40%. Real-time route guidance provided for autonomous vehicles has improved vehicle traffic efficiency by 25%.

[0119] (5) Economic benefits and localization: The localization rate of the system reaches 100%, and the cost of a single unit is reduced by 60% compared with the traditional system using imported chips. The energy efficiency ratio is improved by 40%, which can save 50% of electricity costs every year, while reducing carbon emissions by more than 1,000 tons, which has significant economic and environmental benefits.

[0120] (6) Innovative Functional Advantages: The emotion perception system analyzes pedestrian facial expressions and plays personalized voice prompts, enhancing the safety awareness of traffic participants. The V2X communication module enables real-time interaction between roadside equipment and autonomous vehicles, optimizing vehicle traffic efficiency and reducing human error. The LED variable message display screen publishes real-time traffic information, guiding vehicles to take reasonable detours and effectively alleviating traffic congestion.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A traffic control system based on multimodal perception, characterized in that, The traffic control system includes: A bracket is installed on top of the intersection pole, and the bracket is equipped with an installation platform; A radar monitoring module is installed on the mounting platform, and the radar monitoring module includes millimeter-wave radar and lidar. An angle adjustment component is disposed on the mounting platform, and the angle adjustment component is used to adjust the angle between the main axis of the lidar and the main axis of the millimeter-wave radar. A meteorological monitoring module is installed on the mounting platform, and the meteorological monitoring module includes at least one of a temperature sensor, a humidity sensor, a visibility sensor, and a barometric pressure sensor; A visual monitoring module is installed on the outer wall of the intersection pole, and the visual monitoring module includes a camera.

2. The traffic control system according to claim 1, characterized in that, The traffic control system also includes a switch and an edge computing device that is communicatively connected to the switch; The millimeter-wave radar is communicatively connected to the switch via a first data transmission cable; The lidar is connected to the switch via a second data transmission cable. The meteorological monitoring module is connected to the switch via a third data transmission cable. The camera is connected to the switch via a fourth data transmission cable.

3. The traffic control system according to claim 2, characterized in that, The edge computing device is equipped with a hardware trigger synchronization module, which includes a synchronization pulse generator and multiple synchronization signal output ports connected to the synchronization pulse generator; the millimeter-wave radar, the lidar, and the camera are all equipped with synchronization trigger pins, and multiple synchronization trigger pins are connected to multiple synchronization signal output ports through multiple signal cables; The synchronization pulse generator is configured to simultaneously send trigger signals to the millimeter-wave radar, the lidar, and the camera through multiple synchronization signal output ports, and the millimeter-wave radar, lidar, and camera are configured to collect traffic data in response to the trigger.

4. The traffic control system according to claim 2, characterized in that, The switch and the edge computing device are installed in a junction cabinet adjacent to the junction pole.

5. The traffic control system according to claim 2, characterized in that, The traffic control system also includes a cloud-based decision-making module and an execution control module; The cloud-based decision-making module is connected to the edge computing device and is configured to receive meteorological anomaly signals from the edge computing device, triggering the emergency equipment activation of the execution control module. The emergency equipment of the execution control module includes a holographic projection module and a directional speaker. The intersection pole where the holographic projection module is located and the intersection pole where the directional speaker is located are set at intervals. The holographic projection module is configured to project virtual arrows, and the directional speaker is configured to play voice warnings.

6. The traffic control system according to claim 5, characterized in that, The execution control module also includes an LED information screen, which is set at the intersection. The LED information screen is connected to the edge computing device via a network interface and is configured to receive intersection traffic data from the edge computing device to display intersection traffic information.

7. The traffic control system according to claim 1, characterized in that, The angle between the main axis of the lidar and the main axis of the millimeter-wave radar is 5 degrees to 25 degrees.

8. The traffic control system according to claim 7, characterized in that, The main axis of the lidar is aligned with the main axis of the camera.

9. The traffic control system according to claim 8, characterized in that, The installation platform includes a first platform and a second platform, with the first platform located above the second platform; the meteorological monitoring module is installed on the first platform, and the radar monitoring module is installed on the second platform.

10. The traffic control system according to claim 9, characterized in that, The lidar is also equipped with a connecting block, and the angle adjustment component includes a drive motor. The drive motor is mounted on the second platform, and the motor shaft of the drive motor extends out of the second platform and is fixedly connected to the connecting block.

Images