Method for automated generation of safety messages for road incident management using infrastructure sensors

Abstract

Vehicle-to-Everything (V2X) safety messages are generated automatically using multi-modal environmental data from infrastructure sensors (e.g. roadside infrastructure-mounted cameras, radar and lidar)

Description

Background of the invention This invention pertains to the field of Intelligent Transportation Systems (ITS), specifically utilizing advanced artificial intelligence methodologies to enhance the operational capabilities of Connected and Automated Vehicles (CAVs) or Connected Vehicles (CVs). The evolution of vehicular sensors has facilitated significant advancements in Advanced Driver Assistance Systems (ADAS) and Automated Driving (AD), largely supported by developments in Artificial Intelligence (Al) and Large Language Models (LLMs). Despite these advancements, current Automated Driving Systems (ADS) face significant challenges, particularly in dynamic and adverse traffic conditions. One major limitation is the ability of these systems to accurately detect hazards, plan optimal routes, and make informed decisions amidst variables such as lane closures or road damage. The perception sensors, while sophisticated, are prone to degradation under adverse weather conditions, affecting their functionality and the reliability of Global Navigation Satellite Systems (GNSS). Furthermore, Vehicle-to-Everything (V2X) communication has become indispensable for achieving higher autonomy levels (SAE Levels 4 and 5) by enabling CAVs to receive real-time routing and driving suggestions. Yet, existing systems lack efficiency in traffic hazard management and require substantial human intervention for analyzing traffic scenes and managing responses to incidents. This gap necessitates a large workforce and introduces delays in communication between traffic experts and software systems responsible for generating critical traffic management messages. The current landscape of ITS requires enhancements in the integration of technology and traffic management practices. The dependency on human traffic experts for recognizing and responding to traffic conditions and the subsequent communication to software programmers for message generation introduces inefficiencies. Additionally, the inability of ADS to autonomously navigate through maintenance or hazard zones without explicit human-generated communication underscores the need for an improved solution. In response to these challenges, the proposed invention introduces an innovative approach to integrate Multimodal Large Language Models (MLLMs) directly with CAV operations. The novel pipeline developed as part of this invention can automatedly analyze the traffic conditions, identify potential hazards, and generate standard communication messages. This system enhances the decision-making process of CAVs, reduces the reliance on human intervention, and improves the overall safety and efficiency of ITS. State of the Art Despite the advancements in demonstration projects worldwide, a substantial reliance on human intervention persists, particularly in monitoring traffic conditions and managing road safety incidents. Traffic experts are required to analyze situations and collaborate with software developers to generate the necessary V2X messages, which are then disseminated through RSUs. This workflow is not only time-consuming but also lacks the responsiveness needed to address unpredictable incidents such as road debris or accidents efficiently. The advent of MLLMs, which integrate textual, visual, audio, and video data, has demonstrated significant potential in enhancing various aspects of automated driving. According to a recent survey by Cui et al. (2024), MLLMs enhance driving decision-making, navigation, safety, and efficiency by analyzing traffic scenes, optimizing vehicle planning, and dynamically adapting to changing road conditions. These models also personalize driving experiences and in-vehicle entertainment, enhancing passenger trust in autonomous technologies and providing clear explanations of autonomous actions. A recent study by Wang et al. (2023) explored the application of a multi-modal large language model for accident analysis and prevention. This research focused on accident data analysis but did not address the generation of safety messages. Yunex Traffic has proposed an Al-enhanced ITS application, which optimizes traffic flow, improves road safety, and enhances autonomous driving capabilities, yet it does not incorporate LLMs or MLLMs (https: / / www.yunextraffic.com / wp-content / upioads / 2023 / 05 / Yunex-Tr3ffic awareAl EN.pdf). In US20220332350A1, the invention enhances vehicular safety with Maneuver Coordination Services (MCS) that enable collective emergency actions through Emergency Group Maneuver Coordination (EGMC) for unexpected road situations. It specifies the Maneuver Coordination Message (MCM) format and efficient communication protocols to minimize overhead, although it does not detail how to generate the MCM or consider the latest Al developments. US10867512B2 provides systems and methods for an Intelligent Road Infrastructure System (IRIS), facilitating vehicle operations and control for connected automated vehicle highway systems. This invention delivers customized information and real-time control instructions to vehicles, yet it lacks capabilities for automated message generation based on situational data. In US 20200388154A1, the invention offers services based on traffic information obtained by an Al module but does not employ MLLMs for interpreting the traffic scene, nor can it detect undefined incident types. US20220014963A1 focuses on managing multi-access traffic in edge computing environments using Al and ML techniques. It introduces a scalable AI / ML architecture with reinforcement learning (RL) and Deep RL (DRL) fortraffic management and incorporates Deep contextual bandit RL techniques for edge networks. However, this system does not consider real-time safety message generation or utilize LLMs or MLLMs. Description of the Invention The invention presents a method for the automated generation of standardized safety messages to inform oncoming traffic of potential dangers and / or incidents on the road ahead. This method primarily utilizes infrastructure sensors, including at least a camera, but may also incorporate additional sensor modalities such as radar and lidar. The data collected from these sensors is processed and transmitted to a machine learning language model (MLLM) or an artificial intelligence (Al) system. The MLLM / AI system provides metadata encompassing various details, such as the type of incident, its severity, priority, associated objects, and geographic location. Although the current invention focuses on the use of static sensors, it can be extended to include mobile sensor sources, such as drones and other vehicles on the road, which may collaboratively transmit incident information. The MLLM / AI system can also recommend traffic management measures, including new speed limits, lane closures, and in-lane offsets. The extracted information is then converted into a standardized safety message format. Utilizing a standard message template, the invention creates a V2X message from the interpreted and extracted information from the incident. An optional step allows a human operator to verify the safety message for consistency, integrity and security before it is broadcasted from the associated RSU. After all checks are completed, the V2X message is broadcasted via RSU. This invention significantly reduces response times for the generation of safety-oriented V2X messages in response to road incidents and can theoretically run in soft real-time. This can significantly enhance the safety impact of such V2X messages. The invention presents a method for automated generation of standardized V2X messages involving the following steps: Step 1: Processing multi-modal environmental sensor data Step 2: Transmission of processed / percieved sensor data to a purpose-designed MLLM / AI Step 3: Extraction of meta data about the incident, environment and location Step 4: Transforming the extracted information into a standard V2X message utilizing corresponding templates for each message type Step 5 (optional): This generated safety message is verified by a human operator for consistency, integrity, and security Step 6: Broadcasting the safety message For a better understanding of the invention below figures are given. The figures below are defined as follows: Figure 1: Use Case of the invention which includes environmental sensors, AI / MLLM cloud service, RSU and CVs or CAVs Figure 2: The workflow of the invention for automated generation of safety message for road incident management using infrastructure sensors Figure 3: The data transmission between components of this invention, including environmental sensors, infrastructure cloud incorporating MLLM / AI pipeline, road operator and RSU

Claims

1. A method for automatically generating standardised V2X safety messages for connected vehicles, the method comprisingin a first step the processing of multi-modal environmental data from infrastructure sensors, in a second step the transmission of processed / perceived sensor data to a purpose designed MLLM / AI,in a third step the extraction of meta data about the incident, environment and location,in a fourth step the transformation of the extracted information into a standard V2X message utilizing corresponding templates for each message type, in a fifth step the broadcasting of the safety message.

2. A method according to Claim 1, comprising in an additional step between the fourth and the fifth step the verification of the generated safety message by a human operator for consistency, integrity and security.

3. A method according to Claims 1 and 2, comprising that infrastructure-mounted perception sensors are used to detect hazards certain incidents and unexpected objects on the road.

4. A method according to Claims 1 to 3, comprising the processing of raw perception sensor data utilizing a customized MLLM / AI to generate certain classes of extracted meta data involving a process flow.

5. A method according to Claims 1 to 4, which uses a message template customized with the structure and semantic content of the related V2X message, that should be populated by the extracted metadata.

6. A method according to Claims 1 to 5, comprising the fine tuning or training of the MLLM to generate the standardized V2X message according to a message template.

Images