A system for real-time cooperation between multiple vehicles using an edge-fog composite perception network.

Abstract

A system for real-time cooperation between multiple vehicles using an edge-fog composite perception network, comprising: an edge layer with a multitude of vehicles, each vehicle having one or more sensors and an onboard processing unit configured to generate local perception data and abstract the local perception data into a spatiotemporal perception vector; a fog plane with one or more fog nodes that are communicatively coupled to the multitude of vehicles, each fog node being configured to receive spatiotemporal perception vectors from multiple vehicles and perform federated fusion to generate a local dynamic environment model; and a cloud layer with at least one cloud server configured to coordinate a federated learning process by aggregating model updates received from the fog nodes to generate an updated global perception model, the updated global perception model is distributed to the vehicles via the fog nodes to enable cooperative perception beyond the detection range of a single vehicle.

Description

[0001] The present invention relates generally to the field of intelligent transportation systems and connected and autonomous vehicles (CAVs). In particular, the invention relates to a real-time collaborative perception system for multiple vehicles that uses a hierarchical edge fog cloud computing architecture integrated with federated learning techniques.

[0002] Autonomous and semi-autonomous vehicles rely heavily on onboard perception systems that use sensors such as cameras, LiDAR, radar, and ultrasonic devices to detect, classify, and track objects in their environment. These perception systems enable critical functions such as lane keeping, adaptive cruise control, collision avoidance, and automated navigation. However, the perception capability of a single vehicle is inherently limited by sensor range, obstructions to visibility, adverse weather conditions, and line-of-sight restrictions. This egocentric perception model often fails to detect critical objects located outside the immediate detection range, such as vehicles emerging from blind intersections, pedestrians obscured by buildings, or hazards hidden behind other vehicles.To mitigate these limitations, existing vehicle communication technologies such as Dedicated Short-Range Communications (DSRC) and Cellular Vehicle-to-Everything (C-V2X) enable the exchange of Basic Safety Messages (BSMs) between vehicles. These messages typically transmit low-level kinematic information, including position, speed, acceleration, and direction of travel. While this information improves situational awareness, it lacks a rich semantic and perceptual context, making it insufficient for advanced cooperative perception and coordinated decision-making between multiple vehicles. More advanced approaches propose exchanging raw or partially processed sensor data, such as camera images or LiDAR point clouds, between vehicles or with a central cloud server.However, these approaches are unsuitable for real-time deployment due to excessive bandwidth requirements, high transmission latency, and scalability limitations, particularly in dense urban environments. Furthermore, the transmission of raw sensor data raises significant privacy and data security concerns. Centralized, cloud-based perception and decision-making systems have also been investigated, where vehicles transmit sensor data to a remote server for aggregation and processing. Such systems suffer from unacceptable round-trip latency, often exceeding the tolerances required for safety-critical applications such as collision avoidance and cooperative maneuvering. Moreover, the reliance on a continuous connection to a remote cloud infrastructure reduces the system's robustness and reliability.Recent developments in edge and fog computing aim to process data closer to its source, thereby reducing latency and network congestion. While these architectures improve responsiveness, existing solutions do not effectively integrate cooperative perception with privacy-protecting mechanisms for learning across multiple vehicles and infrastructure nodes. Similarly, federated learning techniques have been proposed to train distributed models without centralizing raw data; however, their application is largely limited to isolated edge devices and has not been systematically combined with hierarchical vehicle perception and real-time cooperation.

[0003] Accordingly, there is a need for a scalable, low-latency, and privacy-compliant cooperative perception system that enables multiple vehicles to exchange and fuse perceptual information beyond egocentric sensor limitations, while efficiently utilizing communication and computing resources. The present invention addresses these challenges by introducing a hierarchical edge-fog federated perception network that enables real-time cooperation between multiple vehicles without the exchange of raw sensor data.

[0004] To solve this problem, the present invention offers a system for real-time cooperation of multiple vehicles using an edge-fog composite perception network.

[0005] The system enables vehicles to perceive objects, obstacles, and events outside their individual sensor range and line of sight through cooperative information exchange.

[0006] The system minimizes end-to-end latency while supporting scalable collaboration between multiple vehicles.

[0007] The system reduces communication bandwidth requirements by abstracting raw sensor data into compact spatiotemporal perception vectors suitable for low-latency vehicle communication.

[0008] The system facilitates the federated fusion of perception data at fog nodes to create a unified local environmental model without the need to exchange raw sensor data.

[0009] The system implements a federated learning mechanism across fog nodes to update a global perception model in a privacy-compliant manner.

[0010] The system ensures data protection and security by preventing the transmission of sensitive raw sensor data to other vehicles or centralized servers.

[0011] The system is compatible with existing vehicle communication standards such as DSRC and C-V2X and can be deployed using road infrastructure such as roadside devices and base stations.

[0012] The present invention provides a system for cooperative real-time perception between multiple vehicles using a hierarchical edge-fog federated perception network. The invention overcomes the limitations of egocentric vehicle sensing by enabling vehicles to cooperatively perceive objects, obstacles, and environmental conditions beyond their individual sensor range and line of sight. According to one aspect of the invention, the system comprises an edge layer with a plurality of vehicles, each vehicle being equipped with one or more sensors and an onboard processing unit configured to generate local perception data. The local perception data is abstracted into a compact space-time perception vector (SPV) representing detected objects, their attributes, and temporal information.The SPVs are transmitted to one or more fog nodes via vehicle communication links. According to another aspect of the invention, the fog plane comprises a plurality of geographically distributed fog nodes, each fog node being configured to receive SPVs from multiple vehicles within a coverage area. Each fog node performs a federated fusion of the received SPVs to generate a local dynamic occupancy grid or a unified environmental model without requiring access to raw sensor data. The fog nodes also compute local updates of a perception model based on the fused perception data.

[0013] According to another aspect of the invention, a cloud layer is provided to coordinate a federated learning process between the fog nodes. The cloud layer aggregates the model updates received from the fog nodes to generate an updated global perception model without centralizing the raw perception data. The updated global perception model is distributed to the fog nodes and subsequently to the vehicles to enhance their onboard perception capabilities. The hierarchical edge fog cloud architecture enables cooperative perception with low latency, efficient bandwidth utilization, and privacy protection, making it suitable for safety-critical vehicle applications.By combining compact perceptual abstraction, federated fusion at the fog level, and federated learning across infrastructure nodes, the invention enables real-time cooperation between multiple vehicles, improved situational awareness, and scalable deployment in connected and autonomous vehicle environments.

[0014] The present invention relates to a system for enabling cooperative real-time perception between multiple vehicles using a hierarchical edge-fog composite perception network. The system was developed to overcome the limitations of egocentric vehicle perception by enabling distributed perception, collaborative information fusion, and privacy-compliant modeling across vehicles and infrastructure nodes. The system comprises an edge layer with a multitude of vehicles, each acting as an intelligent edge computing unit. Each vehicle is equipped with one or more sensors, including, but not limited to, LiDAR, radar, and cameras, configured to capture raw environmental data.Each vehicle also features an integrated processing unit configured to run a perception model that processes raw sensor data to detect and classify objects in the environment. The perception model's output is abstracted into a compact spatiotemporal perception vector representing detected objects, spatial positions, velocities, temporal attributes, and associated uncertainty information. The spatiotemporal perception vector is transmitted from the vehicle to a fog node via a vehicle-to-everything communication interface, reducing bandwidth consumption and avoiding the transmission of raw sensor data.

[0015] The system further comprises a fog layer with one or more geographically distributed fog nodes deployed along the road infrastructure. Each fog node includes a communication module configured to receive spatiotemporal perception vectors from multiple vehicles operating within its coverage area. The fog node also includes a federated fusion machine configured to combine the received perception vectors to generate a local dynamic occupancy grid, or environmental map, which provides a unified view of the surroundings. The local dynamic map includes both static and dynamic objects and provides perception information that extends beyond the detection range and line of sight of individual vehicles.Each fog node further includes a local model update module configured to compute local updates to a perceptual model based on the fused perceptual data. The system also includes a cloud layer with one or more cloud servers configured to coordinate a federated learning process between the fog nodes. The cloud server includes a federated learning orchestrator configured to initiate federated learning rounds and distribute a global perceptual model to selected fog nodes. Each participating fog node trains the received global model locally using its local dynamic map and transmits only the resulting model update parameters to the cloud server.The cloud server aggregates received model updates using federated learning techniques to generate an updated global perception model without accessing raw sensor data or local perception vectors. In operation, each vehicle continuously generates local perception data from onboard sensors and converts this data into spatiotemporal perception vectors. These spatiotemporal perception vectors are transmitted to a local fog node, which receives vectors from multiple vehicles and performs federated fusion to create a local dynamic occupancy grid. The fog node computes a local model update based on the fused perception data and participates in a federated learning process coordinated by the cloud server. The cloud server aggregates the model updates received from multiple fog nodes and updates the global perception model.The updated global perception model is transmitted back to the fog nodes and then distributed to the vehicles. Each vehicle integrates the updated global perception model into its onboard perception system, thereby improving perception accuracy and enabling cooperative, real-time perception beyond its individual sensor capabilities. The hierarchical edge-fog-cloud architecture of the present invention enables low-latency processing at the vehicle and fog levels while leveraging cloud resources for scalable and continuous model optimization. By abstracting raw sensor data into compact perception vectors and employing federated learning across fog nodes, the system ensures efficient bandwidth utilization, improved perception robustness, and data privacy protection.The invention is applicable to a variety of intelligent traffic scenarios, including urban intersections, highways, tunnels and cooperative driving environments.

Claims

[1] A system for real-time cooperation between multiple vehicles using an edge-fog composite perception network, comprising: an edge layer with a multitude of vehicles, each vehicle having one or more sensors and an onboard processing unit configured to generate local perception data and abstract the local perception data into a spatiotemporal perception vector; a fog plane with one or more fog nodes that are communicatively coupled to the multitude of vehicles, each fog node being configured to receive spatiotemporal perception vectors from multiple vehicles and perform federated fusion to generate a local dynamic environment model; and a cloud layer with at least one cloud server configured to coordinate a federated learning process by aggregating model updates received from the fog nodes to generate an updated global perception model, the updated global perception model is distributed to the vehicles via the fog nodes to enable cooperative perception beyond the detection range of a single vehicle. [2] System according to claim 1, wherein the one or more sensors comprise at least one LiDAR sensor, one radar sensor and one camera. [3] System according to claim 1, wherein the space-time perception vector represents abstracted information, including object position, velocity, classification, temporal attributes and uncertainty. [4] System according to claim 1, wherein the spatial-temporal perception vector is transmitted using Vehicle-to-Everything (V2X) communication. [5] System according to claim 1, wherein the federated fusion performed by the Fog node generates a local dynamic occupancy grid that represents static and dynamic objects in an environment. [6] System according to claim 1, wherein the fog node computes a local model update based on the local dynamic environment model without accessing raw sensor data from the vehicles. [7] System according to claim 1, wherein the cloud server aggregates the model updates using a federated learning technique without centralizing the raw perception data. [8] System according to claim 1, wherein the updated global perception model improves perception accuracy and situational awareness for safety-critical vehicle applications.