An emergency vehicle global coordination priority control system and method based on vehicle-road cooperation and multi-agent reinforcement learning

Abstract

The application provides an emergency vehicle global coordination priority control system and method based on vehicle-road cooperation and multi-agent reinforcement learning. The system comprises a cloud and a plurality of agents deployed at intersections. When an emergency task is triggered, the cloud calculates an optimal path of the emergency vehicle based on real-time road network traffic flow data, generates a path node sequence and a path intention, and issues the path node sequence and the path intention to the corresponding agent. The agent determines a role label according to the path intention and the location of the intersection, constructs a state vector, and inputs the state vector into a local branch policy network to generate a hybrid action to control the signal lamp of the intersection and the network-connected vehicle. After the hybrid action is executed, a differentiated immediate reward is calculated, and the state vector, the hybrid action and the differentiated immediate reward constitute interaction experience. The cloud samples experience data from a central experience replay pool, updates the parameters of the policy network, and issues the parameters to each agent to update the local branch policy network. The application can realize global element coordination, has a forward-looking decision-making capability, and can finely adjust and control space-time resources.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation systems, specifically relating to an emergency vehicle global collaborative priority control system and method based on vehicle-road cooperation and multi-agent reinforcement learning. Background Technology

[0002] As urban traffic congestion worsens, the passage efficiency of emergency vehicles such as fire trucks and ambulances faces severe challenges. Traditional control methods that rely on sirens, passive yielding by other vehicles, and simple signal priority (such as extended green lights or phase insertion) are inefficient, unsafe, and prone to causing secondary congestion.

[0003] Existing advanced research mainly focuses on two independent levels: (1) Road segment cooperative traffic strategy: This approach focuses on the road segment where emergency vehicles are located, aiming to dynamically coordinate the driving trajectory of social vehicles through vehicle-to-vehicle communication and vehicle-to-infrastructure communication, using models such as cooperative lane changing and speed guidance, to create a continuous "virtual emergency lane" for emergency vehicles within the existing physical lanes. This type of research aims to reduce the travel delay of emergency vehicles by optimizing the speed and position of social vehicles; (2) Intersection signal priority strategy: This approach focuses on one or more related intersections, using the position and speed information of emergency vehicles obtained by detectors, and dynamically adjusting the phase, timing and period of traffic lights through algorithms based on preset rules or adaptive control models based on reinforcement learning (RL), to provide a green light window for emergency vehicles in the target direction.

[0004] Despite the positive progress made in the above research, the existing methods still have the following shortcomings in practical applications: (1) Segment control and intersection control are independent of each other. Existing systems either focus on vehicle guidance on the segment or on signal optimization at the intersection, and the two lack deep integration in terms of decision-making timing, control objectives, and execution mechanisms. For example, the "virtual emergency lane" strategy may successfully guide vehicles to give way on the segment, but if the signals at the downstream intersection are not coordinated and matched, the emergency vehicle will still encounter a red light queue at the intersection. (2) Vehicle guidance and signal control are independent of each other. At the control decision level, vehicle speed guidance and traffic signal timing are usually modeled as two independent or serial optimization problems, which are handled by different models or algorithms. This independence leads to the possibility that control commands may not be coordinated or even conflict, making it impossible to achieve integrated and fine-grained scheduling of spatiotemporal resources. (3) Single-point local optimization and network global optimization are independent of each other. Most signal control research based on reinforcement learning takes a single intersection as the agent, and its optimization objective is limited to the traffic indicators of the intersection. In multi-intersection scenarios, it is easy to get trapped in local optima and fail to achieve the optimal network global benefits. Although some explorations have been made in multi-agent reinforcement learning (MARL), in emergency vehicle priority scenarios, how to effectively decompose the global path intent of emergency vehicles and transform it into decision signals that drive each intersection agent to conduct forward-looking and differentiated collaboration remains a problem that has not been fully solved in existing research.

[0005] Therefore, there is an urgent need for a systematic solution that can achieve synergy of all elements, has forward-looking decision-making capabilities, and can carry out refined joint regulation and control of spatiotemporal resources. Summary of the Invention

[0006] The technical problem to be solved by this invention: (1) Lack of integrated coordination between road segment and intersection control: The vehicle route guidance and the intersection signal control lack coordination in execution timing and optimization objectives, making it impossible to achieve continuous priority passage of emergency vehicles from "road segment to intersection".

[0007] (2) Lack of coordination between vehicle and signal control commands: The continuous speed guidance command for connected vehicles and the discrete phase and continuous duration control command for traffic lights are generated by independent models, which makes it impossible to make unified decisions and fine allocation of road spatiotemporal resources.

[0008] (3) Lack of coordination between local decision-making and global path: The decision-making model with a single point or local area as the optimization target does not meet the global passage path and dynamic needs of emergency vehicles, and cannot actively and dynamically plan and coordinate the traffic flow of the road network based on global path information.

[0009] To address the aforementioned problems in existing technologies, this paper provides a comprehensive collaborative priority control system and method for emergency vehicles based on vehicle-road cooperation and multi-agent reinforcement learning. This system enables comprehensive element collaboration, possesses forward-looking decision-making capabilities, and allows for refined joint regulation of spatiotemporal resources.

[0010] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for full-domain cooperative priority control of emergency vehicles based on vehicle-road cooperation and multi-agent reinforcement learning is disclosed. The method is applied to a full-domain cooperative priority control system for emergency vehicles, which includes a cloud platform and multiple agents. Each agent corresponds one-to-one with an intersection in the road network and is deployed at the corresponding intersection. The method includes the following steps: The cloud acquires real-time road network traffic flow data, and each intelligent agent acquires traffic flow data at its respective intersection and extracts features according to a specified period. When an emergency task is triggered, the cloud uses real-time acquired road network traffic flow data and a distributed path planning algorithm to calculate the optimal route for emergency vehicles, generate a path node sequence and a path intent containing the estimated arrival time and the next target node, and send the path intent to the intelligent agents corresponding to each node in the path node sequence. The agent that receives the route intent determines its own role label based on the relationship between the received route intent and the location of the intersection, constructs a state vector containing the feature extraction results and its own label, and then inputs the state vector into the local branch policy network to generate a hybrid action containing traffic light phase, green light duration and recommended speed of connected vehicles, so as to control the traffic lights and connected vehicles at the intersection. After the hybrid action is executed, the cloud or agent calculates the differentiated immediate reward, including the basic reward, the current domain passage reward, and the next target domain clearing reward, based on the agent's role label. The differentiated immediate reward, along with the corresponding state vector and hybrid action, is combined to form the interaction experience and stored in the central experience replay pool. The cloud samples experience data from the central experience replay pool and updates the parameters of the policy network through the gradient method. The updated parameters are then distributed to each agent to update the local branch policy network of each agent.

[0011] Furthermore, when each agent acquires traffic flow data at its respective intersection and performs feature extraction at a specified period, for any given intersection agent... In each control cycle Within this scope, the following feature extraction is performed: For each entrance lane, count the number of vehicles stationary within the designated detection area upstream of the parking line to obtain the queue length. ; For each entrance lane, estimate the average dwell time of all queuing vehicles in that lane within the detection area to obtain the average waiting time. ; If an emergency vehicle exists within the control domain, obtain the real-time distance from the emergency vehicle to the intersection stop line. and instantaneous speed And define the existence status. If no emergency vehicle exists within the control domain, then set , , 0, Indicates the maximum sensing distance; Calculate the time occupancy of downstream critical road segments specified by the route intent. The mathematical expression is as follows:

[0012] in, For the statistical period, This represents the total time that the detection section is occupied by vehicles within this cycle.

[0013] Furthermore, when calculating the optimal route for emergency vehicles using distributed path planning algorithms based on real-time acquired road network traffic flow data in the cloud, specifically for each node in the road network, the optimal estimated time from that node to the destination of the emergency vehicle and the optimal adjacent nodes after the vehicle leaves that node are maintained; each node periodically updates its own optimal estimated time and optimal adjacent nodes based on real-time road segment travel time and the optimal estimated time of adjacent nodes; starting from the node where the emergency vehicle is currently located, the optimal path node sequence is obtained by tracing along the optimal adjacent nodes of each node; the optimal estimated time of the node where the emergency vehicle is currently located is used as the expected arrival time, and the optimal adjacent node information of the node where the emergency vehicle is currently located is used as the next target node.

[0014] Furthermore, when each node periodically updates its own optimal estimated time and optimal neighboring node based on the real-time road segment travel time and the optimal estimated time of adjacent nodes, the process includes the following steps: Current node From all adjacent nodes The optimal estimated time to receive its latest message Based on real-time traffic flow data, estimate the distance from adjacent nodes To the current node Travel time of the route The mathematical expression is as follows:

[0015] in, For the length of the road segment, This represents the average speed of vehicles on that road section. The optimal value of local traffic conditions is calculated based on the travel time of the aforementioned road segment, and the mathematical expression is as follows:

[0016]

[0017] in, For the optimal estimated time, The optimal neighboring node. Indicates at time The emergency vehicle is expected to reach its upstream neighboring node. The earliest time, Indicates at time From the upstream node Drive to the current node The estimated travel time.

[0018] Furthermore, when an agent receiving a route intent determines its own role label based on the relationship between the received route intent and the location of the intersection, specifically, the agent determines its own role label based on whether the intersection contains emergency vehicles and the relationship between the node corresponding to the intersection and the next target node, including: If the traffic flow data at the intersection where the agent is located includes data on emergency vehicles, the agent determines its role label as the current domain. If the traffic flow data at the intersection where the agent is located does not include data on emergency vehicles, and the node corresponding to the intersection is the same as the next target node, the agent determines its role label as the next target domain. If the traffic flow data at the intersection where the agent is located does not include data on emergency vehicles, and the node corresponding to the intersection is different from the next target node, the agent determines its role label as another domain. The mathematical expression for the role label is as follows:

[0019] In the formula, This represents the role label of the i-th agent.

[0020] Furthermore, when constructing a state vector containing the feature extraction results and its own label, the state vector includes signal state, traffic flow state, emergency vehicle state, and coordination information, expressed as follows:

[0021] in, The code representing the current signal phase. These represent the average queue lengths at the four entrances: east, south, west, and north. These represent the average waiting times for the four entrances: east, south, west, and north. A binary variable indicating whether an emergency vehicle exists within the control area of ​​this intersection; This indicates the real-time distance between the emergency vehicle and the stop line at this intersection; Indicates the real-time speed of the emergency vehicle; Indicates the role label of the intelligent agent; This indicates the time occupancy rate of key downstream road sections.

[0022] Furthermore, when the state vector is input into the local branch policy network to generate a hybrid action containing traffic light phase, green light duration, and recommended speed of connected vehicles, the local branch policy network extracts features from the state vector and processes the extracted features through a phase selection head, a green light duration head, and a vehicle guidance head, respectively, and finally outputs a hybrid action containing traffic light phase, green light duration, and recommended speed of connected vehicles, including the following steps: Extracting state vectors through one or more fully connected layers High-level feature representation The mathematical expression is as follows:

[0023] in, Let represent the state vector of the i-th agent. To share the feature extraction function, To share the parameters of the feature extraction function; High-level feature representation The input phase selection head represents the high-level features. The original score vector is obtained through processing by the first fully connected layer. After being transformed into a probability distribution for each phase using a probability distribution function, the phase with the highest probability is selected as the traffic light phase using a greedy strategy. The mathematical expression is as follows:

[0024]

[0025] in, The probability distribution represents four phases. This indicates the output action of the phase selector; High-level feature representation Input the green light duration header, and the green light duration header will represent the high-level features. The scalar is obtained through processing in the second fully connected layer. The green light duration ratio coefficient is obtained through nonlinear mapping, and then converted into the actual green light duration through linear mapping. The mathematical expression is as follows:

[0026]

[0027]

[0028] in, This is a normalized green light duration ratio coefficient. These are the preset minimum and maximum green light durations, respectively. This represents the weight matrix corresponding to the green light duration. This indicates the bias term corresponding to the green light duration header. This indicates the output action of the long head when the light is green; High-level feature representation Input the vehicle guidance head, and the vehicle guidance head will represent the high-level features. The scalar is obtained through processing in the third fully connected layer. u The speed adjustment coefficient is obtained through nonlinear mapping. Based on the speed adjustment coefficient and the road segment speed limit, the recommended speed for connected vehicles is calculated. The mathematical expression is as follows:

[0029]

[0030]

[0031] in, The legal speed limit for road sections, For continuous speed adjustment coefficients, This represents the weight matrix corresponding to the vehicle guidance head. This indicates the bias term corresponding to the vehicle's guide head. This indicates the output action of the vehicle's guide head; The output includes a hybrid action comprising traffic light phase, green light duration, and recommended speed for connected vehicles. .

[0032] Furthermore, when calculating the differentiated immediate reward, including the basic reward, the current domain passage reward, and the next target domain clearing reward, based on the agent's role label, the specific steps are as follows: The value of the corresponding indicator function is determined based on the agent's role label, and the basic reward, current domain passage reward, and next target domain clearing reward are calculated. Based on the values ​​of the agent's indicator function, basic reward, current domain passage reward, and next target domain clearing reward, the differentiated immediate reward for the current agent is calculated. The mathematical expression is as follows:

[0033]

[0034]

[0035]

[0036] in, This represents the differentiated instant reward for the i-th agent. Basic rewards, This represents the total number of vehicles queuing after the stop line at each entrance lane of the intersection. This represents the total number of vehicles that left the intersection via the stop line during the previous control period. For current domain access rewards, Additional delay time for emergency vehicles within the current intersection control area; The next target domain will have its rewards cleared. Vehicle density on downstream key road segments related to emergency vehicle routes; , As an indicator function, when the agent's role is the current domain, Otherwise, it is 0; when the agent's role is the next target domain. Otherwise, it is 0; , This is a hyperparameter.

[0037] Furthermore, when the cloud samples experience data from the central experience replay pool and updates the parameters of the policy network using gradient methods, it specifically employs a multi-agent proximal policy optimization algorithm for model training. Utilizing experience data, it updates the network parameters using gradient methods, enabling the global policy to continuously evolve towards obtaining higher cumulative rewards. This includes the following steps: From the central experience replay pool, a batch of experience sequences is randomly sampled, including state, action, reward and next state. The state and next state are the joint state formed by the combination of the state vectors of all agents in the current time step and the next time step, respectively. The action is the joint action formed by the combination of the mixed actions of all agents in the current time step. The reward is the globally shared reward of all agents in the current time step. Based on the state, reward, and next state, the global advantage value at each time step is calculated using generalized advantage estimation. The individualized total reward for each agent is calculated based on the aforementioned reward and the differentiated immediate reward. Then, based on the individualized total reward, state, and next state, the local advantage value of each agent is calculated through generalized advantage estimation. Based on the local advantage value, state, and action of each agent, the parameters of the policy network are updated by maximizing the pruning target of the proximal policy optimization. Based on the global advantage value and state, the parameters of the central evaluation network are updated by minimizing the mean squared error loss of the value function; Again, randomly sample a batch of experience sequences from the central experience replay pool to iteratively execute the aforementioned steps until the parameters of the policy network and the central evaluation network converge, or the maximum number of iterations is reached.

[0038] This invention also proposes a comprehensive cooperative priority control system for emergency vehicles based on vehicle-road cooperation and multi-agent reinforcement learning, comprising a signal controller, connected vehicles, a cloud platform, and multiple agents. Each agent corresponds one-to-one with an intersection in the road network and is deployed at the corresponding intersection. The system is programmed or configured to execute the comprehensive cooperative priority control method for emergency vehicles based on vehicle-road cooperation and multi-agent reinforcement learning. Compared with the prior art, the advantages of the present invention are as follows: First, by calculating the optimal route for emergency vehicles based on real-time road network traffic flow data in the cloud, a route intent containing the estimated arrival time and the next target node is generated and sent to the corresponding intersection agents. Each agent determines its role label based on the route intent and generates mixed actions to uniformly control traffic lights and connected vehicles, realizing the integrated coordination of global route planning and local intersection control, and eliminating the interruption point of the emergency vehicle passage chain.

[0039] Second, by generating a hybrid action that includes traffic light phase selection, green light duration, and recommended speed for connected vehicles through a local branch strategy network, the joint and refined control of traffic light phase, green light duration, and social vehicle speed is achieved. This overcomes the shortcomings of traditional methods where discrete signal control and continuous vehicle guidance are generated by independent models, which can easily lead to control command conflicts, and improves the efficiency of road spatiotemporal resource utilization.

[0040] Third, by having each agent determine its role label based on the path intent, and by calculating differentiated real-time rewards based on the role label, including basic rewards, current domain passage rewards, and next target domain clearing rewards, the agents at each intersection are trained in a centralized manner. This enables them to proactively and differentiatedly coordinate based on the global path intent, actively guide traffic flow to clear downstream roads in advance for emergency vehicles, avoid local optimization traps, and achieve optimal global network benefits.

[0041] Fourth, by coupling role tags with a differentiated reward mechanism, downstream intersections proactively clear key road sections before emergency vehicles arrive, significantly reducing additional delays for emergency vehicles at each intersection; at the same time, while ensuring priority passage for emergency vehicles, the efficiency of social vehicle traffic is also taken into account, achieving a multi-objective balance between emergency priority and overall traffic efficiency. Attached Figure Description

[0042] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0043] Figure 2This is a decision-making flowchart of the local branching strategy network in an embodiment of the present invention.

[0044] Figure 3 Flowchart of a multi-agent proximal policy optimization algorithm. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0046] This embodiment proposes a comprehensive collaborative priority control method for emergency vehicles based on vehicle-road cooperation and multi-agent reinforcement learning, aiming to achieve the following objectives: This embodiment proposes a comprehensive collaborative priority control method for emergency vehicles based on vehicle-road cooperation and multi-agent reinforcement learning. This method is an emergency vehicle comprehensive priority passage method that integrates dynamic path planning, road segment social vehicle collaborative guidance, and intersection signal adaptive control in a vehicle-road cooperative environment through multi-agent deep reinforcement learning and hybrid action space decision-making.

[0047] The method in this embodiment is applied to an emergency vehicle global cooperative priority control system based on vehicle-road cooperation. The system architecture is as follows: 1. Perception and Communication Layer: Composed of emergency vehicle onboard units, connected social vehicle onboard units, roadside units, and traffic detectors, it is used to collect real-time traffic data including vehicle location, speed, signal status, queue length, and other all elements.

[0048] 2. Cloud-based Decision and Planning Layer (hereinafter referred to as the cloud): This layer includes a dynamic path planning module and a central training platform. The dynamic path planning module calculates and updates routes for emergency vehicles based on real-time traffic conditions, generating travel intentions that include the estimated arrival time and next location. The central training platform is responsible for the centralized training of multi-agent policy models.

[0049] 3. Edge Collaborative Decision Layer: This layer comprises multiple agents deployed in the control domains of each intersection. Each agent corresponds one-to-one with an intersection in the road network and is deployed at the corresponding intersection. Each agent generates a hybrid action based on its local perception status and the travel intention transmitted from the cloud, using a pre-trained policy network. The hybrid actions include: (1) Signal control sub-action: discrete phase identifier and continuous phase duration.

[0050] (2) Vehicle guidance sub-action: a continuous recommended speed vector used to guide the speed of connected social vehicles in the affected area.

[0051] 4. Distributed execution layer: This includes intelligent signal controllers and connected vehicles (including civilian vehicles and emergency vehicles), which respectively execute the issued signal control commands and vehicle speed guidance commands.

[0052] like Figure 1 As shown, the method in this embodiment includes the following steps: S1: Task Triggering and Global Path Planning. An emergency task is triggered when an emergency vehicle (such as an ambulance or fire truck) initiates a task and reports its destination via the vehicle-to-everything (V2X) network, or when the traffic management center receives an alarm and designates the emergency vehicle and target location. After the emergency task is triggered, the cloud calculates the initial optimal path for the emergency vehicle based on real-time road network conditions, generating a path node sequence and intent (including estimated time of arrival (ETA) and the next target direction). ), and distribute them to the relevant edge agents.

[0053] Specifically, in this embodiment, the cloud acquires real-time road network traffic flow data, and each agent acquires traffic flow data at its respective intersection and extracts features according to a specified period. When an emergency task is triggered, the cloud calculates the optimal route for the emergency vehicle based on the real-time acquired road network traffic flow data using a distributed path planning algorithm, generates a path node sequence and a path intent containing the estimated arrival time and the next target node, and sends the path intent to the agent corresponding to each node in the path node sequence.

[0054] S2: Agent role assignment and state awareness. Each edge agent assigns roles based on the received path intent ( The agent dynamically determines its own label (current domain, next target domain, other domains). The agent constructs a normalized state vector. .

[0055] Specifically, in this embodiment, the agent that receives the path intent determines its own role label based on the relationship between the received path intent and the location of the intersection, and constructs a state vector containing the feature extraction results and its own label. The state vector specifically includes... (1) Signal state: The current enabled phase is one-hot encoding, for example, for a four-phase scheme, it is a 4-dimensional vector; (2) Traffic flow status: average queue length (4-dimensional) and average vehicle waiting time (4-dimensional) at each entrance lane. (3) Emergency vehicle status: the existence of emergency vehicles (1D), their distance from the parking line (1D), and their real-time speed (1D). (4) Collaborative information: role labels (1-dimensional) determined by the path intent issued by the cloud, and vehicle occupancy rate of downstream key road sections (1-dimensional).

[0056] S3: Multi-agent cooperative decision-making based on a hybrid action space. Each agent will have a state vector... The agent receives the path intent and constructs a state vector containing feature extraction results and its own label. This state vector is then input into the local branching policy network. After feature extraction, the agent further infers and forms a hybrid output action based on the three outputs. To control traffic lights and connected vehicles at the intersection, the local branch policy network includes: (1) Phase selection head: The discrete phase probability distribution is output by the Softmax function, and the argmax is taken to obtain the phase selection result of the traffic light. ; (2) Green light duration header: The continuous value σ is output by the Sigmoid function and linearly mapped to the actual green light duration. .

[0057] (3) Vehicle guide head: The continuous adjustment coefficient is output via the Tanh function. Calculate the recommended velocity vector .

[0058] S4: Instruction Execution and Differentiated Reward Feedback. After the mixed actions are distributed and executed, the system calculates differentiated immediate rewards based on the agent's role. .

[0059] Specifically, in this embodiment, after the hybrid action is executed, the cloud or the agent calculates a differentiated instant reward based on the agent's role label, including a basic reward, a current domain access reward, and a next target domain clearing reward. The mathematical expression for the differentiated instant reward is as follows:

[0060]

[0061]

[0062]

[0063] in, This represents the differentiated instant reward for the i-th agent. Basic rewards, This represents the total number of vehicles queuing after the stop line at each entrance lane of the intersection. This represents the total number of vehicles that left the intersection via the stop line during the previous control period. For current domain access rewards, Additional delay time for emergency vehicles within the current intersection control area; The next target domain will have its rewards cleared. Vehicle density on downstream key road sections related to emergency vehicle routes. , As an indicator function, when the agent's role is the current domain, Otherwise, it is 0; when the agent's role is the next target domain. Otherwise, it is 0. , This is a hyperparameter.

[0064] S5: Experience learning and dynamic replanning. The agent feeds back its interaction experience to the central training platform for continuous iterative optimization of the policy model.

[0065] Specifically, in this embodiment, the interactive experience composed of differentiated instant rewards, corresponding state vectors, and mixed actions is stored in the central experience replay pool. The cloud samples experience data from the central experience replay pool and updates the parameters of the policy network through the gradient method. The updated parameters are then sent to each agent to update the local branch policy network of each agent.

[0066] The following section uses SUMO 1.18.0 as a micro-traffic flow simulator and utilizes its TraCI API to interact with a custom Python reinforcement learning control program, providing a detailed explanation of each step.

[0067] (1) Simulation environment configuration: SUMO 1.18.0 can be used as the simulation engine, and the reinforcement learning control program can be written in Python 3.8 and the TraCI library can be imported. The reinforcement learning algorithm part is implemented using TensorFlow 2.0.

[0068] (2) Road Network and Traffic Flow Generation: A 4x4 Manhattan grid road network with 16 signalized intersections was constructed in SUMO, with each intersection employing a standard four-phase control scheme. Basic traffic flow was generated using SUMO's randomTrips.py tool, with vehicle arrivals following a Poisson distribution. Different vehicle types were defined to distinguish between civilian and emergency vehicles: civilian vehicles used either the default type or a custom type. To enhance the model's learning of priority passage, approximately 5% of the vehicles in the total traffic flow were randomly assigned as emergency vehicles, with a desired speed of 15 m / s.

[0069] (3) Communication and Control Interface: The SUMO TraCI API is used to realize real-time interaction between the simulation environment and the external Python control program. The system control cycle Δt = 10 seconds is set, that is, the edge agent makes a decision and executes once every 10 seconds. At the beginning of each control cycle, the emergency vehicle on-board unit, the connected social vehicle on-board unit, the roadside unit and the traffic detector are simulated through the TraCI interface to obtain the vehicle status (position, speed, lane) and signal status (current phase, phase remaining time) of the entire road network.

[0070] In step S1 of this embodiment, when each agent acquires traffic flow data at its respective intersection and performs feature extraction according to a specified period, for any intersection agent... In each control cycle Within this scope, the following feature extraction is performed: (1) Traffic flow feature extraction: Queue length For each entrance lane, count the number of vehicles that are stationary within the designated detection area upstream of the parking line; Average waiting time For each entry lane, estimate the average dwell time of all queued vehicles in that lane within the detection area.

[0071] (2) Emergency vehicle status extraction: If an emergency vehicle is present within the control domain, its real-time distance to the stop line at this intersection is obtained. and instantaneous speed And define the existence status. Otherwise, suppose (Maximum sensing distance) , 0.

[0072] (3) Calculation of downstream market share: (4) The next target node of the global plan Calculate the time occupancy rate of the specified downstream key road segments. The time occupancy rate This refers to the percentage of time a vehicle passes through a downstream key road section detection cross-section within the control period t. It can be calculated using vehicle trajectory data obtained from sensors embedded in the roadside or based on vehicle-road cooperative communication. The calculation formula is as follows:

[0073] in, For the statistical period, This represents the total time that the detection section is occupied by vehicles within this cycle.

[0074] In step S1 of this embodiment, when the cloud calculates the optimal route for emergency vehicles based on real-time acquired road network traffic flow data using a distributed path planning algorithm, it employs an asynchronous, distributed incremental shortest path algorithm to dynamically plan the optimal route for emergency vehicles and generate travel intentions. Specifically, for each node in the road network, the optimal estimated time from that node to the destination of the emergency vehicle and the optimal neighboring node after the vehicle leaves that node are maintained; each node periodically updates its own optimal estimated time and optimal neighboring node based on real-time road segment travel time and the optimal estimated time of neighboring nodes; the optimal estimated time of the node where the emergency vehicle is currently located is used as the expected arrival time, and the optimal neighboring node of the node where the emergency vehicle is currently located is used as the next target node.

[0075] In this embodiment, each node i in the road network periodically maintains and updates the following two core state variables: (1) Optimal estimated time : Indicates the current moment The vehicle departs from the node Depart and travel along the currently calculated optimal route to the shortest estimated travel time to the emergency vehicle destination D.

[0076] (2) Next node : This indicates that in order to achieve the above-mentioned shortest estimated travel time, the vehicle leaves the node. The optimal neighboring node to be selected next.

[0077] Each node periodically performs the following updates, thereby iteratively updating its own optimal estimated time and optimal neighboring node based on the real-time segment travel time and the optimal estimated time of adjacent nodes: (1) Receive neighbor node information: from all neighboring nodes Receive its latest .

[0078] (2) Estimate travel time Based on real-time traffic flow data, estimate the distance from all adjacent nodes. Receive its latest message Real-time estimation from arrive Travel time of the route :

[0079] in, For the length of the road segment, This represents the average speed of vehicles on this road section.

[0080] (3) Calculation of optimal local traffic conditions:

[0081]

[0082] in, For the shortest arrival time, The optimal next node. Indicates at time The emergency vehicle is expected to reach its upstream neighboring node. The earliest time, Indicates at time From the upstream node Drive to the current node The estimated travel time.

[0083] (4) Intent to be issued: The updated version Assign to As a key passage intention, it is sent to the corresponding intersection edge agent in real time.

[0084] The implementation process of step S2 in this embodiment is as follows: (1) Role Assignment: Receive Role tags are dynamically determined based on the location of emergency vehicles. The mathematical expression is as follows:

[0085] In this embodiment, when the agent receiving the route intent determines its own role label based on the positional relationship between the received route intent and the intersection it is located at, specifically, the agent determining its own role label based on whether the intersection contains an emergency vehicle and the relationship between the node corresponding to the intersection and the next target node, including: If the traffic flow data at the intersection where the agent is located includes data on emergency vehicles, the agent determines its role label as the current domain. If the traffic flow data at the intersection where the agent is located does not include data on emergency vehicles, and the node corresponding to the intersection is the same as the next target node, the agent determines its role label as the next target domain. If the traffic flow data at the intersection where the agent is located does not include data on emergency vehicles, and the node corresponding to the intersection is different from the next target node, the agent determines its role label as another domain.

[0086] (2) State vector normalization and construction: Normalize all extracted original features to construct a 17-dimensional normalized state vector. .

[0087]

[0088] in, It is a 4-dimensional vector representing the encoding of the current signal phase. These represent the average queue lengths at the four entrances: east, south, west, and north. These represent the average waiting times for the four entrances: east, south, west, and north. This represents a binary variable indicating whether an emergency vehicle exists within the control area of ​​this intersection (1 represents existence, 0 represents non-existence). This indicates the real-time distance between the emergency vehicle and the stop line at this intersection; Indicates the real-time speed of the emergency vehicle; The agent role label is represented by 0 (0 represents other domains, 1 represents the next target domain, and 2 represents the current domain). This indicates the time occupancy rate of key downstream road sections.

[0089] In step S3 of this embodiment, when the state vector is input into the local branch policy network to generate a hybrid action including traffic light phase, green light duration, and recommended speed for connected vehicles, such as... Figure 2 The image shows the state vector. The pre-trained local branch policy network is input, and after the local branch policy network extracts features from the state vector, the extracted features are processed by the phase selection head, the green light duration head, and the vehicle guidance head, respectively, and finally outputs a hybrid action that includes the traffic light phase, the green light duration, and the recommended speed of the connected vehicle.

[0090] In this embodiment, the local branching strategy network adopts an architecture of "shared feature extraction layer + task-specific output head", and its forward propagation process is as follows: (a) Shared feature extraction layer: state vector First, a non-linear transformation is performed through one or more fully connected layers to extract high-level feature representations. .Right now ),in To share the feature extraction function, Its parameters.

[0091] (b) Phase selection head (Discrete Actions): Representing High-Level Features The input phase selection head represents the high-level features. A 4-dimensional original fraction vector is obtained by processing through the first fully connected layer. Through probability distribution function After converting to a probability distribution with four phases, a greedy strategy is employed. The phase with the highest probability is selected as the traffic light phase, and the mathematical expression is as follows:

[0092]

[0093] in, The probability distribution represents four phases. The original fraction vector, This indicates the output action of the phase selector.

[0094] (c) Green light duration (Continuous Actions): Representing High-Level Features Input the green light duration header, and the green light duration header will represent the high-level features. The scalar is obtained by processing through another fully connected layer (referred to as the second fully connected layer in this embodiment for distinction). The green light duration ratio is then non-linearly mapped to the (0, 1) interval using the Sigmoid function. This green light duration ratio is then converted into the actual green light duration through a linear mapping, as shown in the following mathematical expression:

[0095]

[0096]

[0097] in, This is a normalized green light duration ratio coefficient. , These are the preset minimum and maximum green light durations, respectively. This represents the weight matrix corresponding to the green light duration. This indicates the bias term corresponding to the green light duration header. This indicates the output action of the long head when the green light is on.

[0098] (d) Vehicle guide head (Continuous Actions): Representing High-Level Features Input the vehicle guidance head, and the vehicle guidance head will represent the high-level features. The scalar is obtained through processing in the third fully connected layer. u The speed adjustment coefficient is obtained by nonlinear mapping using the Tanh function within the interval [-1, 1]. Based on the speed adjustment coefficient and the road segment speed limit, the recommended speed for connected vehicles is calculated. The mathematical expression is as follows:

[0099]

[0100]

[0101] in, The legal speed limit for road sections, For continuous speed adjustment coefficients, , This represents the weight matrix corresponding to the vehicle guidance head. This indicates the bias term corresponding to the vehicle's guide head. This indicates the output action of the vehicle guidance head. 0.3 is the speed adjustment coefficient, used to limit the recommended speed to within 70% to 130% of the legal speed limit to comply with actual traffic management safety constraints. The output includes a mixture of actions related to traffic light phase, green light duration, and the recommended speed for connected vehicles. .

[0102] In step S4 of this embodiment, the action... After execution, the environment will be in the next cycle. Calculate instant rewards Specifically, it includes: Based on the current agent's role label, determine the value of the corresponding instruction function. Specifically, when the agent's role is the current domain... Otherwise, it is 0; when the agent's role is the next target domain. Otherwise, it is 0; The mathematical expressions for calculating the base reward, the current domain passage reward, and the next target domain clearing reward are as follows:

[0103]

[0104]

[0105] in, Basic rewards, This represents the total number of vehicles queuing after the stop line at each entrance lane of the intersection. This represents the total number of vehicles that left the intersection via the stop line during the previous control period. For current domain access rewards, Additional delay time for emergency vehicles within the current intersection control area; The next target domain will have its rewards cleared. Vehicle density on downstream key road sections related to emergency vehicle routes.

[0106] Based on the values ​​of the current agent's instruction function, basic reward, current domain passage reward, and next target domain clearing reward, the differentiated immediate reward for the current agent is calculated, as shown in the following mathematical expression:

[0107] in, , For indicator functions, , For hyperparameters, take , .

[0108] In step S5 of this embodiment, when the cloud samples experience data from the central experience replay pool and updates the parameters of the policy network using a gradient method, it specifically employs the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm for model training. The core of this algorithm is to utilize the collected experience data to update network parameters using a gradient method, enabling the global policy to continuously evolve towards obtaining higher cumulative rewards. During training, the cloud-based central evaluation network (Critic) uses shared rewards that reflect the global benefits of the road network to evaluate the value of the joint state. However, when each distributed edge agent generates actions, its policy optimization must simultaneously consider both the global system benefits and the local objectives corresponding to its own role.

[0109] In the training context, the following simplified notation is used: : Represents the joint state of all agents at time step t, which can be regarded as the local state of all agents. Composed of various elements.

[0110] : Represents the joint action performed by all agents at time step t, i.e. ,in This refers to the mixed actions of an intelligent agent.

[0111] : This represents the global shared reward for all agents at time step t. In the MAPPO algorithm, the global shared reward is defined as the aggregation of the local rewards of each agent, that is, the global shared reward is the aggregation of the differentiated instantaneous rewards of each agent.

[0112] : This represents the differentiated instant reward calculated based on the aforementioned step S4.

[0113] To clarify the relationship between globally shared rewards and differentiated instant rewards, the following definition is provided: An agent at time step The individualized total reward used for gradient updates in the policy network is as follows:

[0114] in, Indicates the total individualized reward. Rewards are shared globally. The differentiated instant reward calculated according to step S4 above is used to guide the agent to complete a specific task based on its own role (current domain, next target domain, other domains). , The reward weighting coefficient is a pre-set hyperparameter that can be determined through parameter tuning in an offline simulation environment. Its value needs to balance the priority of global traffic efficiency and local collaborative tasks.

[0115] like Figure 3 As shown, the training process is based on the following core parameters and calculation steps: (1) Preparation of empirical data: A batch of empirical sequences, including states, is randomly sampled from the central empirical replay pool. ,action ,award and the next state .

[0116] (2) Dominance function estimation: based on the state ,award and the next state The global advantage value at each time step is calculated using Generalized Advantage Estimation (GAE). :

[0117]

[0118] in, Indicates timing difference error. This is the discount factor (usually set to 0.99). For the state value function, is the GAE attenuation coefficient, with a value range of [0, 1]. This is the time step, which is usually a positive integer.

[0119] (3) Policy network (Actor) update: based on the aforementioned rewards With the aforementioned differentiated instant rewards Calculate the individualized total reward for each agent. Then based on individualized total reward The local dominance values ​​of each agent are calculated using generalized dominance estimation, based on the agent's local dominance values, state, and actions. Then, the policy network parameters of all agents are updated by maximizing the pruning objective of proximal policy optimization (PPO). :

[0120]

[0121] in, Let the loss function be the pruning strategy. This represents the expectation for a given time step, and in actual training, it is the average of a batch of empirical data. Let be the probability ratio, when A value greater than 1 indicates that the new strategy is more inclined to this action than the old strategy. This represents the clipping function. This is a hyperparameter for the cropping range. For mixed actions, For state vectors, The advantage value is the local advantage value calculated based on the individualized total reward for each agent. (4) Value network (Critic) update: based on the global advantage value calculated in step (2). ,state The parameters of the Critic network are updated by minimizing the mean squared error loss of the value function. :

[0122] in, Let be the value loss function. For the state value function, The advantage function is estimated. Training loop. A batch of experience sequences is randomly sampled again from the central experience replay pool to iteratively execute the aforementioned steps until the parameters of the policy network and the central evaluation network converge, or the maximum number of iterations is reached.

[0123] The pseudocode for the iterative training process described above is shown below. In this embodiment, training is performed for 500 rounds, with each round containing 200 control steps, until the policy converges, and the optimal parameters are saved to the edge agent.

[0124]

[0125] In summary, this invention proposes a comprehensive collaborative priority control system and method for emergency vehicles based on vehicle-road cooperation and multi-agent reinforcement learning. Based on "cloud-edge-device" collaboration and "centralized training-distributed execution" multi-agent reinforcement learning, it achieves the following objectives: (1) Achieve integrated collaborative control: By designing an integrated collaborative control architecture and hybrid action space for "road segment-intersection", we can achieve global collaborative closed-loop optimization of dynamic path planning, intersection signal priority, and road segment vehicle guidance.

[0126] (2) Achieve refined resource scheduling: Through hybrid action space design, achieve joint refined control of signal phase (discrete), green light duration (continuous) and social vehicle speed (continuous).

[0127] (3) Achieve global optimization: Through the multi-agent reinforcement learning framework of "centralized training-decentralized execution" and the role-based collaborative mechanism, the system introduces role-based agents and reward mechanisms based on path intent, enabling the system to proactively guide traffic flow to clear roads in advance for emergency vehicles and achieve the optimal passage corridor in the entire region.

[0128] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0129] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for full-domain cooperative priority control of emergency vehicles based on vehicle-road cooperation and multi-agent reinforcement learning, characterized in that, The method is applied to an emergency vehicle global collaborative priority control system that includes a cloud platform and multiple intelligent agents. Each intelligent agent corresponds one-to-one with an intersection in the road network and is deployed at the corresponding intersection. The method includes the following steps: The cloud acquires real-time road network traffic flow data, and each intelligent agent acquires traffic flow data at its respective intersection and extracts features according to a specified period. When an emergency task is triggered, the cloud uses real-time acquired road network traffic flow data and a distributed path planning algorithm to calculate the optimal route for emergency vehicles, generate a path node sequence and a path intent containing the estimated arrival time of each node in the path node sequence and the next target node, and then distributes the path intent to the intelligent agents corresponding to each node in the path node sequence. The agent that receives the route intent determines its own role label based on the relationship between the received route intent and the location of the intersection, constructs a state vector containing the feature extraction results and its own label, and then inputs the state vector into the local branch policy network to generate a hybrid action containing traffic light phase, green light duration and recommended speed of connected vehicles, so as to control the traffic lights and connected vehicles at the intersection. After the hybrid action is executed, the cloud or agent calculates the differentiated immediate reward, including the basic reward, the current domain passage reward, and the next target domain clearing reward, based on the agent's role label. The differentiated immediate reward, along with the corresponding state vector and hybrid action, is combined to form the interaction experience and stored in the central experience replay pool. The cloud samples experience data from the central experience replay pool and updates the parameters of the policy network through the gradient method. The updated parameters are then distributed to each agent to update the local branch policy network of each agent.

2. The emergency vehicle full-domain cooperative priority control method based on vehicle-road cooperation and multi-agent reinforcement learning according to claim 1, characterized in that, When each agent acquires traffic flow data and extracts features at its respective intersection according to a specified period, for any given intersection agent... In each control cycle Within this scope, the following feature extraction is performed: For each entrance lane, count the number of vehicles stationary within the designated detection area upstream of the parking line to obtain the queue length. ; For each entrance lane, estimate the average dwell time of all queuing vehicles in that lane within the detection area to obtain the average waiting time. ; If an emergency vehicle exists within the control domain, obtain the real-time distance from the emergency vehicle to the intersection stop line. and instantaneous speed And define the existence status. If no emergency vehicle exists within the control domain, then set , , 0, Indicates the maximum sensing distance; Calculate the time occupancy of downstream critical road segments specified by the route intent. The mathematical expression is as follows: in, For the statistical period, This represents the total time that the detection section is occupied by vehicles within this cycle.

3. The emergency vehicle full-domain cooperative priority control method based on vehicle-road cooperation and multi-agent reinforcement learning according to claim 1, characterized in that, When calculating the optimal route for emergency vehicles using a distributed path planning algorithm based on real-time acquired road network traffic flow data in the cloud, specifically for each node in the road network, the optimal estimated time from that node to the destination of the emergency vehicle and the optimal neighboring node after the vehicle leaves that node are maintained. Each node periodically updates its own optimal estimated time and optimal neighboring node based on real-time road segment travel time and the optimal estimated time of neighboring nodes. Starting from the node where the emergency vehicle is currently located, the optimal path node sequence is obtained by tracing the optimal neighboring nodes of each node. The optimal estimated time of the node where the emergency vehicle is currently located is used as the estimated arrival time, and the optimal neighboring node of the node where the emergency vehicle is currently located is used as the next target node.

4. The emergency vehicle global cooperative priority control method based on vehicle-road cooperation and multi-agent reinforcement learning according to claim 3, characterized in that, When each node periodically updates its own optimal estimated time and optimal neighboring node based on the real-time road segment travel time and the optimal estimated time of adjacent nodes, the process includes the following steps: Current node From all adjacent nodes The optimal estimated time to receive its latest message Based on real-time traffic flow data, estimate the distance from adjacent nodes To the current node Travel time of the route The mathematical expression is as follows: in, For the length of the road segment, This represents the average speed of vehicles on that road section. The optimal value of local traffic conditions is calculated based on the travel time of the aforementioned road segment, and the mathematical expression is as follows: in, For the optimal estimated time, The optimal neighboring node. Indicates at time The emergency vehicle is expected to reach its upstream neighboring node. The earliest time, Indicates at time From the upstream node Drive to the current node The estimated travel time.

5. The emergency vehicle full-domain cooperative priority control method based on vehicle-road cooperation and multi-agent reinforcement learning according to claim 1, characterized in that, When an agent receives a route intent, it determines its own role label based on the relationship between the received route intent and the location of the intersection. Specifically, the agent determines its own role label based on whether the intersection contains emergency vehicles and the relationship between the node corresponding to the intersection and the next target node, including: If the traffic flow data at the intersection where the agent is located includes data on emergency vehicles, the agent determines its role label as the current domain. If the traffic flow data at the intersection where the agent is located does not include data on emergency vehicles, and the node corresponding to the intersection is the same as the next target node, the agent determines its role label as the next target domain. If the traffic flow data at the intersection where the agent is located does not include data on emergency vehicles, and the node corresponding to the intersection is different from the next target node, the agent determines its role label as another domain. The mathematical expression for the role label is as follows: In the formula, This represents the role label of the i-th agent.

6. The emergency vehicle full-domain cooperative priority control method based on vehicle-road cooperation and multi-agent reinforcement learning according to claim 1, characterized in that, When constructing a state vector containing feature extraction results and its own label, the state vector includes signal state, traffic flow state, emergency vehicle state, and coordination information, as expressed below: in, The code representing the current signal phase. These represent the average queue lengths at the four entrances: east, south, west, and north. These represent the average waiting times for the four entrances: east, south, west, and north. A binary variable indicating whether an emergency vehicle exists within the control area of ​​this intersection; This indicates the real-time distance between the emergency vehicle and the stop line at this intersection; Indicates the real-time speed of the emergency vehicle; Indicates the role label of the intelligent agent; This indicates the time occupancy rate of key downstream road sections.

7. The emergency vehicle full-domain cooperative priority control method based on vehicle-road cooperation and multi-agent reinforcement learning according to claim 1, characterized in that, When the state vector is input into the local branch policy network to generate a hybrid action containing traffic light phase, green light duration, and recommended speed of connected vehicles, the local branch policy network extracts features from the state vector and processes the extracted features through a phase selection head, a green light duration head, and a vehicle guidance head, respectively, and finally outputs a hybrid action containing traffic light phase, green light duration, and recommended speed of connected vehicles, including the following steps: Extracting state vectors through one or more fully connected layers High-level feature representation The mathematical expression is as follows: in, Let represent the state vector of the i-th agent. To share the feature extraction function, To share the parameters of the feature extraction function; High-level feature representation The input phase selection head represents the high-level features. The original score vector is obtained through processing by the first fully connected layer. After being transformed into a probability distribution for each phase using a probability distribution function, the phase with the highest probability is selected as the traffic light phase using a greedy strategy. The mathematical expression is as follows: in, The probability distribution represents four phases. This indicates the output action of the phase selector; High-level feature representation Input the green light duration header, and the green light duration header will represent the high-level features. The scalar is obtained through processing in the second fully connected layer. The green light duration ratio coefficient is obtained through nonlinear mapping, and then converted into the actual green light duration through linear mapping. The mathematical expression is as follows: in, This is a normalized green light duration ratio coefficient. These are the preset minimum and maximum green light durations, respectively. This represents the weight matrix corresponding to the green light duration. This indicates the bias term corresponding to the green light duration header. This indicates the output action of the long-time indicator when the light is green; High-level feature representation Input the vehicle guidance head, and the vehicle guidance head will represent the high-level features. The scalar is obtained through processing in the third fully connected layer. u The speed adjustment coefficient is obtained through nonlinear mapping. Based on the speed adjustment coefficient and the road segment speed limit, the recommended speed for connected vehicles is calculated. The mathematical expression is as follows: in, The legal speed limit for road sections, For continuous speed adjustment coefficients, This represents the weight matrix corresponding to the vehicle guidance head. This indicates the bias term corresponding to the vehicle's guide head. This indicates the output action of the vehicle's guide head; The output includes a hybrid action comprising traffic light phase, green light duration, and recommended speed for connected vehicles. .

8. The emergency vehicle full-domain cooperative priority control method based on vehicle-road cooperation and multi-agent reinforcement learning according to claim 1, characterized in that, When calculating the differentiated immediate reward, including the basic reward, the current domain passage reward, and the next target domain clearing reward, based on the agent's role label, the specific steps are as follows: The value of the corresponding indicator function is determined based on the agent's role label, and the basic reward, current domain passage reward, and next target domain clearing reward are calculated. Based on the values ​​of the agent's indicator function, basic reward, current domain passage reward, and next target domain clearing reward, the differentiated immediate reward for the current agent is calculated. The mathematical expression is as follows: in, This represents the differentiated instant reward for the i-th agent. Basic rewards, This represents the total number of vehicles queuing after the stop line at each entrance lane of the intersection. This represents the total number of vehicles that left the intersection via the stop line during the previous control period. For current domain access rewards, Additional delay time for emergency vehicles within the current intersection control area; The next target domain will have its rewards cleared. Vehicle density on downstream key road segments related to emergency vehicle routes; , As an indicator function, when the agent's role is the current domain, Otherwise, it is 0; when the agent's role is the next target domain. Otherwise, it is 0; , This is a hyperparameter.

9. The emergency vehicle global cooperative priority control method based on vehicle-road cooperation and multi-agent reinforcement learning according to claim 1, characterized in that, When the cloud samples experience data from the central experience replay pool and updates the parameters of the policy network using gradient methods, it specifically employs a multi-agent proximal policy optimization algorithm for model training. Utilizing experience data, it updates the network parameters using gradient methods, enabling the global policy to continuously evolve towards obtaining higher cumulative rewards. This includes the following steps: From the central experience replay pool, a batch of experience sequences is randomly sampled, including state, action, reward and next state. The state and next state are the joint state formed by the combination of the state vectors of all agents in the current time step and the next time step, respectively. The action is the joint action formed by the combination of the mixed actions of all agents in the current time step. The reward is the globally shared reward of all agents in the current time step. Based on the state, reward, and next state, the global advantage value at each time step is calculated using generalized advantage estimation. The individualized total reward for each agent is calculated based on the aforementioned reward and the differentiated immediate reward. Then, based on the individualized total reward, state, and next state, the local advantage value of each agent is calculated through generalized advantage estimation. Based on the local advantage value, state, and action of each agent, the parameters of the policy network are updated by maximizing the pruning target of the proximal policy optimization. Based on the global advantage value and state, the parameters of the central evaluation network are updated by minimizing the mean squared error loss of the value function; Again, randomly sample a batch of experience sequences from the central experience replay pool to iteratively execute the aforementioned steps until the parameters of the policy network and the central evaluation network converge, or the maximum number of iterations is reached.

10. An emergency vehicle full-domain cooperative priority control system based on vehicle-road cooperation and multi-agent reinforcement learning, characterized in that, The system includes a signal controller, connected vehicles, a cloud platform, and multiple intelligent agents. Each intelligent agent corresponds to a road network intersection and is deployed at the corresponding intersection. The system is programmed or configured to execute the emergency vehicle full-domain collaborative priority control method based on vehicle-road cooperation and multi-agent reinforcement learning as described in any one of claims 1 to 9.

Images