Special Vehicle Priority Passing Method and System Based on Reinforcement Learning

Abstract

The present invention provides a method and system for giving priority to special vehicles based on reinforcement learning, comprising the following steps: an arrangement step: arranging a signal light control agent and a communication judgment agent at an intersection; a signal light control step: using the signal light control agent to control the operation of the signal light at the intersection, and deciding the phase that needs to be switched based on the acquired traffic status. A communication judgment step: using the communication judgment agent to judge and select surrounding agents when a special vehicle arrives and communicate, notifying the downstream intersection of the special vehicle information; a coordination step: optimizing the phase arrangement strategy by combining the signal light control agent and the communication judgment agent, and giving priority to special vehicles. The present invention makes the phase arrangement strategy better by combining the traffic light control agent and the communication judgment agent, thereby achieving the purpose of giving priority to special vehicles.

Description

Technical Field

[0001] The present invention relates to the technical field of special vehicle passage, and in particular to a method and system for giving priority to special vehicles based on reinforcement learning. Background Art

[0002] Special vehicles generally refer to vehicles with special purposes, distinguishing them from everyday vehicles. In urban areas, when traffic accidents or other emergencies occur, emergency vehicles like ambulances and fire trucks are needed to arrive at the scene for rescue operations, or specialized vehicles like road sweepers, police cars, and construction vehicles are needed to complete specialized tasks. This necessitates ensuring that special vehicles arrive quickly on the scene, effectively giving them priority access.

[0003] According to relevant traffic laws in my country, special vehicles are granted the absolute right of way, provided they do not interfere with the normal flow of traffic. They are not subject to restrictions on routes, lanes, directions, speeds, or traffic lights. However, data shows that even special vehicles with the absolute right of way are often powerless in traffic jams. Emergency vehicles may be forced to stop and wait for green lights due to the presence of non-essential vehicles ahead. Furthermore, the failure of non-essential vehicles to yield in a timely manner can significantly increase the travel delays of special vehicles.

[0004] At present, the strategy implemented for special vehicle priority is the signal priority strategy. When an emergency vehicle approaches a traffic light and is identified by roadside detection equipment, the current traffic light phase is switched to the lane where the emergency vehicle is located, allowing the emergency vehicle and the vehicles in front to pass quickly. However, this method cannot solve the situation where multiple emergency vehicles arrive at the intersection at the same time. It is not suitable for traffic that is already congested, and it will also cause traffic congestion to increase.

[0005] The Chinese invention patent document with publication number CN113096419A discloses a signal control method serving vehicle priority passage, with improving green light utilization as the optimization goal. The priority needs of vehicles at intersections are perceived based on the real-time positioning data of vehicles. The matching degree between the timing scheme and traffic demand is analyzed dynamically based on the real-time traffic volume and queue length of each flow direction at the intersection. The green time utilization rate of non-priority flow directions is improved without changing the cycle and phase sequence, and the priority vehicle passage phase can be extended to the greatest extent to ensure that special vehicles have reduced queue time and priority passage through the intersection, effectively reducing the impact of special vehicle priority passage on the normal passage of social vehicles. At the same time, by predicting the arrival time of special vehicles at the intersection, the timing is fine-tuned two cycles in advance to make the transition of the intersection signal scheme smooth.

[0006] Regarding the aforementioned related technologies, the inventors believe that traffic flow at intersections is primarily controlled by traffic lights. Therefore, designing efficient traffic light controllers has always been a key issue in traffic engineering. Due to the complexity and uncertainty of the traffic environment, traditional models are quite difficult to solve. Against this backdrop, providing a traffic light control algorithm suitable for prioritizing special vehicles is of great significance. Summary of the Invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a special vehicle priority passage method and system based on reinforcement learning.

[0008] According to the present invention, a method for giving priority to special vehicles based on reinforcement learning includes the following steps:

[0009] Arrangement steps: Arrange signal light control agents at the intersection;

[0010] Traffic light control steps: Use the traffic light control agent to control the operation of the intersection traffic lights and decide the phase that needs to be switched based on the acquired traffic status.

[0011] Preferably, in the arranging step, a communication judgment agent is arranged at the intersection;

[0012] The method further includes a communication judgment step: using a communication judgment agent to judge and select surrounding agents when the special vehicle arrives and communicate, and notifying the downstream intersection of the special vehicle information;

[0013] Coordination steps: By combining the traffic light control agent and the communication judgment agent to optimize the phase arrangement strategy, special vehicles are given priority.

[0014] Preferably, the method further includes an agent training step: using reinforcement learning to train a traffic light control agent and a communication judgment agent.

[0015] Preferably, in the signal light control step, the signal light control agent observes the real-time traffic status of the intersection, and then makes periodic decisions based on the observed traffic status to dynamically arrange the intersection signal lights so that special vehicles have priority;

[0016] The traffic status perceived by the traffic light control agent includes the status of special vehicles, social vehicles and the status of traffic lights;

[0017] Special vehicle status ev Including the lane information of the special vehicle i and the instantaneous speed s of the special vehicle i , the status code is:

[0018] state ev=[p1, p2, ..., p n ,s1,s2,…,s n ]

[0019] Among them, n is the total number of entrance lanes of the intersection, and all values of this state are initialized to 0. If a special vehicle is detected in lane i, then p i =1, the corresponding s i Set to the current speed of the special vehicle;

[0020] Social vehicle state social The queue length q of social vehicles within the observation range of each import lane i is i And the vehicle density d within the sensing range i ;

[0021] state social =[d1, d2, ..., d n ,q1,q2,…,q n ]

[0022] Traffic light state cross including the current traffic signal phase;

[0023] The perception state S of the traffic light control agent is

[0024] S=state ev , state social , state cross .

[0025] Preferably, in the signal light control step, the reward function of the signal light control agent includes a reward function for special vehicles and a reward function for social vehicles;

[0026] Special vehicle reward function reward ev Calculated by the following formula:

[0027]

[0028] R p =20

[0029]

[0030] Among them, E c represents the set of special vehicles observed in this round of decision making, E p represents the set of special vehicles that did not appear in this round but were observed in the previous round of decision making, R c (e) represents E c The reward function for the special car in the set, R p Indicates E pThe reward function for the special car in the set, speed e represents the speed of vehicle e, wait e represents the waiting time of vehicle e at the intersection;

[0031] Reward function of social vehicles social Calculated by the following formula:

[0032]

[0033] Where L is the total number of import lanes, q l represents the queue length of lane l, N is the total number of vehicles within the lane detection range, wait i represents the waiting time of vehicle i, α is the coefficient;

[0034] The reward function R of the traffic light control agent is the weighted sum of the social vehicle reward function and the special vehicle reward function, and is calculated by the following formula:

[0035] R=reward ev +β*reward social

[0036] Among them, β is a coefficient that adjusts the proportion of social vehicles in the reward.

[0037] Preferably, in the communication judgment step, the communication judgment agent observes the real-time traffic status. If the communication judgment agent detects the arrival of a special vehicle, it determines the direction of travel of the special vehicle and then notifies the agent at the downstream intersection.

[0038] The traffic status perceived by the communication judgment agent includes the status of the special vehicle and the status of the surrounding intersection lanes;

[0039] Special vehicle status ev Including the lane information and category of the special vehicle, the status code is as follows:

[0040] state ev =[p1, p2, ..., p n , c1, c2, …, c g ]

[0041] Among them, g is the number of special vehicles. All values of this state are initialized to 0. If a special vehicle is detected in lane i, then p i =1, if the type of special vehicle is type i, then c i =1;

[0042] The state of the surrounding intersection lanes to Including the vehicle density of the entire lane from the intersection to the neighboring intersection, the status coding is as follows:

[0043] state to =[t1, t2, ..., t h ]

[0044] Where h is the total number of neighbors at the intersection, t i represents the vehicle density on the road from this intersection to the neighboring intersection i;

[0045] Communication judgment agent's perception state S j for:

[0046] S j =state ev , state to .

[0047] Preferably, in the communication judgment step, the reward function R of the communication judgment agent is j Calculated by the following formula:

[0048]

[0049]

[0050] Among them, N target is the node that the special vehicle finally arrives at, N impossible is the set of nodes that cannot be reached, N judge is the set of nodes to be reached by the agent. R(x) represents the number of nodes in N judge The reward of the node in the set, x is N judge Node in the set; R j is the agent’s reward function.

[0051] Preferably, in the combining step, when the two layers of agents are combined, the state of the signal light control agent is added with the information state of the special vehicle that is about to arrive. coming And encode, give each lane a position, using the following definition:

[0052] state coming =[l1, l2, ..., l n ]

[0053] Among them, l i represents lane i. If a special vehicle is detected on a certain side, l corresponding to all lanes on that side is set to 1. The perception state S of the traffic light control agent is updated as follows:

[0054] S=state ev , state social , state cross , statecoming .

[0055] According to the present invention, a special vehicle priority passage system based on reinforcement learning is provided, which includes the following modules:

[0056] Arrangement module: Arrange signal light control agents at intersections;

[0057] Traffic light control module: Use the traffic light control agent to control the operation of the intersection traffic lights and decide the phase that needs to be switched based on the acquired traffic status.

[0058] Preferably, in the arrangement module, a communication judgment agent is arranged at the intersection;

[0059] The method further includes a communication judgment module: using a communication judgment agent to judge and select surrounding agents when the special vehicle arrives and communicate with them, and notifying the downstream intersection of the special vehicle information;

[0060] Coordination module: By combining the traffic light control agent and the communication judgment agent to optimize the phase arrangement strategy, special vehicles are given priority passage.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. The present invention combines traffic light control agents with communication judgment agents to optimize the phase arrangement strategy and achieve the goal of giving special vehicles priority passage.

[0063] 2. The intelligent agent of the present invention learns optimal control decisions by interacting with the environment, and can flexibly respond to different traffic situations. It allows special vehicles to pass through intersections quickly without affecting the flow of other vehicles, and can also handle situations where multiple special vehicles are present at the same time.

[0064] 3. At intersections where no special vehicles are present, the intelligent traffic lights of the present invention can improve the traffic conditions of social vehicles, reduce intersection congestion, and increase the utilization rate of green light time. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0066] Figure 1 This is a structural diagram of the traffic light control agent of the present invention;

[0067] Figure 2 A structural diagram of the communication judgment agent of the present invention;

[0068] Figure 3 This is a system framework diagram that combines the traffic light control agent and the communication judgment agent. DETAILED DESCRIPTION

[0069] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0070] The present invention discloses a special vehicle priority passage method based on double-layer intelligent agent reinforcement learning, comprising: a traffic light control intelligent agent, a communication judgment intelligent agent and a combination of two layers of reinforcement learning.

[0071] The method comprises the following steps:

[0072] Arrangement steps: Arrange signal light control intelligent agents at intersections; arrange communication judgment intelligent agents at intersections.

[0073] Traffic light control steps: Use the traffic light control agent to control the operation of the intersection traffic lights and decide the phase that needs to be switched based on the acquired traffic status.

[0074] Communication judgment step: Use the communication judgment agent to judge and select surrounding agents when the special vehicle arrives and communicate, and notify the downstream intersection of the special vehicle information.

[0075] Coordination steps: By combining the traffic light control agent and the communication judgment agent to optimize the phase arrangement strategy, special vehicles are given priority.

[0076] Specifically, this method uses a traffic light control agent to control intersection signal operation. Based on the traffic flow state, it periodically determines the phase to switch to. A communication judgment agent determines which surrounding agents to communicate with upon the arrival of a special vehicle, notifying downstream intersections of the special vehicle's presence. Finally, by combining the traffic light control agent and the communication judgment agent, a more optimal phase scheduling strategy is achieved, ensuring that special vehicles have priority passage.

[0077] Traffic Light Control Agent: A traffic light control agent is deployed at each intersection. This agent observes the intersection's real-time traffic status, which is captured by on-board GPS, roadside sensors, intersection cameras, and other devices. Based on this observed status, the agent periodically makes decisions and dynamically schedules appropriate traffic light signals, alleviating congestion at the intersection while giving priority to special vehicles. This agent is trained using reinforcement learning.

[0078] Regarding the definition of the environmental state space perceived by the intersection signal control agent, this invention defines states as special vehicle states, public vehicle states, and traffic signal states. These states can be captured by roadside detectors, intersection cameras, and other equipment. Each intersection agent perceives the vehicle states of the entrance lane within 50 to 60 meters of the intersection. If a lane is smaller than this range, the agent observes the state of the entire lane.

[0079] The status of the special vehicle includes the lane information and the instantaneous speed of the special vehicle; its status is coded as:

[0080] state ev =[p1, p2, ..., p n ,s1,s2,…,s n ]

[0081] Among them, n is the total number of entrance lanes of the intersection, and all values of this state are initialized to 0. If a special vehicle is detected in lane i, then p i =1, the corresponding s i Set to the current speed of the vehicle.

[0082] The status of social vehicles includes the queue length of social vehicles within the observation range of each import lane and the vehicle density within the perception range.

[0083] state social =[d1, d2, ..., d n ,q1,q2,…,q n ]

[0084] Where n is the total number of entrance lanes at the intersection, for each lane i, d i is the vehicle density on the lane, q i Defined as the lane queue length on that lane.

[0085] The status of the traffic light includes the current traffic signal phase.

[0086] For the state of the traffic light cross , including the current traffic signal phase, is set as a one-hot vector. One-hot represents a unique hot code. In summary, the state S perceived by the traffic light control agent is

[0087] S=state ev , state social , state cross .

[0088] In the agent's action space, each action is defined as a green light phase, that is, the red and green combination of the intersection's traffic lights. Specifically, a set U of all possible phases for each intersection is predefined. The agent makes a decision every 5 seconds, selecting one of the phases from this set to execute. If the phase remains unchanged, the traffic light continues to operate as before. If the phase changes, the current phase of the traffic light is switched to the new phase, and the corresponding yellow signal is inserted. This switching process is the process by which the agent executes the action.

[0089] The agent's reward function includes the reward functions for both the emergency vehicle and social vehicles. Regarding the design of the agent's reward function, the agent's reward function consists of two parts: one for the emergency vehicle and the other for social vehicles. The emergency vehicle reward function takes into account its speed, waiting time, and whether it passes through the intersection.

[0090] Calculated by the following formula.

[0091]

[0092] R p =20

[0093]

[0094] Among them, E c represents the set of special vehicles observed in this round of decision making, E p represents the set of special vehicles that did not appear in this round but were observed in the previous round of decision making, R c (e) represents E c The reward function for the special car in the set, R p Indicates E p The reward function for the special car in the set, speed e represents the speed of vehicle e, wait e represents the waiting time of vehicle e at the intersection.

[0095] The reward function for social vehicles mainly takes into account the waiting time of social vehicles and the length of the queue. It can be calculated by the following formula:

[0096]

[0097] Where L is the total number of import lanes, q l represents the queue length of lane l, N is the total number of vehicles within the lane detection range, wait i represents the waiting time of vehicle i, and α is the coefficient.

[0098] The agent's reward function is the weighted sum of the social vehicle reward function and the emergency vehicle reward function. It is calculated using the following formula:

[0099] R=reward ev +β*reward social

[0100] Among them, β is a coefficient that can adjust the proportion of social vehicles in the reward.

[0101] The structure of the signal control agent in the present invention is as follows Figure 1 As shown, the agent combines the perceived states and then extracts traffic flow features through an LSTM layer. These features are then fed into the policy network and value network to obtain the corresponding policy and value. The traffic light control agent is trained using the reinforcement learning A2C algorithm. The specific steps are as follows.

[0102] The reinforcement learning training of the signal light control agent is carried out according to the following steps: first, the environment and network parameters are initialized. The structure of the signal light control agent in the present invention is as follows: Figure 1 As shown in the figure, the structure includes three modules. The first module is the state S observed by the agent. State S includes the traffic signal phase, special vehicle speed, special vehicle lane, lane vehicle density, and lane queue length. The state is then merged and spliced to the input of the next module. The second module is the LSTM layer. LSTM refers to the long short-term memory neural network, which is used to input the merged state and output the traffic flow characteristics within a period of time. The features are then input into the third module. The extracted features are input into the policy network and the value network respectively. The policy network is used to output the strategy π that the agent should make. θ , the value network outputs the value V corresponding to the action.

[0103] Then the agent is trained. The agent inputs the observed environment state S into the agent network and obtains the strategy π θ , and the value V output by the value network, and then use the strategy π θ The agent's action a is obtained by probability sampling. After the agent performs action a, the environment changes, and the observable environmental state is S'. Based on state S and state S', the method provided by the present invention is used to calculate the agent's reward R in this decision, and then the agent's parameters are updated using the following formula (this is the general practice of the A2C algorithm).

[0104]

[0105]

[0106] Among them, θ is the parameter of the policy network, ω is the parameter of the value network, α is the learning rate of the policy network, and πθ (S, a) represents the value of the policy corresponding to state S and action a, Represents logπ θ (S, a) takes the derivative with respect to θ, where R is the reward R in this decision, γ is the decay factor, V(S′) is the value of state S′, and V(S) is the value of state S.

[0107] After the parameters are updated as described above, the next round begins, where the environment state S is assigned to S′. The above method is then used to update the state until convergence is achieved. This is the reinforcement learning training process for the traffic light control agent.

[0108] Communication and Decision Agent: A communication and decision agent is placed at each intersection. This agent can observe real-time traffic conditions, which can be obtained through on-board GPS, roadside detectors, intersection cameras, and other equipment. If the agent detects the arrival of a special vehicle, it determines its direction of travel and notifies the agent at the downstream intersection. The agent of this invention does not determine a single neighbor to communicate with; rather, it can notify multiple neighbors simultaneously.

[0109] Regarding the definition of the environmental state space perceived by the communication judgment agent, this invention defines the state as the state of the special vehicle and the state of the surrounding intersection lanes, which can be obtained by roadside detectors, intersection cameras, and other equipment. The agent at each intersection perceives the vehicle state of the entrance lane within 50 to 60 meters of the intersection. If a lane is smaller than this range, the agent observes the state of the entire lane.

[0110] The status of the special vehicle includes (taking into account) the lane information p where the special vehicle is located i , and category c of special vehicles i , its status code is as follows:

[0111] state ev =[p1, p2, ..., p n , c1, c2, …, c g ]

[0112] Among them, n is the total number of entrance lanes of the intersection, g is the number of types of special vehicles, all values of this state are initialized to 0, if a special vehicle is detected in lane i, then p i =1, if the type of special vehicle is type i, then c i =1.

[0113] The state of the lanes at the surrounding intersections includes (taking into account) the vehicle density of the lanes from the current intersection to the neighboring intersections. The state is coded as follows:

[0114] state to=[t1, t2, ..., t h ]

[0115] Where h is the total number of neighbors at the intersection, t i Represents the vehicle density on the road from this intersection to the neighboring intersection i. In summary, we can get the state S perceived by the communication judgment agent j for:

[0116] S j =state ev , state to .

[0117] For the agent's reward function R j In the design of , we must consider that the notification agent should be notified as much as possible, and the agents that should not be notified should be avoided as much as possible. It is calculated by the following formula:

[0118]

[0119]

[0120] Among them, N target is the node that the special vehicle finally arrives at, N impossible is the set of nodes that cannot be reached, N judge is the set of nodes to be reached by the agent. R(x) represents the number of nodes in N judge The reward of the node in the set, x is the parameter of the R(x) function, which is N judge Node in the set. j is the agent’s reward function.

[0121] For the action space of the agent, each action is defined as the set of neighbors to be notified. For each neighbor, it can decide whether to notify or not, so the size of the action space is H = 2 h , where h is the total number of neighbors of the agent. When the agent observes the arrival of a special vehicle, it makes a decision based on the defined state, selects the best set of neighbors to notify from the action set, and then communicates with them, conveying the information of the special vehicle's impending arrival to the downstream agents.

[0122] The structure of the communication judgment agent in the present invention is as follows Figure 2 As shown, the agent combines the perceived states as features of the observed traffic flow and inputs them into the value network to obtain the value corresponding to each action. The action with the highest value is then selected for execution. The communication judgment agent is trained using the reinforcement learning DQN algorithm. The specific steps are as follows.

[0123] The reinforcement learning training of the communication judgment agent is carried out according to the following steps: first, the environment and network parameters are initialized. The structure of the signal control agent in the present invention is as follows: Figure 1 As shown in Figure 2, the structure includes two modules. The first module is the state S observed by the agent. j , state S j It includes the type of special vehicle, the lane where the special vehicle is located, and the traffic density to the surrounding intersection lanes. Then its state is merged and spliced into the input of the next module. The second module is the value network module, which is used to input the merged state and output the value corresponding to each action. In the figure, Q(S, a0) represents the value of the action in state S. j The value of executing action a0 is shown in the figure Q(S, a H ) indicates that in state S j Next, perform action a H In the figure, H represents the total number of actions of the communication judgment agent.

[0124] Then the agent is trained. The agent is trained according to the observed environment state S j , input into the agent network, and get the value of each action Q(S j , a), then take the action corresponding to the maximum value among these outputs to get the action a of the agent j , the agent performs action a j After the environment changes, the observable environmental state is S' j , according to the state S j and state S′ j , use the method provided by this invention to calculate the reward R of the agent in this decision j , and then use the following formula to update the agent parameters (this is the common practice of the DQN algorithm).

[0125]

[0126]

[0127] Among them, R j is the reward value, γ is the discount coefficient, which represents the discount of future rewards, S j is the state observed by the environment, Q represents the value function of the value network, and a j is the selected action, S′ j State S j Execute action a j The state observed later, w j is the network parameter of the value network, and α is the learning rate of the value network.

[0128] After the parameters are updated according to the above method, the next round begins and the environment state Sj Assigned to S′ j , and then continue to update according to the above method until learning converges. This is the reinforcement learning training process of the communication judgment agent.

[0129] Combination of two-layer agents: This step combines the traffic light control agent and the communication judgment agent to work together to achieve the goal of emergency vehicles having priority passage. In the process of combining the two, the traffic light control agent is responsible for controlling the operation of the traffic light, and the communication judgment agent determines which neighbors to communicate with when there are special vehicles. The system framework of the combination of traffic light control agent and communication judgment agent is as follows: Figure 3 shown.

[0130] In the case of combining two layers of agents, the state of the traffic light control agent needs to add some new content, which is the information state of the special vehicle that is about to arrive. coming Encoding, assigning a position to each lane. The following definition is used:

[0131] state coming =[l1, l2, ..., l n ]

[0132] Where n represents the total number of intersection entrances, l i represents lane i. If a special vehicle is detected on a certain side, l corresponding to all lanes on that side is set to 1. Therefore, the state perceived by the agent is updated as follows:

[0133] S=state ev , state social , state cross , state comming

[0134] In addition, the combination of the two layers of agents requires a change in the reward function of the traffic light control agent, which means that additional reward settings are required. Under the basic reward settings, if the selected action can completely cover the edge where the special car is coming, the reward value is increased by a constant.

[0135] R=R+c

[0136] For the training of two-layer agents. First, use the reinforcement learning training steps of the communication judgment agent mentioned above to train the communication judgment agent. Then, after it converges, use the reinforcement learning training steps of the signal light control agent mentioned above to train the signal light control agent. In the process of training the signal light control agent, the agent is based on the observed environmental state S. If the arrival of a special vehicle is detected, the notification agent notifies the neighbors, and the state S of the signal light control agent is updated according to the present invention (the above steps). Then, the action a selected by the agent is executed, and the reward R is calculated according to the updated reward function. Finally, the parameters of the network are updated according to the original steps. If the arrival of the special vehicle is not detected, the agent is trained according to the original steps. Finally, a combination of two layers of agents can be obtained.

[0137] The present invention also provides a special vehicle priority passage system based on reinforcement learning. The special vehicle priority passage system based on reinforcement learning can be implemented by executing the process steps of the special vehicle priority passage method based on reinforcement learning, that is, those skilled in the art can understand the special vehicle priority passage method based on reinforcement learning as a preferred implementation method of the special vehicle priority passage system based on reinforcement learning.

[0138] The system includes the following modules:

[0139] Arrangement module: Arrange signal light control agents at intersections; arrange communication judgment agents at intersections.

[0140] Traffic light control module: Use the traffic light control agent to control the operation of the intersection traffic lights and decide the phase that needs to be switched based on the acquired traffic status.

[0141] Communication judgment module: Use the communication judgment agent to judge and select surrounding agents when the special vehicle arrives and communicate with them, and notify the downstream intersection of the special vehicle information.

[0142] Coordination module: By combining the traffic light control agent and the communication judgment agent to optimize the phase arrangement strategy, special vehicles are given priority passage.

[0143] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0144] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A special vehicle priority method based on reinforcement learning, characterized in that: The steps include: Arrangement steps: Arrange signal light control agents at the intersection; Traffic light control step: Use the traffic light control agent to control the operation of the intersection traffic lights and decide the current phase that needs to be switched based on the acquired traffic status; In the arrangement step, a communication judgment agent is arranged at the intersection; The method further includes a communication judgment step: using a communication judgment agent to judge and select surrounding agents when the special vehicle arrives and communicate, and notifying the downstream intersection of the special vehicle information; Coordination steps: By combining the signal light control agent and the communication judgment agent to optimize the phase arrangement strategy, special vehicles are given priority passage; The method also includes an agent training step: using reinforcement learning to train a traffic light control agent and a communication judgment agent; In the signal light control step, the signal light control agent observes the real-time traffic status of the intersection, and then makes periodic decisions based on the observed traffic status to dynamically arrange the intersection signal lights so that special vehicles have priority; The traffic status perceived by the traffic light control agent includes the status of special vehicles, social vehicles and the status of traffic lights; Special vehicle status Including information about the lane where the special vehicle is located and the instantaneous speed of the special vehicle , the status code is: in, n The total number of entrance lanes at the intersection. All values of this state are initialized to 0. If a lane is detected If there is a special car, , corresponding to Set to the current speed of the special vehicle; Social vehicle status Including each import lane The queue length of social vehicles within the observation range and vehicle density within the sensing range ; Traffic light status including the current traffic signal phase; Traffic lights control the agent's perception state for ； In the signal light control step, the reward function of the signal light control agent includes a reward function for special vehicles and a reward function for social vehicles; Reward function for special vehicles Calculated by the following formula: in, represents the set of special vehicles observed in this round of decision-making process, represents the set of special vehicles that did not appear in this round but were observed in the previous round of decision-making. express The reward function for the special car in the set, express The reward function for the special car in the set, Indicates vehicle e speed, Indicates vehicle e Waiting time at intersections; Reward function for social vehicles Calculated by the following formula: in, is the total number of import lanes, Indicates lane The queue length, is the total number of vehicles within the lane detection range, Indicates vehicle The waiting time, is the coefficient; Reward function for the traffic light control agent is the weighted sum of the social vehicle reward function and the special vehicle reward function, calculated by the following formula: in, is the coefficient, which adjusts the proportion of social vehicles in the reward; In the communication judgment step, the communication judgment agent observes the real-time traffic status. If the communication judgment agent detects the arrival of a special vehicle, it determines the direction of travel of the special vehicle and then notifies the agent at the downstream intersection. The traffic status perceived by the communication judgment agent includes the status of the special vehicle and the status of the surrounding intersection lanes; Special vehicle status Including the lane information and category of the special vehicle, the status code is as follows: in, g The number of special vehicles. All values of this state are initialized to 0. If a lane is detected If there is a special car, , if the type of special vehicle is species, then ; Status of surrounding intersection lanes Including the vehicle density of the entire lane from the intersection to the neighboring intersection, the status coding is as follows: in, h is the total number of neighbors at the intersection, Indicates the intersection from this intersection to the neighboring intersection vehicle density of the road; Communication determines the perception state of the intelligent agent for: ； In the communication judgment step, the reward function of the communication judgment agent is Calculated by the following formula: in, is the node where the special vehicle finally arrives. is the set of nodes that cannot be reached. The set of nodes to be reached as determined by the agent; Indicates The rewards for nodes in the collection, x for Nodes in a collection; is the agent’s reward function; In the combining step, when the two layers of agents are combined, the state of the signal light control agent is added with the information of the special vehicle that is coming. And encode, give each lane a position, using the following definition: in, Indicates lane i If a special vehicle is detected on a certain side, all lanes on that side will be Set to 1; the signal light controls the agent's perception state S Updated to: 。 2. A special vehicle priority system based on reinforcement learning, characterized in that: The method of claim 1 comprises the following modules: Arrangement module: Arrange signal light control agents at intersections; Traffic light control module: Uses the traffic light control agent to control the operation of the intersection traffic lights and decides the current phase that needs to be switched based on the acquired traffic status; In the arrangement module, a communication judgment agent is arranged at the intersection; The method further includes a communication judgment module: using a communication judgment agent to judge and select surrounding agents when the special vehicle arrives and communicate with them, and notifying the downstream intersection of the special vehicle information; Coordination module: By combining the traffic light control agent and the communication judgment agent to optimize the phase arrangement strategy, special vehicles are given priority passage.

Images