Two-stage emergency signal priority control method, device and medium based on RL

Abstract

The present invention relates to a two-stage emergency signal priority control method, device, and medium based on RL, wherein the control method includes: obtaining historical traffic data information of intersections based on V2I; offline training of a DQN model based on the historical traffic data information, wherein the reward function in the offline training DQN model is in the form of a piecewise function, and whether the emergency vehicle passes through the intersection is used as a judgment condition, wherein the reward function of the first stage is composed of two parts: the priority right of way of the emergency vehicle and the traffic efficiency of social vehicles, and the reward function of the second stage only includes the traffic efficiency of social vehicles; and the control decision of the intersection signal scheme is generated by using the trained DQN model. Compared with the prior art, the present invention designs a set of control strategies based on the trained DQN model, which can provide signal priority for emergency vehicles according to the real-time traffic status collected by V2I, while reducing the delay effect on other social vehicles at the intersection.

Description

Technical Field

[0001] The invention relates to the field of priority control of emergency vehicle traffic signals at intersections, and in particular to a two-stage emergency signal priority control method, equipment and medium based on reinforcement learning. Background Art

[0002] The ability of a city to respond to and handle emergencies is crucial to protecting the normal operation of the city and the safety of people's lives and property, and the design of the transportation system as the "meridian" of the city is particularly important. Emergency vehicles can quickly reach their destinations from a certain place in the city to respond to and handle emergencies. For example, fire trucks can pass through several city intersections unimpeded to reach the fire site to deal with the fire, and ambulances can pass through several city intersections unimpeded to treat the injured in time. This is of great significance to preventing the spread of fires and protecting the lives of the injured. As the main factor hindering the safe and rapid passage of emergency vehicles, intersections should be reasonably considered for signal control schemes when facing the passage of emergency vehicles.

[0003] An existing Chinese patent (CN202010610564.2) provides a signal timing method for emergency vehicles passing through intersections, including obtaining the traffic flow status of a road section based on V2X, calculating the emergency vehicle travel time based on the Davidson function, and calculating the shortest path based on the Dijkstra algorithm, to achieve optimal adjustment of intersection traffic signals when emergency vehicles pass through.

[0004] There is also a Chinese patent (CN201610804554.6) that provides an emergency vehicle signal priority control method, including vehicle shortest path analysis based on emergency vehicle GPS data collection and map matching, and executing the phase locking and unlocking timing of intersection signal control by analyzing the distance of the emergency vehicle to the intersection, so as to achieve lane clearance before the arrival of the emergency vehicle and absolute priority passage for the emergency vehicle.

[0005] In addition, there is a Chinese patent (CN202310601141.8) that provides a dynamic emergency lane clearing method, which includes collecting traffic data of the corresponding road sections, discretizing the road sections into grids and calculating their lengths, and determining the optimal clearing position and plan based on a heuristic algorithm, while taking into account the queue length to achieve efficient passage of emergency vehicles.

[0006] At present, there is a certain foundation for the research on signal control schemes for emergency vehicles passing through intersections, but there are still the following defects:

[0007] First, most of the research focuses on how to ensure that emergency vehicles can pass quickly and efficiently at intersections. Since intersection signal control schemes guarantee priority for emergency vehicles, they will have a negative impact on the traffic efficiency of the entire intersection, mainly in the form of increased delays for social vehicles at the intersection. Currently, there are few signal control schemes that take into account both priority for emergency vehicles and efficiency for social vehicles.

[0008] Secondly, the design of existing emergency vehicle signal control schemes does not adequately consider the restoration of the chaotic order of social vehicles caused by emergency vehicles passing through intersections. That is, there are few solutions for how to handle the transition from the traffic signal priority control scheme for emergency vehicles passing through intersections to the general traffic signal control scheme. Summary of the invention

[0009] The purpose of the present invention is to provide a two-stage emergency signal priority control method, device and medium based on reinforcement learning (RL) in order to give priority to emergency vehicles at intersections and minimize the delay of social vehicles. At the same time, in order to quickly restore the chaotic order of social vehicles after the emergency vehicles pass through the intersection,

[0010] The purpose of the present invention can be achieved by the following technical solutions:

[0011] The first aspect of the present invention provides a two-stage emergency signal priority control method based on reinforcement learning, comprising:

[0012] S1: Obtain historical traffic data information of intersections based on V2I;

[0013] S2: Offline training of the DQN model based on the historical traffic data information,

[0014] The reward function in the offline training DQN model is in the form of a piecewise function, and whether the emergency vehicle passes through the intersection is used as the judgment condition. The reward function in the first stage consists of the priority right of emergency vehicles and the traffic efficiency of social vehicles, and the reward function in the second stage only includes the traffic efficiency of social vehicles.

[0015] S3: Generate control decisions for intersection signal schemes using the trained DQN model.

[0016] Furthermore, in S1, the historical traffic data information of the V2I-based intersection includes:

[0017] Historical traffic data information, including: the number of vehicles on each entrance road n j,k , speed v j,k,l , queue length l j,k ;

[0018] Real-time traffic information at intersections, including the location of emergency vehicles e , the speed of the emergency vehicle v e , the position of social vehicles at each entrance of the intersection T s , the number of social vehicles on each entrance of the intersection n j,k , the speed v of social vehicles at each entrance of the intersection j,k,l , where j=n,s,e,w, respectively represent the north entrance, south entrance, east entrance and west entrance of the intersection, K=1,2,3..., respectively represent the lane numbers of each entrance direction, and L=1,2,3..., respectively represent the numbers of private vehicles on each lane.

[0019] Furthermore, S2 specifically includes the following steps:

[0020] S2-1: Obtain the traffic status information of the emergency vehicle and the intersection through which the emergency vehicle will pass, as the state space S of the Markov decision process of the emergency vehicle passing through the target intersection;

[0021] S2-2: Define the action space A of the Markov decision process of the emergency vehicle passing through the target intersection as the signal phase of the target intersection;

[0022] S2-3: Construct the reward function of the Markov decision process of the emergency vehicle passing through the target intersection;

[0023] S2-4: Train the neural network Q(s,a;ω).

[0024] Furthermore, in S2-1, the state space S includes the distance d between the emergency vehicle and the stop line of the intersection entrance, the speed v of the emergency vehicle, and the e , the number of social vehicles on each entrance of the intersection n j,k , the speed v of social vehicles at each entrance of the intersection j,k,l , the queue length of social vehicles at each entrance of the intersection l j,k ;

[0025] In S2-2, the action space A includes east-west straight going and right turn (EW), east-west left turn (EWL), north-south straight going and right turn (NS), and north-south left turn (NSL).

[0026] Furthermore, in S2-3, the specific process includes:

[0027] The reward function is designed as R according to whether the emergency vehicle exits the intersection exit. E and R S To cope with the traffic signal priority control stage and the traffic signal recovery control stage, the different training objective reward functions R are constructed as follows:

[0028]

[0029] R E =μ 1 v e +μ 2 n stop +μ 3 t

[0030] R S =μ 4 n j,k +μ 5 t j,k +μ 6 l j,k

[0031] Among them, R E R is the reward function for emergency vehicle actions. E The speed v of the emergency vehicle e , the number of emergency vehicle stops n stop , the waiting time t of the emergency vehicle;

[0032] R S is the reward function reflecting the social vehicle efficiency, R E The number of social vehicles on each entrance road of the intersection n j,k , waiting time t j,k , queue length correlation l j,k ;

[0033] μ i ,i=1,2,3...6,μ i is the weight coefficient of each component of the reward function;

[0034] d is the distance between the emergency vehicle and the stop line of the intersection entrance lane.

[0035] Furthermore, in S2-4, the specific process includes:

[0036] S2-4-1: Initialize the neural network Q;

[0037] S2-4-2: The emergency vehicle observes the current state s from the environment t , using the ε-greedy strategy to explore and select action a in the neural network Q T , and get the next state s t+1 The reward value r corresponding to this action t ;

[0038] S2-4-3: Repeat step S2-4-2 to store the gained experience: (s t ,a t ,r t ,st+1 ), (s t+1 ,a t+1 ,r t+1 ,s t+2 ), …, build a playback buffer, extract data from the playback buffer for training, so as to reduce data correlation and improve data utilization. When the number of experience items in the playback buffer reaches the preset maximum value N, the oldest experience will be removed;

[0039] S2-4-4: Calculate the loss function L using the empirical data in the playback buffer in step S2-4-3 t :

[0040]

[0041] Among them, γ is the discount factor;

[0042] And use the Gradient Descent algorithm to update the neural network Q(s t ,a t ; ω) parameter ω:

[0043]

[0044] After updating the preset number of times, copy the updated ω to Q(s t+1 ,a;ω t ), where α is the learning rate of the GradientDescent algorithm;

[0045] S2-4-5: Repeat step S2-4-4 to continuously train the neural network Q(s,a;ω) until the neural network Q(s,a;ω) converges or reaches a preset number of training times to obtain a trained DQN model.

[0046] Furthermore, in S2-4-2, the ε-greedy strategy selects action a t The specific formula is:

[0047]

[0048] The meaning of the above formula is to generate a random number between 0 and 1 at each judgment node and compare it with ε to determine action a t The generation strategy has a probability of 1-ε to execute the maximum action a under the current neural network Q(s,a;ω) by using the behavior selection t , with probability ε, performs exploration behavior and randomly selects action a from the action space A t .

[0049] Furthermore, in S3, the specific process is:

[0050] When the emergency vehicle is not within the detection range of the intersection, the signal scheme of the intersection remains the normal traffic signal control (TSC-Normal) scheme;

[0051] When the emergency vehicle reaches the detection range of the intersection, the trained DQN model takes over the signal control scheme of the intersection: before the emergency vehicle passes through the intersection exit, the intersection signal control is in the signal priority (TSC-Priority) stage;

[0052] When the emergency vehicle passes through the intersection exit, the intersection signal control switches to the signal recovery (TSC-Recovery) stage.

[0053] A second aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the processor is used to execute a program in the memory, so as to implement the above-mentioned two-stage emergency signal priority control method based on reinforcement learning.

[0054] A third aspect of the present invention provides a storage medium containing computer executable instructions, which, when executed by a computer processor, is used to execute the two-stage emergency signal priority control method based on reinforcement learning as described above.

[0055] Compared with the prior art, the present invention has the following technical advantages:

[0056] The present invention proposes a two-stage emergency signal priority control method based on reinforcement learning. Based on V2I technology, traffic information in the intersection is collected for training the DQN model, and a signal priority control strategy is provided for emergency vehicles passing through the intersection. In the training process of the DQN model, the present invention is characterized in that the emergency signal priority control is divided into two stages: signal priority (TSC-Priority) and signal recovery (TSC-Recovery). The state space of the reinforcement learning algorithm includes both the emergency vehicle information (speed, position, etc.) and other social vehicle information in each entrance lane of the intersection; the action space includes each phase in the four-phase signal control scheme, and the reward function is designed in the form of a piecewise function, and whether the emergency vehicle passes through the intersection is used as the judgment condition. Among them, the reward function of the first stage (signal priority) consists of two parts: the priority right of emergency vehicles and the traffic efficiency of social vehicles; the reward function of the second stage (signal recovery) only includes the traffic efficiency of social vehicles. In the process of signal priority control, the present invention designs a control strategy based on the trained DQN model, which can provide signal priority for emergency vehicles according to the real-time traffic status collected by V2I, while reducing the delay impact on other social vehicles at the intersection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the method of the present invention;

[0058] Figure 2 This is a V2I-based traffic data collection diagram;

[0059] Figure 3 Training graph for the DQN model;

[0060] Figure 4 It is a flow chart for judging the traffic signal control stage;

[0061] Figure 5 The distance-time diagram for emergency vehicles to pass through four intersections;

[0062] Figure 6 This is a schematic diagram of the social vehicle delay situation for emergency vehicles passing through a single intersection;

[0063] Figure 7 Schematic diagram of average delay and delay trend of intersection under the control of the present invention. DETAILED DESCRIPTION

[0064] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Component models, material names, connection structures, control methods, algorithms and other features not clearly described in this technical solution are all considered to be common technical features disclosed in the prior art.

[0065] Example 1

[0066] The present invention proposes a two-stage emergency signal priority control method based on reinforcement learning, which comprises the following steps:

[0067] Step 1: Obtain historical traffic data information of the intersection based on V2I, including the number of vehicles on each entrance lane n j,k , speed v j,k,l , queue length l j,k , as training data for model training; in the DQN model control decision stage, obtain real-time traffic information of intersections based on V2I, including: the location of emergency vehicles T e , the speed of the emergency vehicle v e , the position of social vehicles at each entrance of the intersection T s , the number of social vehicles on each entrance of the intersection n j,k , the speed v of social vehicles at each entrance of the intersection j,k,l .like Figure 2 As shown in the figure, j = n, s, e, w, respectively represent the north entrance, south entrance, east entrance, and west entrance of the intersection. K = 1, 2, 3..., respectively represent the lane numbers of each entrance direction. L = 1, 2, 3..., respectively represent the numbers of private vehicles on each lane.

[0068] Step 2: Offline training of DQN model (training process as follows Figure 3 (indicated)

[0069] 2.1. Obtain the traffic status information of the emergency vehicle and the intersection through which the emergency vehicle will pass, as the state space S of the Markov decision process of the emergency vehicle passing through the intersection. The state space is the important factors that affect the training target abstracted from the environment, including: the distance d between the emergency vehicle and the stop line of the intersection entrance, the speed v of the emergency vehicle e , the number of social vehicles on each entrance of the intersection n j,k , the speed v of social vehicles at each entrance of the intersection j,k,l , the queue length of social vehicles at each entrance of the intersection l j,k ;

[0070] 2.2. Define the action space A of the Markov decision process of the emergency vehicle passing through the intersection as the signal phase of the intersection. The design is a traditional four-phase control, including: {east-west straight and right turn (EW), east-west left turn (EWL), north-south straight and right turn (NS), north-south left turn (NSL)}. The algorithm makes a judgment through the analysis of the state space and selects the most appropriate action to realize the dynamic control of the intersection signal scheme;

[0071] 2.3. Define the reward function of the Markov decision process for the emergency vehicle passing through the intersection. The reward is the feedback for the action taken, and the reward function will largely determine whether the training results meet the envisioned goal. In the present invention, the reward function is designed as R according to whether the emergency vehicle exits the intersection exit. E and R S The reward function R of different training objectives in the traffic signal priority control (TSC-Priority) stage and the traffic signal recovery control (TSC-Recovery) stage is:

[0072]

[0073] R E =μ 1 v e +μ 2 n stop +μ 3 t

[0074] R S =μ 4 n j,k +μ 5 t j,k +μ 6 l j,k

[0075] Among them, R Eis the reward function for the emergency vehicle action, and the speed v of the emergency vehicle e , the number of emergency vehicle stops n stop , and the waiting time t of the emergency vehicle. S is a reward function that reflects the efficiency of social vehicles, and the number of social vehicles n at each entrance of the intersection j,k , waiting time t j,k , queue length correlation l j,k .μ i , i=1,2,3...6, are the weight coefficients of each component of the reward function. d is the distance between the emergency vehicle and the stop line of the intersection entrance lane.

[0076] 2.4. Training the Neural Network Q(s,a;ω)

[0077] 2.4.1. Initialize the neural network Q, where the number of input neurons of the neural network Q is 5 and the number of output neurons is 4;

[0078] 2.4.2. The emergency vehicle observes the current state s from the environment t , using the ε-greedy strategy to explore and select action a in the neural network Q t , and get the next state s t+1 The reward value r corresponding to this action t ;

[0079] The ε-greedy greedy strategy selects action a t The specific formula is:

[0080]

[0081] The meaning of this formula is to generate a random number between 0 and 1 at each judgment node and compare it with ε to determine the action a. T The generation strategy has a probability of 1-ε to execute the maximum action a under the current neural network Q(s,a;ω) by using the behavior selection T , with probability ε, performs exploration behavior and randomly selects action a from the action space A T .

[0082] 2.4.3. Repeat step 2.4.2 to store the gained experience: (s T ,a T ,r t ,s T+1 ), (s t+1 ,a t+1 ,r t+1 ,s t+2), …, build a replay buffer Replay Buffer. The replay buffer is a module for storing experience data. Sampling data from the replay buffer for training can reduce data correlation and improve data utilization. When the number of experience items in the replay buffer reaches the preset maximum value N, the oldest experience will be removed;

[0083] 2.4.4. Calculate the loss function L using the empirical data in the playback buffer in step 2.4.3 t :

[0084]

[0085] Among them, γ is the discount factor.

[0086] And use the Gradient Descent algorithm to update the neural network Q(s t ,a t ; ω) parameter ω:

[0087]

[0088] After updating a certain number of times, copy the updated ω to Q(s t+1 ,a;ω t ). Among them, α is the learning rate of the GradientDescent algorithm.

[0089] 2.4.5. Repeat step 2.4.4 to continuously train the neural network Q(s,a;ω) until the neural network Q(s,a;ω) converges or reaches the desired number of training times, and obtain the trained DQN model;

[0090] Step 3: Use the trained DQN model to implement the control decision of the intersection signal scheme. When the emergency vehicle is not within the detection range of the intersection, the intersection signal scheme remains the general traffic signal control (TSC-Normal) scheme. When the emergency vehicle reaches the detection range of the intersection, the trained DQN model takes over the signal control scheme of the intersection: before the emergency vehicle passes through the intersection exit, the intersection signal control is the signal priority (TSC-Priority) stage; after the emergency vehicle passes through the intersection exit, the intersection signal control is converted to the signal recovery (TSC-Recovery) stage. The judgment process of the signal control stage is as follows: Figure 4 Indicates.

[0091] Figure 5The distance-time diagram of emergency vehicles passing through four intersections is shown. The figure shows the situation of emergency vehicles passing through four intersections 500 meters apart. In the figure, no signal priority control means that the signal control scheme of the intersection will not change due to the arrival of the emergency vehicle. Emergency vehicles are regarded as general social vehicles, so under this scheme, the traffic efficiency of emergency vehicles is low. Before approaching the intersection and arriving at the intersection, it is necessary to stop and wait (i.e., the part where the situation distance remains unchanged over time in the figure). Absolute priority signal control means that the signal scheme of the emergency vehicle before arriving at the intersection is not optimized as a whole, so it may encounter obstacles, slow down or stop and wait before arriving at the intersection. However, once the emergency vehicle arrives at the intersection, it will immediately get the green light to let it go, so there is no need to wait when arriving at the intersection, so that the traffic efficiency of the emergency vehicle is medium, but it brings great delays to social vehicles. The signal control method of the present invention will comprehensively consider the position, speed and number and speed of social vehicles on each import lane of the emergency vehicle. According to the position of the emergency vehicle, we will adjust the signal scheme to ensure that the social vehicles in front of the emergency vehicle are vacated in advance during the driving process, so that the emergency vehicle will not encounter obstacles before and when arriving at the intersection. Therefore, under this scheme, the distance-time graph in the figure is approximately a straight line, showing that the emergency vehicle has a high efficiency. Due to the setting of the piecewise reward function, this method can optimize the efficiency of social vehicles while considering the emergency vehicle before and after passing through the intersection. Figure 6 A schematic diagram of the delay of social vehicles when an emergency vehicle passes through a single intersection is shown. The figure shows the cumulative total delay of other social vehicles before and after the emergency vehicle passes through the intersection. No signal priority control means that the signal scheme is not adjusted and the emergency vehicle is not given priority. The control method of the present invention tilts the green light duration to give priority to emergency vehicles, so its impact on social vehicles is greater than that of the no signal priority control method. Therefore, the control method of the present invention causes the cumulative total delay of social vehicles to be greater than that of the no signal priority control method. Absolute priority control does not consider the position and speed of social vehicles, but forces emergency vehicles to have priority. Moreover, after the emergency vehicle passes through the intersection, the absolute priority control does not perform adaptive timing recovery of the chaotic state of the intersection. Therefore, under absolute priority control, the total delay of social vehicles is the highest, and the partial tangent slope after the emergency vehicle passes through the intersection is greater than that of the other two methods. The control method of the present invention recovers the chaotic situation of social vehicles after the emergency vehicle passes through the intersection. Therefore, the partial tangent slope after the emergency vehicle passes through the intersection is similar to that of the no signal priority control method.

[0092] Figure 7A schematic diagram of the average delay and delay trend of an intersection is shown. The figure describes the average delay of all vehicles at an intersection and its changing trend before and after an emergency vehicle passes through a single intersection, while taking into account multiple signal cycle cycles. In a complete signal cycle, the alternation of red and green signals causes the average delay of the intersection to increase and decrease. However, during the cycle in which the emergency vehicle passes, the average delay of the cycle increases overall due to the interference of the emergency vehicle, so the delay trend is upward. After the emergency vehicle passes, the signal recovery control stage of the present invention takes effect, and the average delay of the intersection gradually decreases. This is because the recovery control stage will adapt to the new traffic conditions and adopt appropriate signal adjustment strategies to improve the traffic efficiency of the intersection. Therefore, the overall trend of the average delay of the intersection shows a downward trend.

[0093] This embodiment also proposes a two-stage emergency signal priority control device based on reinforcement learning, which includes a processor and a memory, the processor and the memory are coupled, the memory stores program instructions, and the above-mentioned task management method is implemented when the program instructions stored in the memory are executed by the processor. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components; the memory may include a random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The memory can be an internal memory of the random access memory (RAM) type, and the processor and the memory can be integrated into one or more independent circuits or hardware, such as an application-specific integrated circuit (ASIC). It should be noted that the computer program in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0094] This embodiment also proposes a computer-readable storage medium, wherein the storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned two-stage emergency signal priority control method based on reinforcement learning. The storage medium can be an electronic medium, a magnetic medium, an optical medium, an electromagnetic medium, an infrared medium, or a semiconductor system or a propagation medium. The storage medium can also include semiconductor or solid-state memory, a magnetic tape, a removable computer disk, a random access memory (RAM), a read-only memory (ROM), a hard disk, and an optical disk. The optical disk can include a compact disk-read only memory (CD-ROM), a compact disk-read / write (CD-RW), and a DVD.

[0095] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. A two-stage emergency signal priority control method based on reinforcement learning, characterized in that: include: S1: Obtain historical traffic data information of intersections based on V2I; S2: Offline training of the DQN model based on the historical traffic data information, The reward function in the offline training DQN model is a piecewise function, and the judgment condition is whether the emergency vehicle passes through the intersection. The reward function in the first stage is composed of the priority right of way for the emergency vehicle and the traffic efficiency of social vehicles. The reward function in the second stage only includes the traffic efficiency of social vehicles. S3: Generate control decisions for intersection signal schemes using the trained DQN model; S2 specifically includes the following steps: S2-1: Obtain traffic status information of the emergency vehicle and the intersection through which the emergency vehicle will pass, as the state space S of the Markov decision process of the emergency vehicle passing through the target intersection; S2-2: Define the action space A of the Markov decision process of the emergency vehicle passing through the target intersection as the signal phase of the target intersection; S2-3: Construct the reward function of the Markov decision process for the emergency vehicle to pass through the target intersection; S2-4: Training the Neural Network ; In S2-1, the state space S includes the distance between the emergency vehicle and the stop line of the intersection entrance lane. , the speed of emergency vehicles , the number of social vehicles at each entrance of the intersection , the speed of social vehicles at each entrance of the intersection , the length of the queue of social vehicles at each entrance of the intersection ; In S2-2, the action space A includes east-west straight and right turn (EW), east-west left turn (EWL), north-south straight and right turn (NS), and north-south left turn (NSL); In S2-3, the specific process includes: The reward function is designed according to whether the emergency vehicle exits the intersection exit. and To cope with the traffic signal priority control stage and the traffic signal recovery control stage, the different training objective reward functions R are constructed as follows: ； ； ； in, is the reward function for measuring the emergency vehicle's actions, The speed of emergency vehicles , Number of stops for emergency vehicles , waiting time for emergency vehicles Related; is the reward function that reflects the efficiency of social vehicles, The number of social vehicles at each entrance of the intersection , waiting time , queue length related ; , is the weight coefficient of each component of the reward function; d is the distance between the emergency vehicle and the stop line of the intersection entrance lane.

2. A two-stage emergency signal priority control method based on reinforcement learning according to claim 1, characterized in that: In S1, the V2I-based historical traffic data information of the intersection includes: Historical traffic data information, including: the number of vehicles at each entrance ,speed , queue length ; Real-time traffic information at intersections, including the location of emergency vehicles , the speed of emergency vehicles , the location of social vehicles at each entrance of the intersection , the number of social vehicles at each entrance of the intersection , the speed of social vehicles at each entrance of the intersection , where j=n,s,e,w represent the north entrance, south entrance, east entrance, and west entrance of the intersection respectively, K=1,2,3... represent the lane numbers of each entrance direction respectively, and L=1,2,3... represent the numbers of private vehicles on each lane respectively.

3. The two-stage emergency signal priority control method based on reinforcement learning according to claim 1 is characterized in that: In S2-4, the specific process includes: S2-4-1: Initialize the neural network Q; S2-4-2: The emergency vehicle's current state based on observations from the environment ,use The greedy strategy explores and selects actions in the neural network Q , and get the next state The reward value corresponding to this action ; S2-4-3: Repeat step S2-4-2 to store the gained experience: , ,…,Build a replay buffer and extract data from the replay buffer for training to reduce data correlation and improve data utilization. When the number of experience items in the replay buffer reaches the preset maximum value N, the oldest experience will be removed; S2-4-4: Calculate the loss function using the empirical data in the playback buffer in step S2-4-3 : ； in, is the discount factor; And use the Gradient Descent algorithm to update the neural network Parameters : ； After updating the preset number of times, the updated Copy to ,in, is the learning rate of the Gradient Descent algorithm; S2-4-5: Repeat step S2-4-4 to continuously train the neural network , until the neural network The training ends when convergence or the preset number of training times is reached, and the trained DQN model is obtained.

4. A two-stage emergency signal priority control method based on reinforcement learning according to claim 3, characterized in that: In S2-4-2, Greedy strategy selects actions The specific formula is: ； The meaning of the above formula is to generate a random number between 0 and 1 at each judgment node and Compare and decide on actions The generation strategy is Probabilistic execution utilizes behavior selection to select the current neural network Next maximum action ,have The probability of performing exploration behavior is to randomly select actions from the action space A .

5. The two-stage emergency signal priority control method based on reinforcement learning according to claim 3 is characterized in that: In S3, the specific process is: When the emergency vehicle is not within the detection range of the intersection, the signal scheme of the intersection remains the normal traffic signal control (TSC-Normal) scheme; When the emergency vehicle reaches the detection range of the intersection, the trained DQN model takes over the signal control plan of the intersection: before the emergency vehicle passes the intersection exit, the intersection signal control enters the signal priority (TSC-Priority) stage; When the emergency vehicle passes through the intersection exit, the intersection signal control switches to the signal recovery (TSC-Recovery) stage.

6. An electronic device comprising a memory and a processor, characterized in that: The processor is used to execute the program in the memory to implement the two-stage emergency signal priority control method based on reinforcement learning as described in any one of claims 1 to 5.

7. A storage medium containing computer-executable instructions, characterized in that: The storage medium of the computer-executable instructions, when executed by a computer processor, is used to execute the two-stage emergency signal priority control method based on reinforcement learning as described in any one of claims 1 to 5.

Images