A Collaborative Control Method for Highway Merging Zones Based on Deep Reinforcement Learning

Abstract

This invention discloses a collaborative control method for highway merging zones based on deep reinforcement learning. A LiikeSim-Python co-simulation environment is established, and loop detectors are set up in the simulation environment to acquire traffic flow data upstream and downstream of the highway merging zone. An EM algorithm based on Gaussian mixture distribution is used as a traffic state classifier, taking the traffic flow data of the highway merging zone as input to classify the traffic state of the merging zone. A state space, action space, and reward function are designed. Using the state space of the highway merging zone as input and the actions of the variable speed limit agent and the ramp metering agent as output, a multi-agent shared experience network model under time-series characteristics is constructed. An independent experience pool is set up for the variable speed limit agent and the ramp metering agent, and the interaction experience between the agent and the traffic simulation environment is collected at the control cycle frequency. The agent model is trained using sampled data. The trained agent model is used to realize collaborative control of the highway merging zone. This invention can reduce travel delays in highway merging zones.

Description

Technical Field

[0001] This invention relates to the fields of deep reinforcement learning and traffic control, and specifically to a collaborative control method for merging zones on highways based on deep reinforcement learning. Background Technology

[0002] Merging zones on highways are areas where traffic flow is highly susceptible to bottlenecks, leading to congestion and frequent accidents. Existing traffic control methods, including variable speed limit control and ramp control, can significantly regulate traffic flow in ramp bottleneck areas and are the most effective control methods for alleviating highway congestion. With increasing traffic volume, traffic management and control in merging zones are becoming increasingly complex and critical. Traditional traffic control methods, such as fixed signal control and rule-based control, often struggle to respond to rapidly changing traffic conditions in real time, resulting in low traffic efficiency and increased vehicle delays.

[0003] Deep Reinforcement Learning (DRL), as a method combining the advantages of deep learning and reinforcement learning, has demonstrated excellent decision-making and adaptive capabilities in complex dynamic environments. Through continuous interaction with the environment, the agent can autonomously learn optimal strategies and adapt to environmental changes, making it particularly suitable for handling complex traffic problems with multiple variables and objectives. A DRL-based collaborative control method for highway merging zones, by sensing information such as traffic flow, vehicle speed, and queue length, allows the agent to adjust merging strategies in real time, such as speed control and signal scheduling, to optimally coordinate traffic flow in the merging zone. Compared to traditional methods, DRL is more flexible and adaptable, effectively reducing traffic congestion, improving road efficiency, and lowering traffic accident rates. By comprehensively utilizing advanced sensing technologies, communication technologies, and intelligent algorithms, DRL-based collaborative control systems represent an important direction for the future development of intelligent transportation. Summary of the Invention

[0004] The purpose of this invention is to propose a collaborative control method for highway merging zones based on deep reinforcement learning. On the one hand, it achieves collaborative control of variable speed limits and ramp metering through a shared reward mechanism. On the other hand, through the sharing of experience, the model can be effectively applied to highway corridors with multiple merging zones to achieve overall control optimization.

[0005] The technical solution to achieve the objective of this invention is: a collaborative control method for merging zones of highways based on deep reinforcement learning, comprising the following steps:

[0006] Step 1: Establish a LiikeSim-Python co-simulation environment based on the real road network environment and traffic flow data, and set up a loop detector in the simulation environment to obtain traffic flow data of the upstream and downstream of the merging zone of the highway;

[0007] Step 2: The EM algorithm based on Gaussian mixture distribution is used as the traffic state classifier. The traffic flow data of the highway merging zone is used as input to classify the traffic state of the highway merging zone.

[0008] Step 3: Based on the traffic flow data of the upstream and downstream of the merging zone obtained by the coil detector in Step 1 and the traffic state of the merging zone obtained by the traffic state classifier in Step 2, design the state space, action space and reward function.

[0009] Step 4: Using the state space of the merging zone of the highway in Step 3 as input and the actions of the variable speed limit agent and the ramp metering agent as output, construct a network model of multi-agent shared experience under time-series features. This model includes an LSTM time-series feature fusion module for extracting time-series features of highway traffic flow and a D3QN decision module for outputting agent actions.

[0010] Step 5: Set up an independent experience pool for the variable speed limit and ramp metering agents respectively. The interaction experience between the agent and the traffic simulation environment is collected using the control cycle in step 3 as the frequency.

[0011] Step 6: Randomly sample from the experience pool according to the set batch size, and use the sampled samples to train the agent model, including the variable speed limit agent and the ramp metering agent.

[0012] Step 7: Repeat steps 5 and 6 until the reward reaches a convergent state, and save the agent model parameters. Use the trained agent model to achieve coordinated control of the highway merging zone.

[0013] Further, in step 1, a LiikeSim-Python co-simulation environment is established based on the real road network environment and traffic flow data. Loop detectors are then set up within the simulation environment to acquire traffic flow data upstream and downstream of the highway merging zone. This specifically includes the following steps:

[0014] Step 1-1: Use the LiikeSim visualization interface to draw the highway merging zone road network. Based on real traffic flow data, set the traffic flow information in the road network, including maximum vehicle speed, maximum acceleration / deceleration, vehicle path, and departure time. Furthermore, install loop detectors at the end of the upstream section of the highway merging zone, the end of the variable speed limit section, ramp entrances, the merging zone itself, and the downstream section of the merging zone, setting the detection cycle to 60 seconds. See the schematic diagram below. Figure 2As shown, save it as a Scenario.xml road network file;

[0015] Step 1-2: Based on the road network file and application programming interface in Step 1-1, establish the LiikeSim-Python co-simulation environment, and acquire traffic flow data upstream and downstream of the highway merging zone using loop detectors set at fixed locations in Step 1-1, including the traffic flow q of the upstream road segment of the merging zone. in The flow rate q at the ramp entrance r Ramp vehicle queue length w r The flow rate q in the downstream section of the merging zone out The flow rate q in the merging zone m The average vehicle density ρ in the merging zone m and the average vehicle speed v in the merging zone m The average vehicle density ρ in the merging zone m The calculation formula is as follows; all other traffic flow data can be directly obtained from the detector.

[0016]

[0017] Among them, the average vehicle density ρ in the merging zone m The unit is veh / km, N is the number of vehicles in the merging zone, and L is the length of the merging zone in meters.

[0018] Further, in step 2, the EM algorithm based on Gaussian mixture distribution is used as the traffic state classifier. Using traffic flow data from the merging zone of the highway as input, the traffic state of the merging zone is classified. This specifically includes the following steps:

[0019] Step 2-1: A traffic state classifier is constructed using the EM algorithm based on Gaussian mixture distribution, with the flow rate q of the merging zone obtained by the loop detector at each time step. m The average vehicle density ρ in the merging zone m The average vehicle speed v in the merging zone m The input is the traffic state Q of the merging zone at each time step, and the output is the traffic state Q of the merging zone at each time step. The traffic state of the merging zone is divided into three categories: smooth, moderately congested, and congested.

[0020] Step 2-2, the historical merging zone flow rate q collected in the simulation environment m The average vehicle density ρ in the merging zone m The average vehicle speed v in the merging zone m The traffic state classifier is trained using the training set; for the trained traffic state classifier, the flow rate q of the merging zone collected in the simulation environment is input in real time. m The average vehicle density ρ in the merging zone mThe average vehicle speed v in the merging zone m That is, to obtain the traffic status Q of the merging zone at the current time step.

[0021] Furthermore, in step 3, based on the traffic flow data upstream and downstream of the merging zone obtained by the loop detector in step 1 and the traffic state of the merging zone obtained by the traffic state classifier in step 2, the state space, action space, and reward function are designed, specifically as follows:

[0022] (1) State space S

[0023] The state space consists of a one-dimensional vector s = [q in ,q r ,w r ,q out ,q m ,ρ m ,v m [q, Q] represents the traffic flow status in the merging zone of the highway, where [q in ,q r ,w r ,q out ,q m ,ρ m ,v m [Q] represents the traffic flow data upstream and downstream of the merging zone of the highway, and Q represents the traffic state of the merging zone obtained by the traffic state classifier.

[0024] (2) Action space

[0025] The actions of the variable speed limit and ramp metering agent are set as discrete speed limits for variable speed limit road sections and green light phase durations at ramp entrances, respectively. The speed limit for variable speed limit road sections is set to [60, 65, 70, 75, 80, 85, 90, 100, 110, 120] km / h, and the green light phase duration at ramp entrances is set to [6, 12, 18, 24, 30, 36, 42, 48, 54, 60] seconds. The control cycles of both the variable speed limit and ramp metering agent are set to 60 seconds and kept synchronized.

[0026] (3) Reward function r

[0027] The variable speed limit and ramp metering agent within the same merging zone share a reward, which is determined by the average vehicle speed in the merging zone and the queue length on the ramp. The reward function is designed as follows: Where ω1 and ω2 are the weighting parameters for average vehicle speed and ramp vehicle queue length, These represent the ramp queue length and the ideal ramp queue length, respectively. m v is the average speed of the vehicles in the merging zone. vsl This represents the average speed of vehicles on road sections with variable speed limits.

[0028] Further, in step 4, using the state space of the highway merging zone from step 3 as input and the actions of the variable speed limit agent and the ramp metering agent as output, a multi-agent shared experience network model under temporal features is constructed. This model includes an LSTM temporal feature fusion module for extracting temporal features of highway traffic flow and a D3QN decision module for outputting agent actions. Specifically, it includes the following steps:

[0029] Step 4-1: Construct a temporal feature fusion module based on LSTM long short-term memory network. Input the historical temporal features of traffic flow in the merging zone of the highway into the temporal feature fusion module. Through input gate, forget gate and output gate, the module effectively manages the historical temporal features of traffic flow in the merging zone and captures the long and short-term dependencies. Then, output the traffic flow features that fuse the historical temporal features. At the same time, set the length of the input traffic flow historical temporal features to len.

[0030] Step 4-2: Construct a decision module for the variable speed limit and ramp metering agent based on the D3QN deep reinforcement learning algorithm. The decision module takes the output of the temporal feature fusion module in Step 4-1 as input and introduces a duel network to separate state values ​​and action advantages. The duel network has two branches: a state value network and an advantage network. The advantage network consists of two fully connected layers connected in series, with sizes of 256×256 and 256×10, respectively. The state value network consists of fully connected layers connected in series, with sizes of 256×256 and 256×1, respectively. The state-action value is the average difference of the advantage network output and the sum of the difference with the state value network output, i.e., Q(s,a;θ)=V(s;θ)+A(s,a;θ)-mean a A(s,a;θ), where Q(s,a;θ) represents the agent's state-action value function, V(s;θ) represents the agent's state-value network output, and A(s,a;θ) and mean a A(s,a;θ) represent the dominant network output and the average value of the dominant network output of the agent, respectively; then, the road segment speed limit and the green light phase duration are selected as the output strategy of the decision module according to the ε-greedy strategy.

[0031]

[0032] Where r is a random number that follows a uniform distribution, i.e. a i Q represents the output strategy of agent i, namely the road segment speed limit and the duration of the green light phase. i (s,a i ;θ i Let be the state-action value function of agent i. Let i be the action space of agent i, where i = 1 and 2 represent variable speed limit and ramp metering agents, respectively.

[0033] Furthermore, in step 5, an independent experience pool is set up for both the variable speed limiter and the ramp metering agent. The interaction experience between the agent and the traffic simulation environment is collected using the control cycle in step 3 as the frequency, specifically:

[0034] Create two Python lists as experience pools for variable speed limits and ramp metering, respectively. Each element in the list represents a state transition. <s,a i ,s next ,r,T>, where s represents the traffic flow state in the merging zone of the highway, a i s represents the action selected by agent i based on the ε-greedy policy in step 4-2. next This represents the traffic flow state of the highway merging zone in the next control cycle after the agent executes its action. r is the reward calculated for the next state, and T is the end-of-round flag, defined as follows: T = 1 indicates the round has ended, and T = 0 indicates the round has not yet ended. The variable speed limit and ramp metering agents interact with the simulation environment to generate experience sequences. <s,a 1 ,s next ,r,T> and <s,a 2 ,s next Each of the following sets of experience sequences is cached in its respective experience pool: ,r,T>. In this context, i = 1, 2, if the number of experiences exceeds the capacity of the experience pool, the old experiences will be overwritten and discarded.

[0035] Further, step 6 involves randomly sampling from the experience pool according to the set batch size, and using the sampled samples to train the agent model, including the variable speed limit agent and the ramp metering agent. This specifically includes the following steps:

[0036] Step 6-1, in the experience pools of the variable speed limiter and ramp metering agents respectively. A batch of empirical samples is randomly selected according to the set fusion time sequence length len as the input to each agent network, and the target value function of each agent is calculated respectively. and loss function And according to the loss function L i Update the parameters of the evaluation network for each agent, where i = 1, 2 represent the variable speed limit agent and the ramp metering agent, respectively, and θ i To evaluate the network parameters, The parameters of the target network are set, with γ as the discount factor set to 0.98; every certain number of steps... The target network parameters of the variable speed limit and ramp metering agents are softly updated, i.e. τ is the update scaling factor, set to 0.005;

[0037] Step 6-2: Update the agent's greedy coefficient ε each round, and the update formula for the greedy coefficient ε is as follows: Where ε0 is the initial greedy coefficient, ε end This indicates that the greedy coefficient ε converges to ε. end D is the attenuation coefficient, set to 90, and epsiode represents the current simulation round number.

[0038] A collaborative control method for merging areas on highways based on deep reinforcement learning is proposed. The method is implemented to achieve collaborative control of merging areas on highways based on deep reinforcement learning.

[0039] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned deep reinforcement learning-based highway merging zone collaborative control method to achieve highway merging zone collaborative control based on deep reinforcement learning.

[0040] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the aforementioned deep reinforcement learning-based highway merging zone cooperative control method is implemented, thereby realizing highway merging zone cooperative control based on deep reinforcement learning.

[0041] Compared with the prior art, the significant advantages of this invention are: it utilizes the microscopic traffic simulation software LikeSim and its application programming interface to build a LikeSim-Python co-simulation environment, and designs a multi-agent reinforcement learning algorithm that shares experience, which enables efficient training of the algorithm model. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the highway cooperative control method based on deep reinforcement learning according to the present invention.

[0043] Figure 2 This is a schematic diagram of the placement of the coil detector in this invention.

[0044] Figure 3 This is the curve showing the relationship between the reward function and the vehicle queue length in this invention.

[0045] Figure 4 This is the time-series feature fusion module of the present invention.

[0046] Figure 5 This is the D3QN network structure of the present invention.

[0047] Figure 6 This is the pseudocode of the control algorithm of this invention.

[0048] Figure 7 This is a LiikeSim road network scenario according to an embodiment of the present invention.

[0049] Figure 8 These are the relevant comparison results of the embodiments of the present invention. Detailed Implementation

[0050] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0051] This invention provides a collaborative control method for merging zones on highways based on deep reinforcement learning. A schematic diagram of the overall control method is shown below. Figure 1 As shown, the control scheme includes traffic state division and the collaborative operation of a variable speed limit agent and a ramp metering agent. The variable speed limit agent and the ramp metering agent acquire traffic flow state data from the highway merging zone through the LiikeSim simulation environment, using this data as model input. Their outputs are the speed limit for the variable speed limit section and the duration of the green light phase at the ramp entrance, respectively. Through an application programming interface, the two agents can effectively manage the traffic flow on the simulation platform in real time.

[0052] Step 1: Establish a LiikeSim-Python co-simulation environment based on the real road network environment and traffic flow data, and set up loop detectors in the simulation environment to acquire traffic flow data upstream and downstream of the highway merging zone. This includes the following steps:

[0053] Step 1-1: Use the LiikeSim visualization interface to draw the highway merging zone road network. Based on real traffic flow data, set the traffic flow information in the road network, including maximum vehicle speed, maximum acceleration / deceleration, vehicle path, and departure time. Furthermore, install loop detectors at the end of the upstream section of the highway merging zone, the end of the variable speed limit section, ramp entrances, the merging zone itself, and the downstream section of the merging zone, setting the detection cycle to 60 seconds. See the schematic diagram below. Figure 2 As shown, save it as a Scenario.xml road network file.

[0054] Step 1-2: Based on the road network file and application programming interface in Step 1-1, establish the LiikeSim-Python co-simulation environment, and acquire traffic flow data upstream and downstream of the highway merging zone using loop detectors set at fixed locations in Step 1-1, including the traffic flow q of the upstream road segment of the merging zone. in The flow rate q at the ramp entrance r Ramp vehicle queue length w r The flow rate q in the downstream section of the merging zone out The flow rate q in the merging zone m The average vehicle density ρ in the merging zone m and the average vehicle speed v in the merging zone m The average vehicle density ρ in the merging zone m The calculation formula is as follows; all other traffic flow data can be directly obtained from the detector.

[0055]

[0056] Among them, the average vehicle density ρ in the merging zone m The unit is veh / km, N is the number of vehicles in the merging zone, and L is the length of the merging zone in meters.

[0057] Step 2: The EM algorithm based on Gaussian mixture distribution is used as the traffic state classifier. Traffic flow data from highway merging zones is used as input to classify the traffic states of these zones, serving as additional features in the state space of the reinforcement learning algorithm. This includes the following steps:

[0058] Step 2-1: Construct a traffic state classifier using the EM algorithm based on Gaussian mixture distribution, using the flow rate q of the merging zone obtained by the loop detector at each time step in Step 1-2. m The average vehicle density ρ in the merging zone m The average vehicle speed v in the merging zone m The input is , and the output is the traffic state Q of the merging zone at each time step. The traffic state of the merging zone is divided into 3 categories: smooth, moderately congested, and congested.

[0059] Step 2-2, the historical merging zone flow rate q collected in the simulation environment m The average vehicle density ρ in the merging zone m The average vehicle speed v in the merging zone m The traffic state classifier is trained using the training set; for the trained traffic state classifier, the flow rate q of the merging zone collected in the simulation environment is input in real time. m The average vehicle density ρ in the merging zone m The average vehicle speed v in the merging zone mThis will give you the traffic status Q of the merging zone at the current time step.

[0060] Step 3: Based on the traffic flow data upstream and downstream of the merging zone obtained by the loop detector in Step 1 and the traffic state of the merging zone obtained by the traffic state classifier in Step 2, design the state space, action space, and reward function:

[0061] (1) State space S

[0062] The state space consists of a one-dimensional vector s = [q in ,q r ,w r ,q out ,q m ,ρ m ,v m [q, Q] represents the traffic flow status in the merging zone of the highway, where [q in ,q r ,w r ,q out ,q m ,ρ m ,v m ] represents the traffic flow data upstream and downstream of the merging zone of the highway obtained in steps 1-2, and Q represents the traffic state of the merging zone obtained by the traffic state classifier in step 2.

[0063] (2) Action space

[0064] The actions of the variable speed limit and ramp metering intelligent agents are set as discrete speed limits (km / h) for variable speed limit road sections and green light phase duration (s) for ramp entrances. The speed limit for variable speed limit road sections is set to [60,65,70,75,80,85,90,100,110,120] km / h, and the green light phase duration for ramp entrances is set to [6,12,18,24,30,36,42,48,54,60] seconds. The control cycle of both the variable speed limit and ramp metering intelligent agents is set to 60 seconds and kept synchronized.

[0065] (3) Reward function r

[0066] The variable speed limit and ramp metering agent within the same merging zone share a reward, which is determined by the average vehicle speed in the merging zone and the queue length on the ramp. The reward function is designed as follows: Where ω1 and ω2 are the weighting parameters for average vehicle speed and ramp vehicle queue length, These represent the ramp queue length and the ideal ramp queue length, respectively. m v is the average speed of the vehicles in the merging zone. vsl For the average speed of a variable speed limit road segment, the relationship between the reward function and the queue length of vehicles on the ramp is as follows: Figure 3 As shown, the reward function takes into account the average speed of vehicles in the merging zone and the queue length of vehicles on the ramp, and then ensures smooth and efficient traffic flow in the ramp area through the coordinated control of variable speed limits and entrance ramps.

[0067] Step 4: Using the state space of the merging zone on the highway in Step 3 as input, and the actions of the variable speed limit agent and the ramp metering agent as output, construct a network model for multi-agent shared experience under temporal features. This model includes an LSTM temporal feature fusion module for extracting temporal features of highway traffic flow, and a D3QN decision module for outputting agent actions. Specifically, it includes the following steps:

[0068] Step 4-1: Construct a temporal feature fusion module based on an LSTM (Long Short-Term Memory) network, such as... Figure 4 As shown, the historical time-series features of traffic flow in the merging zone of the highway are input into the time-series feature fusion module. Through the input gate, forget gate and output gate, the historical time-series features of traffic flow in the merging zone are effectively managed and the long-short-term dependencies are captured. Then, the traffic flow features fused with the historical time-series features are output. Here, the length of the input historical time-series features is set to len.

[0069] Step 4-2: Construct a decision-making module for the variable speed limit and ramp metering agent based on the D3QN deep reinforcement learning algorithm. The decision-making module takes the output of the temporal feature fusion module from Step 4-1 as input, and introduces a duel network to separate state values ​​and action advantages. The duel network has two branches: a state value network and an advantage network, as shown below. Figure 5 As shown, the dominance network consists of two fully connected layers connected in series, with sizes of 256×256 and 256×10, respectively. The state value network consists of two fully connected layers connected in series, with sizes of 256×256 and 256×1, respectively. The state-action value is the sum of the average difference between the outputs of the dominance network and the output of the state value network, i.e., Q(s,a;θ)=V(s;θ)+A(s,a;θ)-mean a A(s,a;θ), where Q(s,a;θ) represents the agent's state-action value function, V(s;θ) represents the agent's state-value network output, and A(s,a;θ) and mean a A(s,a;θ) represent the dominant network output and the average value of the dominant network output, respectively. Then, the road segment speed limit and green light phase duration are selected as the output strategy of the decision module according to the ε-greedy strategy shown in the following formula.

[0070]

[0071] Where r is a random number that follows a uniform distribution, i.e. a iQ represents the output strategy of agent i, namely the road segment speed limit and the duration of the green light phase. i (s,a i ;θ i Let be the state-action value function of agent i. Let i be the action space of agent i, where i = 1 and 2 represent variable speed limit and ramp metering agents, respectively.

[0072] Step 5: Set up an independent experience pool for the variable speed limit and ramp metering agents respectively. The interaction experience between the agent and the traffic simulation environment is collected using the control cycle in step 3 as the frequency.

[0073] Create two Python lists as experience pools for variable speed limits and ramp metering, respectively. Each element in the list represents a state transition. <s,a i ,s next ,r,T>, where s represents the traffic flow state in the merging zone of the highway, a i s represents the action selected by agent i based on the ε-greedy policy in step 4-2. next This represents the traffic flow state of the highway merging zone in the next control cycle after the agent executes its action. r is the reward calculated for the next state, and T is the end-of-round flag, defined as follows: T = 1 indicates the round has ended, and T = 0 indicates the round has not yet ended. The variable speed limit and ramp metering agents interact with the simulation environment to generate experience sequences. <s,a 1 ,s next ,r,T> and <s,a 2 ,s next Each of the following sets of experience sequences is cached in its respective experience pool: ,r,T>. In this context, i = 1, 2, if the number of experiences exceeds the capacity of the experience pool, the old experiences will be overwritten and discarded.

[0074] Step 6: Randomly sample from the experience pool according to the set batch size, and use the sampled samples to train the agent model. This includes the following steps:

[0075] Step 6-1, the pseudocode for the training process of the variable speed limit and ramp metering agent network is as follows: Figure 6 As shown, the experience pools of the variable speed limit and ramp metering agents are respectively... In step 4-1, a batch of empirical samples is randomly selected as the input to each agent network according to the fusion time sequence length len. The target value function of each agent is then calculated. and loss function And according to the loss function L iUpdate the parameters of the evaluation network for each agent, where i = 1, 2 represent the variable speed limit agent and the ramp metering agent, respectively, and θ i To evaluate the network parameters, The parameters of the target network are defined as follows: γ is the discount factor set to 0.98; the target network parameters of the variable speed limit and ramp metering agents are softly updated every certain number of steps c, i.e. τ is the update scaling factor, which is set to 0.005 here.

[0076] Step 6-2: Update the agent's greedy coefficient ε each round, and the update formula for the greedy coefficient ε is as follows: Where ε0 is the initial greedy coefficient, ε end This indicates that the greedy coefficient ε converges to ε. end D is the attenuation coefficient, set to 90, and epsiode represents the current simulation round number.

[0077] Step 7: Repeat steps 5 and 6 until the reward reaches a convergent state, and save the model parameters. Use the trained model to achieve coordinated control of the highway merging zone.

[0078] Example

[0079] To verify the effectiveness of the present invention, the following simulation experiment was conducted.

[0080] In this embodiment, the traffic simulation environment is built using the microscopic traffic simulation software LikeSim. LikeSim can be used to study traffic flow characteristics, evaluate traffic management strategies, and many other fields. Meanwhile, the application programming interface (API) in the Python environment can be used to achieve dynamic interaction with the LikeSim simulation environment, such as the acquisition of vehicle data and lane data, traffic light phase control, and road speed limit control, thereby realizing the joint simulation of LikeSim-Python.

[0081] In this embodiment, the simulation scenario is a highway merging zone road network scenario built based on a real road network, as shown in the example below. Figure 7 As shown in Table 1, and based on real traffic flow data as the vehicle input for the simulation environment, this embodiment includes 7200 seconds of traffic flow data, as shown in Table 1. To effectively evaluate the control effect of this invention, the average vehicle speed in the merging zone is used as the evaluation index.

[0082] Table 1 Traffic Demand Flow

[0083]

[0084] In this embodiment, the traffic control unit consists of a variable speed limit and a ramp metering agent. The traffic control unit acquires the traffic flow status s upstream and downstream of the highway merging zone in real time and selects a speed limit a.1 and the duration of the green phase of the traffic light a 2 Speed ​​limit information is displayed by marking variable speed limit zones. 1 and change the duration of the traffic light phase a 2 It can significantly alter the traffic flow state s' within the current control cycle of the highway merging zone, while the variable speed limit and ramp metering agents receive immediate rewards r from the environmental state s' and cache experience sequences. <s,a 1 ,r,s',T> and <s,a 2 The variable speed limit and ramp metering intelligent agent network learns from experience and training parameters through continuous interaction with the environment. As the network parameters θ... i As the update count i→∞, the network's action value evaluation function Q(s,a; θ) i Approximates the optimal action value evaluation function Q infinitely. * (s,a), i.e., Q(s,a; θ) i →Q * (s,a) Finally, the trained variable speed limit and ramp metering agent network model can be loaded, and the optimal action policy, a, can be obtained through a greedy strategy. * =argmax a Q * (s,a).

[0085] This embodiment trains the above method through simulation five times, with each training session lasting 300 rounds and each simulation round lasting 7200 seconds. Finally, the average of the five training results is used as the final evaluation of the scheme's effectiveness. To verify the effectiveness of this scheme, this embodiment is compared with no control strategy, fixed timing control, and the PI_ALINEA control method. The relevant comparison results are as follows: Figure 8 As shown in the figure. Therefore, it can be concluded that this scheme demonstrates a more significant advantage over traditional control methods in improving traffic efficiency in highway merging zones.

[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A highway merging area cooperative control method based on deep reinforcement learning, characterized in that, Comprise the following steps: Step 1, according to the real road network environment and traffic flow data to establish LiikeSim-Python joint simulation environment, and set the coil detector in the simulation environment to obtain the traffic flow data of the upstream and downstream of the highway merging area; Step 2, using EM algorithm based on Gaussian mixture distribution as traffic state classifier, taking the traffic flow data of the highway merging area as input, the traffic state of the highway merging area is divided; Step 3, according to the traffic flow data of the upstream and downstream of the highway merging area obtained by the coil detector in step 1 and the traffic state of the merging area obtained by the traffic state classifier in step 2, the state space, action space and reward function are designed; Step 4, taking the state space of the highway merging area in step 3 as input, and the action of the variable speed limit agent and the ramp metering agent as output, a network model of multi-agent shared experience under time sequence characteristics is constructed, including LSTM time sequence feature fusion module for extracting highway traffic flow time sequence features, and D3QN decision module for outputting agent action; Step 5, set a separate experience pool B for variable speed limit and ramp metering agent respectively i And collect the interaction experience of agent and traffic simulation environment with the control cycle in step 3 as the frequency; Step 6, according to the set batch size, randomly sample from the experience pool, and train the agent model using the sampled samples, including variable speed limit agent and ramp metering agent; Step 7, repeat step 5 and step 6 until the reward reaches the convergence state, and save the agent model parameters, and realize the cooperative control of the highway merging area by using the trained agent model; Wherein, Step 3, according to the traffic flow data of the upstream and downstream of the highway merging area obtained by the coil detector in step 1 and the traffic state of the merging area obtained by the traffic state classifier in step 2, the state space, action space and reward function are designed, specifically: (1) State space S The state space is represented by a one-dimensional vector s = [q in , q r , w r , q out , q m , p m , v m , Q] to represent the traffic flow state of the freeway merging area, wherein [q in , q r , w r , q out , q m , p m , v m ] are the traffic flow data obtained on the upstream and downstream of the freeway merging area, Q is the traffic state of the merging area obtained by the traffic state classifier, q in is the flow of the upstream road section of the merging area, q r is the flow at the ramp entrance, w r is the queue length of the ramp vehicle, q out is the flow of the downstream road section of the merging area, q m is the flow of the merging area, p m is the average vehicle density of the merging area, and v m is the average vehicle speed of the merging area; (2) Action space The actions of variable speed limit and ramp metering agent are set as discrete speed limit of variable speed limit section and green light phase duration of ramp entrance, the speed limit of variable speed limit section is set as [60, 65, 70, 75, 80, 85, 90, 100, 110, 120] km / h, the green light phase duration of ramp entrance is set as [6, 12, 18, 24, 30, 36, 42, 48, 54, 60] seconds, and the control period of variable speed limit and ramp metering agent is set as 60 seconds and keeps synchronization; (3) Reward function r The variable speed limit and ramp metering agent of the same merging area share the reward, which is determined by the average speed of the merging area and the queue length of the ramp vehicle, and the reward function is designed as where ω1, ω2 are weight parameters of the average speed and the queue length of the ramp vehicle, w r , respectively represent the queue length of the ramp vehicle and the ideal queue length of the ramp vehicle, v m is the average speed of the merging area, v vsl is the average speed of the variable speed limit section; Step 4, taking the state space of the highway merging area in step 3 as input, and the action of the variable speed limit agent and the ramp metering agent as output, a network model of multi-agent shared experience under time sequence characteristics is constructed, including LSTM time sequence feature fusion module for extracting highway traffic flow time sequence features, and D3QN decision module for outputting agent action, specifically including the following steps: Step 4-1, a time sequence feature fusion module is constructed based on an LSTM long short-term memory network, the traffic flow historical time sequence features of the merging area of the expressway are input into the time sequence feature fusion module, the input gate, the forgetting gate and the output gate are used to realize effective management and long short-term dependence capture of the traffic flow historical time sequence features of the merging area, and then traffic flow features fused with the historical time sequence features are output, and the length of the input traffic flow historical time sequence features is set as len; Step 4-2, a decision module of the variable speed limit and ramp metering agent is constructed based on the D3QN deep reinforcement learning algorithm, the decision module takes the output of the time sequence feature fusion module in step 4-1 as input, and separates the state value and action advantage by introducing a duel network, the duel network has two branches of state value network and advantage network, wherein the advantage network is composed of two fully connected layers in series with sizes of 256*256 and 256*10 respectively, and the state value network is composed of fully connected layers in series with sizes of 256*256 and 256*1 respectively, the state action value is obtained by averaging the output of the advantage network and taking the difference, and then summing with the output of the state value network, that is, Q(s,a; θ) = V(s; θ) + A(s,a; θ) - mean a A(s,a; θ), wherein Q(s,a; θ) represents the state action value function of the agent, V(s; θ) represents the output of the state value network of the agent, A(s,a; θ) and mean a A(s,a; θ) represent the output of the advantage network of the agent and the average value of the output of the advantage network respectively; subsequently, the road section speed limit and green light phase duration are selected as the output strategy of the decision module according to the ε-greedy strategy; where r is a random number obeying uniform distribution, i.e. a i represents the output policy of agent i, i.e. the speed limit and green phase duration, Q i (s,a i ; θ i ) is the state-action value function of agent i, is the action space of agent i, where i = 1, 2 represents the variable speed limit and ramp metering agent, respectively.

2. The highway merging area cooperative control method based on deep reinforcement learning according to claim 1, characterized in that, Step 1, a LiikeSim-Python joint simulation environment is established according to a real road network environment and traffic flow data, and a coil detector is set in the simulation environment to obtain traffic flow data upstream and downstream of the merging area of the expressway, and the specific steps include the following steps: Step 1-1, a visual interface of LiikeSim is used to draw the road network of the merging area of the expressway, vehicle flow information in the road network is set according to real traffic flow data, including maximum speed, maximum acceleration / deceleration, vehicle path and departure time, further, coil detectors are set at the end of the upstream road section of the merging area of the expressway, the end of the variable speed limit road section, the ramp entrance, the merging area and the downstream road section of the merging area, and the detection period is set as 60 seconds, and the Scenario.xml road network file is saved; Step 1-2, according to the road network file and the application program interface in step 1-1, a LiikeSim-Python joint simulation environment is established, and traffic flow data upstream and downstream of the merging area on the expressway is obtained through the coil detector set at a fixed position in step 1-1, wherein the average vehicle density ρ of the merging area is 0.5 m The calculation formula is as follows, and the remaining traffic flow data is directly obtained by the detector; wherein the average vehicle density p of the merging area m in veh / km, N is the number of vehicles in the merging area, and L is the length of the merging area in meters.

3. The highway merging area cooperative control method based on deep reinforcement learning according to claim 1, characterized in that, Step 2, an EM algorithm based on Gaussian mixture distribution is used as a traffic state classifier, traffic flow data of the merging area of the expressway is input, and traffic states of the merging area of the expressway are divided, and the specific steps include the following steps: Step 2-1, build the traffic state classifier using EM algorithm based on Gaussian Mixture Distribution, taking the flow q of the merge area obtained by the coil detector at each time step as input m , the average vehicle density p of the merge area m , and the average speed v of the merge area m as input, and the traffic state Q of the merge area corresponding to each time step as output, where the traffic state of the merge area is divided into 3 categories, including smooth, relatively congested, and congested; Step 2-2, the flow rate q of the merging area collected in the simulation environment m , the average vehicle density p of the merging area m , and the average vehicle speed v of the merging area m The traffic state classifier is trained as a training set; for the trained traffic state classifier, the flow rate q of the merging area collected in the simulation environment is input in real time m , the average vehicle density p of the merging area m , and the average vehicle speed v of the merging area m , that is, the traffic state Q of the merging area at the current time step is obtained.

4. The highway merging area cooperative control method based on deep reinforcement learning according to claim 1, characterized in that, Step 5, set a separate experience pool for variable speed limit and ramp metering agent respectively And collect the interaction experience of agent and traffic simulation environment with the control cycle in step 3 as the frequency, specifically: Two python lists are created as experience pools for variable speed limit and ramp metering respectively, each element in the list represents a state transition <s, a i ,s next ,r, T>, where s represents the traffic flow state of the freeway merging area, a i represents the action selected by agent i in step 4-2 based on the ε-greedy strategy, s next represents the traffic flow state of the freeway merging area in the next control period after the agent performs the action, r is the reward calculated for the next state, and T is the end flag of the round. The end flag is defined as: when T is 1, it means the round is over; when T is 0, it means the round has not ended. The variable speed limit and ramp metering agents interact with the simulation environment to generate experience sequences <s, a 1 ,s next ,r, T> and <s, a 2 ,s next ,r, T> respectively, and cache the experience sequences in their respective experience pools , i = 1, 2. If the number of experiences exceeds the capacity of the experience pool, the old experiences will be overwritten and discarded.

5. The highway merging area cooperative control method based on deep reinforcement learning according to claim 4, characterized in that, Step 6, samples are randomly sampled from the experience pool according to the set batch size, and the sampled samples are used to train the agent model including the variable speed limit agent and the ramp metering agent, and the specific steps include the following steps: Step 6-1, respectively, in the variable speed limit and ramp metering agent's experience pool β i A batch of experience samples are randomly extracted as the input of each agent network according to the set fusion time sequence length len in the middle And the loss function And the loss function L i Update the parameters of the evaluation network of each agent, where i=1,2 respectively represent the variable speed limit and ramp metering agent, θ i The parameter of the evaluation network, The parameter of the target network, and γ is the discount factor set to 0.98; every certain number of steps Softly update the target network parameters of the variable speed limit and ramp metering agent, that is τ is the update proportion factor, set to 0.005; Step 6-2, the greed coefficient ε of the agent is updated every round, and the update formula of the greed coefficient ε is wherein ε0 is the initial greed coefficient, ε end represents that the greed coefficient ε converges to ε end , D is the attenuation coefficient, which is set to 90, and epsiode represents the number of rounds of the current simulation.

6. A highway merging area cooperative control system based on deep reinforcement learning, characterized in that, The deep reinforcement learning based expressway merging area cooperative control method according to any one of claims 1-5 is implemented, and deep reinforcement learning based expressway merging area cooperative control is realized. 7.A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, when the processor executes the computer program, the deep reinforcement learning based expressway merging area cooperative control method according to any one of claims 1-5 is implemented, and deep reinforcement learning based expressway merging area cooperative control is realized. 8.A computer readable storage medium, having a computer program stored thereon, when the computer program is executed by a processor, the deep reinforcement learning based expressway merging area cooperative control method according to any one of claims 1-5 is implemented, and deep reinforcement learning based expressway merging area cooperative control is realized.

Images