A method for ecological guidance and collaborative control of connected intersections based on deep Q-networks

Abstract

This invention discloses a method for ecological guidance and collaborative control of connected intersections based on deep Q-networks, applicable to the fields of traffic information engineering and control. This invention addresses the limitations of existing technologies in achieving optimal traffic efficiency and right-of-way allocation within intersections. It proposes a method for proactively coordinating connected vehicles with intelligent traffic signal controllers, predicting the behavior of upstream vehicles, and responding to traffic demands. The traffic signal controller generates a signal control strategy based on phase sequence and phase coupling, considering the incorporation of future time-period signal states into current-stage control optimization to avoid short-sightedness, and uses Deep Q-Network optimization to solve the problem. Control is optimized with minimum energy consumption as the objective, employing distributed control to improve control effectiveness and prediction accuracy, and constructing an energy consumption model to evaluate and optimize vehicle trajectory control. Compared to traditional methods, this invention improves traffic efficiency at connected intersections, ensures fairness in right-of-way allocation across traffic flows, and reduces vehicle travel energy consumption. This method can be applied to the control of connected intersections.

Description

Technical Field

[0001] This invention belongs to the field of traffic information engineering and control, specifically relating to a networked intersection ecological guidance and collaborative control method based on deep Q-networks. Background Technology

[0002] Intersections, as key nodes in urban transportation networks, have long faced problems such as low traffic efficiency and frequent congestion. With the development of intelligent connected vehicle technology, V2I communication is used to install centralized autonomous intersection management controllers at intersections to coordinate the movement of CAVs, ensuring efficient and conflict-free passage of CAVs and optimizing ride comfort (Zhao, Rui; Li, Yun; Wang, Kui; Fan, Yuze; Gao, Fei; Gao, Zhenhai. Centralized Cooperation for Connected Autonomous Vehicles at Intersections by Safe Deep Reinforcement Learning[J].IEEE Transactions on Mobile Computing, 2024, Vol.23(12):1-18). In an ideal fully connected autonomous driving scenario, vehicles can autonomously coordinate their passage order through information interaction, theoretically without relying on traditional signal control. However, in practical applications, signal control is still needed to ensure the fair allocation of right-of-way for various traffic participants such as motor vehicles, non-motor vehicles, and pedestrians. Therefore, designing a collaborative guidance strategy between connected vehicles and signal systems has important practical significance in improving the operational efficiency of intersections.

[0003] CAV trajectory optimization has received widespread attention in recent years. To improve the overall performance of the system, various trajectory optimization methods, including distributed control models (DM), have been proposed. In DM, each vehicle is treated as an autonomous agent, determining its own control strategy based on information perceived or received from other vehicles and roadside units to maximize its performance. The literature (Rios-torres, J., Malikopoulos, AA, 2017. A Survey on the Coordination of Connected and Automated Vehicles at Intersections and Merging at Highway On-Ramps. IEEE Trans. Intell. Transp. Syst. 18, 1066–1077.) shows that sharing intersection information with drivers in advance can reduce vehicle energy consumption by up to 20%. Therefore, if vehicles can perform planned speed management at intersections in advance, energy consumption can be reduced while ensuring driving safety.

[0004] In summary, although existing methods have achieved certain results, they still cannot achieve optimal traffic efficiency and right-of-way allocation within intersections. Therefore, proposing a new method is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the problem that existing technologies cannot achieve optimal traffic efficiency and right-of-way allocation at intersections, and to propose a networked intersection ecological guidance and collaborative control method based on deep Q-networks.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a method for ecological guidance and collaborative control of connected cross-sections based on deep Q-networks, the method specifically including the following steps:

[0007] Step 1: Deploy intelligent signal control equipment, roadside units, and vehicle-mounted units. The roadside units will send the collected information to the intelligent signal control equipment.

[0008] And establish a communication link between the roadside unit and the vehicle-mounted unit;

[0009] Step 2: For each approach lane in the current intersection, the intelligent signal control equipment predicts the arrival time of the convoy at the stop line based on the received information;

[0010] Step 3: Based on the predicted arrival time of the convoy at the stop line, dynamically divide the vehicles at each entrance lane of the current intersection into two parts: the release convoy and the stuck convoy.

[0011] Step 4: Based on the number of vehicles in the release queue of each direction, perform phase aggregation processing on the vehicles in each direction. Based on the aggregation processing, construct the state of the deep reinforcement learning model, and output the actual phase release sequence and the green light start time of each phase through the deep reinforcement learning model.

[0012] Step 5: Based on the prediction results in Step 2 and the output of the deep reinforcement learning model, determine whether to maintain the current phase release scheme of the deep reinforcement learning model.

[0013] If the current phase release scheme output by the deep reinforcement learning model is maintained, then the vehicle motion planning for each phase is performed according to the green light start time of each phase output by the deep reinforcement learning model, and the first phase in the phase release scheme is released. Then, the predicted time of the convoy to the stop line in each phase is obtained according to the vehicle motion planning result, and then the prediction result is used to return to execute step three.

[0014] If the current phase release scheme output by the deep reinforcement learning model is not maintained, vehicle motion planning for each phase is performed based on the green light start time of each phase output by the deep reinforcement learning model. Then, the predicted time of the convoy arriving at the stop line in each phase is obtained based on the vehicle motion planning results. Finally, the prediction results are used to return to step three.

[0015] Furthermore, the roadside unit is equipped with multi-source sensing devices, which are used to collect the instantaneous speed of each connected vehicle, the distance to the stop line of the approach lane, and the running time in real time; and send the real-time collected information and real-time traffic flow data to the intelligent signal control equipment.

[0016] Furthermore, the specific process of step two is as follows:

[0017] For any flow of vehicles within the current intersection, taking the exit of the previous intersection's merging zone as the starting point of the control zone, and allowing vehicles to enter the control zone and travel at maximum acceleration to maximum speed, the predicted value for the time when the nth vehicle reaches the stop line is:

[0018]

[0019] in, This represents the predicted time when the nth vehicle will arrive at the stop line;

[0020] This indicates the start time of control for the nth vehicle, i.e., the moment the nth vehicle enters the control zone;

[0021] Indicates that the nth car is in The speed of time;

[0022] This represents the maximum permissible acceleration of the nth vehicle;

[0023] This represents the predicted time when the (n-1)th vehicle will arrive at the stop line;

[0024] Indicates that the nth car is in The speed of time;

[0025] Indicates that the nth car is in The speed of time;

[0026] τ n Let n be the headway between the nth vehicle and the vehicle in front.

[0027] h n Let n be the distance between the nth vehicle and the vehicle in front of it.

[0028] Let be the length of the car ahead of the nth car;

[0029] D represents the distance of the nth car relative to the stop line at the current moment;

[0030] When n = 0, let When the nth car is traveling at maximum acceleration and accelerates to v before the stop line n,max When the predicted time of arrival of the nth vehicle at the stop line is reached, it can be rewritten as:

[0031]

[0032] in, express The distance between the nth car and the stop line at time n;

[0033] This represents the speed of the nth vehicle when it enters the control zone.

[0034] Furthermore, in step three, the specific process of dynamically dividing vehicles into tiers is as follows:

[0035] (1) The spatiotemporal trajectory of the released convoy vehicles satisfies:

[0036] h n ′<d c And g min ≤t duration ≤g max

[0037] Among them, h n ′ represents the distance between the front ends of the nth car and the (n-1)th car;

[0038] d c Indicates the critical spacing;

[0039] gmin Indicates the shortest green light time for a phase;

[0040] g max Indicates the maximum green light time for a phase;

[0041] t duration Let Δt represent the sum of the travel time of the nth vehicle within the intersection, and let Δt represent the difference between the time the nth vehicle leaves the stop line and the time the first vehicle leaves the stop line.

[0042] (2) The set of vehicles stuck in the queue is N' = MN, where N represents the set of vehicles in the release queue and M represents the set of all vehicles in the current flow.

[0043] Furthermore, the constraint satisfied when calculating the predicted value of the nth vehicle's arrival time at the parking line is:

[0044] ① Speed ​​constraint:

[0045]

[0046] in, This represents the speed of the nth vehicle at time t;

[0047] ②Acceleration constraints:

[0048]

[0049] in, Indicates the maximum acceleration that the vehicle is allowed to adopt;

[0050] Indicates the maximum deceleration that the vehicle is allowed to adopt;

[0051] Let represent the acceleration of the nth vehicle at time t;

[0052] ③ Phase constraint:

[0053]

[0054] Among them, g e =t e , t e Indicates the end time of the current vehicle flow phase;

[0055] make This indicates the moment when the nth car leaves the stop line.

[0056] Furthermore, the end time t of the current flow vehicle phase e The calculation method is as follows:

[0057] The time it takes for vehicles currently flowing through the intersection to pass through is taken as the green light duration G, and the green light duration G is:

[0058] G = t d +t l

[0059] Among them, t d The dissipation time of vehicles currently flowing in the direction of travel;

[0060] t l The travel time of the last vehicle in the current flow phase within the intersection;

[0061] Calculate the end time t of the vehicle phase in the current flow direction. e for:

[0062]

[0063] Furthermore, the specific process of step four is as follows:

[0064] Step 4.1. Construct a phase information matrix Ψ based on the prediction results. Let the i-th row of the phase information matrix Ψ be...

[0065] Where: n i This represents the number of vehicles in the i-th flow release echelon;

[0066] This indicates the time when the first vehicle in the i-th direction arrives at the stop line;

[0067] This indicates the time when the first vehicle in the i-th direction leaves the stop line.

[0068] g i,e This indicates the end time of the vehicle phase in the i-th direction;

[0069] G i This represents the duration of the green light for the i-th flow direction;

[0070] Step 42: Perform phase aggregation processing on vehicles flowing in each direction. The specific process is as follows:

[0071] Step 421: Establish a merging decision matrix C of dimension I×I, where I represents the total number of vehicle flow directions at the current intersection, and the element C in the i-th row and j-th column of matrix C is... i,j ∈{0,1}, the phase of the i-th flow direction and the phase of the j-th flow direction satisfy the merging condition if and only if conditions (1) to (3) are satisfied, i.e., C i,j =1;

[0072] Condition (1) There is no spatial conflict between the vehicle trajectory of the i-th direction and the vehicle trajectory of the j-th direction;

[0073] Condition (2), Q i -Q j | / max(Q i Q j Q ≤ δ, where δ is the threshold, Q i Q represents the phase equivalent flow rate in the i-th flow direction. j This represents the phase equivalent flow rate in the j-th flow direction;

[0074] Condition (3), Δt i,j =|T i -T j |≤ε, where ε is the threshold, T i T represents the phase requirement duration for the i-th flow direction. j This represents the phase requirement duration for the j-th flow direction;

[0075] Step 422: Perform matrix fusion.

[0076]

[0077] in, It represents the Hadamardi (or Hadama) stack;

[0078] Ψ represents Ψ T Transpose of;

[0079] Ψ' represents the merged phase set, i.e., the result of the compaction process;

[0080] Step 43: Use the state space s = {o, Ψ′} as the input to the deep reinforcement learning model, where, I0 represents the total number of phases after merging. This represents a phase release order, and the adjusted phase release order and the start time of the green light for each phase are output through a deep reinforcement learning model.

[0081] Furthermore, in the merged phase set Ψ', for the flow direction that satisfies the phase merging condition, the green light start time g of the merged phase is... b ′=min{g i,b}, i∈I′, the green light end time g of the merged phase e ′=max{g i,e}, i∈I′、 The green light duration of the merged phase is G′=g b ′-g e ′;

[0082] I′ represents the set of phases that are allowed to merge with the i-th flow phase and the set of phases formed by the i-th flow phase;

[0083] n i ′ represents the total number of vehicles in the release echelon of the merged phases after merging the phases within set I′.

[0084] Furthermore, the reward function used during the training of the deep reinforcement learning model is:

[0085]

[0086] in, This indicates the predicted time when the first vehicle in the i-th direction arrives at the stop line;

[0087] g i ′ ,b This indicates the start time of the green light for the i-th direction of traffic, as output by the deep reinforcement learning model.

[0088] α1 and α2 represent the equilibrium constants;

[0089] R represents the reward function.

[0090] Furthermore, the specific process of step five is as follows:

[0091] Define the phase adjustment error function E:

[0092] E = ψ p -ψ r

[0093] Where, ψ p A vector representing the time when the first vehicle arrives at the stop line in each predicted release phase;

[0094] ψ r This represents a vector composed of the actual green light start times within each of the current release phases of the deep reinforcement learning model's output;

[0095] When E=0, the current phase release scheme output by the deep reinforcement learning model is maintained. Based on the actual green light opening time in each release phase output by the deep reinforcement learning model, motion planning is performed on the vehicles in each phase to obtain the vehicle motion planning results in each phase.

[0096] The intelligent signal control equipment sends the motion planning results to the vehicle unit through the roadside unit. Vehicles in the first release phase are released according to the motion planning results. Based on the motion planning results of vehicles in each phase, the predicted time of arrival of the platoon at the stop line in each phase is obtained. The predicted results are then used to return to step three.

[0097] The specific process of motion planning is as follows:

[0098]

[0099] Where K is a constant;

[0100] Let represent the acceleration of the i′-th vehicle;

[0101] This indicates the time when the i′-th vehicle enters the control zone;

[0102] This indicates the time when the i′-th vehicle leaves the controlled area;

[0103]

[0104] Where f represents the second-order linear dynamic equation;

[0105] d i′ (t) represents the position of the i′th vehicle at time t;

[0106] This represents the speed of the i′-th vehicle at time t;

[0107] Indicates the start time of control for the i′th vehicle;

[0108] Indicates the distance from the start of control to the stop line;

[0109] When E≠0, based on the actual green light start time in each release phase output by the deep reinforcement learning model, motion planning is performed on the vehicles in each phase to obtain the vehicle motion planning results in each phase.

[0110] Based on the vehicle motion planning results in each phase, the predicted arrival time of the convoy at the stop line in each phase is obtained, and the predicted results are used to return to step three.

[0111] The beneficial effects of this invention are:

[0112] This invention addresses the control problem at connected intersections by proposing a method for active coordination between connected vehicles and intelligent traffic signal controllers, prediction of upstream vehicle behavior, and response to traffic demand. The traffic signal controller generates a signal control strategy based on phase sequence and phase coupling, considering the incorporation of future time-period signal states into current-stage control optimization to avoid short-sightedness. For the complex and high-dimensional problems encountered in traditional control solutions, DQN optimization is employed. Connected vehicles optimize control under constraints with the goal of minimizing energy consumption, employing distributed control to improve control effectiveness and prediction accuracy. An energy consumption model is constructed to evaluate and optimize vehicle trajectory control performance. Results show that, compared to traditional control methods, this invention improves traffic efficiency at connected intersections, ensures fairness in right-of-way across traffic flows, and reduces vehicle travel energy consumption.

[0113] By designing a DQN network model, the state input and reward function are accurately calibrated, and the signal phase adjustment strategy is dynamically optimized, thereby achieving multi-objective collaborative optimization while reducing computational load. Attached Figure Description

[0114] Figure 1 This is a flowchart of an ecological guidance and collaborative control method for connected intersections based on deep Q-networks, according to the present invention.

[0115] Figure 2 This is a diagram illustrating the echelon division;

[0116] Figure 3 This is a diagram illustrating intersection conflicts;

[0117] Figure 4 This is a schematic diagram of a network-connected intersection;

[0118] Figure 5 This is a diagram illustrating the early arrival of traffic due to signal phase.

[0119] Figure 6 This is a signal phase delay indicating the arrival of traffic. Figure 1 ;

[0120] Figure 7 This is a signal phase delay indicating the arrival of traffic. Figure 2 ;

[0121] Figure 8 It is a traffic simulation environment at urban intersections. Detailed Implementation

[0122] Specific implementation method one: Combining Figure 1 This embodiment describes an ecological guidance and collaborative control method for connected cross-sections based on deep Q-networks. The method specifically includes the following steps:

[0123] Step 1: Deploy intelligent signal control equipment, roadside units (RSUs), and on-board units (OBUs) with V2X communication capabilities. The roadside units will send the collected information to the intelligent signal control equipment.

[0124] And establish a communication link between the roadside unit and the vehicle-mounted unit;

[0125] Step 2: For each approach lane in the current intersection, the intelligent signal control equipment predicts the arrival time of the convoy at the stop line based on the received information;

[0126] Step 3: Based on the predicted arrival time of the convoy at the stop line, dynamically divide the vehicles at each entrance lane of the current intersection into two parts: the release convoy and the stuck convoy.

[0127] Step 4: Based on the number of vehicles in the release queue of each direction, perform intensive processing on the phases of vehicles in each direction. Based on the intensive processing, construct the state of the deep reinforcement learning model. Output the actual phase release sequence and the green light start time of each phase through the deep reinforcement learning model (it should be noted that for the first release phase in the actual phase release sequence, the actual green light start time of the first phase is equal to the predicted time when the first vehicle in the first phase arrives at the stop line, the actual green light start time of the second phase is the time when the last vehicle in the release gradient of the first phase leaves the intersection, and so on, to obtain the actual green light start time of each phase).

[0128] Step 5: Based on the prediction results in Step 2 and the output of the deep reinforcement learning model, determine whether to maintain the current phase release scheme of the deep reinforcement learning model.

[0129] If the current phase release scheme output by the deep reinforcement learning model is maintained, then the vehicle motion planning for each phase is performed according to the green light start time of each phase output by the deep reinforcement learning model, and the first phase in the phase release scheme is released. Then, the predicted time of the convoy to the stop line in each phase is obtained according to the vehicle motion planning result, and then the prediction result is used to return to execute step three.

[0130] It should be noted that each time decision E=0 is met, only vehicles in the first phase are allowed to pass, and then the remaining vehicles in the first phase re-participate in the phase sorting based on the predicted arrival time.

[0131] If the current phase release scheme output by the deep reinforcement learning model is not maintained, vehicle motion planning for each phase is performed based on the green light start time of each phase output by the deep reinforcement learning model. Then, the predicted time of the convoy arriving at the stop line in each phase is obtained based on the vehicle motion planning results. Finally, the prediction results are used to return to step three.

[0132] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the roadside unit is equipped with multi-source sensing devices. These devices are used to collect the instantaneous speed of each connected vehicle, its distance from the stop line of the approach lane, and its running time in real time; and to send the real-time collected information and real-time traffic flow data to the intelligent signal control equipment.

[0133] The other steps and parameters are the same as in Specific Implementation Method 1.

[0134] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the specific process of step two is as follows:

[0135] For any flow of vehicles within the current intersection, taking the exit of the previous intersection's merging zone as the starting point of the control zone, and allowing vehicles to enter the control zone and travel at maximum acceleration to maximum speed, the predicted value for the time when the nth vehicle reaches the stop line is:

[0136]

[0137] in, This represents the predicted time when the nth vehicle will arrive at the stop line;

[0138] This indicates the start time of control for the nth vehicle, i.e., the moment the nth vehicle enters the control zone;

[0139] Indicates that the nth car is in The speed of time;

[0140] This represents the maximum permissible acceleration of the nth vehicle;

[0141] This represents the predicted time when the (n-1)th vehicle will arrive at the stop line;

[0142] Indicates that the nth car is in The speed of time;

[0143] Indicates that the nth car is in The speed of time;

[0144] τ n Let n be the headway between the nth vehicle and the vehicle in front.

[0145] h n Let n be the distance between the nth vehicle and the vehicle in front of it.

[0146] Let be the length of the car ahead of the nth car;

[0147] D represents the distance of the nth car relative to the stop line at the current moment;

[0148] When n = 0, let When the nth car is traveling at maximum acceleration and accelerates to v before the stop line n,max When the predicted time of arrival of the nth vehicle at the stop line is reached, it can be rewritten as:

[0149]

[0150] in, express The distance between the nth car and the stop line at time n;

[0151] This represents the speed of the nth vehicle when it enters the control zone.

[0152] Other steps and parameters are the same as in specific implementation method one or two.

[0153] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, in step three, as follows... Figure 2 As shown, the specific process of dynamic tiering of vehicles is as follows:

[0154] (1) The spatiotemporal trajectory of the released convoy vehicles satisfies:

[0155] h n ′<d c And g min ≤t duration ≤g max

[0156] Among them, h n ′ represents the distance between the front ends of the nth car and the (n-1)th car;

[0157] d c This represents the critical distance, taken as 15 m / s, used to divide intermittent traffic flow;

[0158] g min Indicates the shortest green light time for a phase;

[0159] g max Indicates the maximum green light time for a phase;

[0160] t duration It represents the sum of Δt and the time the nth vehicle travels within the intersection (usually taken as a fixed value), where Δt represents the difference between the time the nth vehicle leaves the stop line and the time the first vehicle leaves the stop line;

[0161] like Figure 3 The diagram shows an intersection with 8 phases. For each phase, the judgment method of condition (1) is used. For any phase, all vehicles in that phase that meet condition (1) are assigned to the release queue.

[0162] (2) The set of vehicles stuck in the queue is N' = MN, where N represents the set of vehicles in the release queue and M represents the set of all vehicles in the current flow.

[0163] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0164] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the constraint satisfied when calculating the predicted value of the nth vehicle's arrival time at the parking line is as follows:

[0165] ① Speed ​​constraint:

[0166]

[0167] in, This represents the speed of the nth vehicle at time t;

[0168] ②Acceleration constraints:

[0169]

[0170] in, Indicates the maximum acceleration that the vehicle is allowed to adopt;

[0171] Indicates the maximum deceleration that the vehicle is allowed to adopt;

[0172] Let represent the acceleration of the nth vehicle at time t;

[0173] ③ Phase constraints: Since the start and end times of the optimized phases change in the control time domain, phase constraints are applied to the vehicle fleets corresponding to each phase to ensure safe passage within the intersection merging area.

[0174]

[0175] Among them, g e =t e , t e Indicates the end time of the current vehicle flow phase;

[0176] make This indicates the moment when the nth car leaves the stop line.

[0177] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0178] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One to Five in that the end time t of the current flow vehicle phase... e The calculation method is as follows:

[0179] The time it takes for vehicles currently flowing through the intersection to pass through is taken as the green light duration G, and the green light duration G is:

[0180] G = t d +t l

[0181] Among them, t d The dissipation time of vehicles currently in the flow;

[0182] t lThe travel time of the last vehicle in the current flow phase within the intersection;

[0183] Calculate the end time t of the vehicle phase in the current flow direction. e for:

[0184]

[0185] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0186] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One to Six in that the specific process of step four is as follows:

[0187] Step 41, as follows Figure 4 The diagram shows the arrival patterns of all vehicles within the control domain. A phase information matrix Ψ is constructed based on the prediction results. Let the i-th row of the phase information matrix Ψ...

[0188] Where: n i This represents the number of vehicles in the i-th flow release echelon;

[0189] This indicates the time when the first vehicle in the i-th direction arrives at the stop line;

[0190] This indicates the time when the first vehicle in the i-th direction leaves the stop line.

[0191] g i,e This indicates the end time of the vehicle phase in the i-th direction;

[0192] G i This represents the duration of the green light for the i-th flow direction;

[0193] Since the basic phase sequence satisfies the traffic requirements of each direction, it ensures that vehicles in each direction are in a protected phase to avoid conflicts. To reduce phase switching, phases are merged through intensive processing. During phase merging, the following must be considered: 1. Maintaining a non-conflicting relationship between vehicles, such as... Figure 3 As shown, vehicle trajectories should not intersect at the same time; 2. The traffic flow of the phases used for merging should not differ too much, so as to reduce the green light being wasted in the direction with less traffic flow in the merged phase.

[0194] Step 42: Perform phase aggregation processing on vehicles flowing in each direction. The specific process is as follows:

[0195] Step 421: Establish a merging decision matrix C of dimension I×I, where I represents the total number of vehicle flow directions at the current intersection, and the element C in the i-th row and j-th column of matrix C is... i,j∈{0,1}, the phase of the i-th flow direction and the phase of the j-th flow direction satisfy the merging condition if and only if conditions (1) to (3) are satisfied, i.e., C i,j =1;

[0196] Condition (1) There is no spatial conflict between the vehicle trajectory of the i-th direction and the vehicle trajectory of the j-th direction;

[0197] Condition (2), Q i -Q j | / max(Q i Q j )≤δ, where δ is 0.25, Q i Q represents the phase equivalent flow rate in the i-th flow direction. j This represents the phase equivalent flow rate of the j-th flow direction. The phase equivalent flow rate of each flow direction is calculated based on the number of vehicles in the release queue of each flow direction.

[0198] Condition (3), Δt i,j =|T i -T j |≤ε (ε is taken as 3s), T i T represents the phase requirement duration for the i-th flow direction. j This represents the phase requirement duration for the j-th flow direction;

[0199] Step 422: Perform matrix fusion.

[0200]

[0201] in, It represents the Hadamardi (or Hadama) stack;

[0202] Ψ represents Ψ T Transpose of;

[0203] Ψ' represents the merged phase set, i.e., the result of the compaction process;

[0204] Step 43: Use the state space s = {o, Ψ′} as the input to the deep reinforcement learning model, where, I0 represents the total number of phases after merging. This represents a phase release order (obtained randomly), and the adjusted phase release order and the start time of the green light for each phase are output by a deep reinforcement learning model.

[0205] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0206] This invention utilizes a trained DQN network model to solve the problem. During the training process of the DQN network model, the Q-value network is used to calculate the Q-value of the current control strategy selection and the Q-value iterative update, the Target network is used to calculate the Q-value of the next state in the time-difference algorithm, and the network parameter update comes from the Q-value network parameter copy.

[0207]

[0208] The network parameters are updated using stochastic gradient descent, and the weights θ and θ' of the main network and the target network are also updated. - .

[0209]

[0210] θ - ←τθ+(1-τ)θ -

[0211] In the formula, α is the learning rate. It is the loss function, which is usually the mean squared error (MSE) between the predicted Q value and the target Q value. τ is the soft update parameter.

[0212] Experience playback is used to store historical intersection spatiotemporal states, signal decisions, and benefit values. Control strategies are generated through global information learning to adapt to traffic conditions.

[0213] T k =(s k ,a k ,R k ,s k+1 )

[0214] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One through Seven in that, in the merged phase set Ψ', for the flow direction that meets the phase merging condition, the green light start time g of the merged phase... b ′=min{g i,b}, i∈I′, the green light end time g of the merged phase e ′=max{g i,e}, i∈I′、 I′ represents the set of phases that are allowed to merge with the i-th flow phase and the set consisting of the i-th flow phase, n i G′ represents the total number of vehicles in the release queue of the merged phase after merging phases within set I′, and the green light duration of the merged phase is G′=g. b ′-g e ′.

[0215] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0216] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that the reward function used during the training of the deep reinforcement learning model is:

[0217]

[0218] in, This indicates the predicted time when the first vehicle in the i-th direction arrives at the stop line;

[0219] g i ′ ,b This indicates the start time of the green light for the i-th direction of traffic, as output by the deep reinforcement learning model.

[0220] α1 and α2 represent the equilibrium constants;

[0221] R represents the reward function.

[0222] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0223] In this implementation, the reward function is designed to consider both intersection traffic efficiency and right-of-way fairness. The goal is to maximize the number of vehicles passing through each phase, and further balance right-of-way fairness across all directions to minimize headway delays in each direction.

[0224] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One to Nine in that the specific process of step five is as follows:

[0225] Define the phase adjustment error function E:

[0226] E = ψ p -ψ r

[0227] Where, ψ p A vector representing the time when the first vehicle arrives at the stop line in each predicted release phase;

[0228] ψ r This represents a vector composed of the actual green light start times within each of the current release phases of the deep reinforcement learning model's output;

[0229] When E=0, the final time is equal to the predicted time, and the final state is equal to the predicted state. The phase release scheme currently output by the deep reinforcement learning model is maintained. Based on the actual green light opening time in each release phase currently output by the deep reinforcement learning model, motion planning is performed on the vehicles in each phase to obtain the vehicle motion planning results in each phase.

[0230] The intelligent signal control equipment sends the motion planning results to the vehicle unit through the roadside unit. Vehicles in the first release phase are released according to the motion planning results. Based on the motion planning results of vehicles in each phase, the predicted time of arrival of the platoon at the stop line in each phase is obtained. The predicted results are then used to return to step three.

[0231] The specific process of motion planning is as follows:

[0232] Fuel consumption of each vehicle and its control input a i There is a monotonic relationship between (t). Therefore, the problem of fuel consumption optimization can be transformed into optimizing vehicle acceleration, and can be further expressed as minimizing the L2 norm of acceleration.

[0233]

[0234] Where K is a constant;

[0235] Let represent the acceleration of the i′-th vehicle;

[0236] This indicates the time when the i′-th vehicle enters the control zone;

[0237] This indicates the time when the i′-th vehicle leaves the controlled area;

[0238]

[0239] The solution to the vehicle's velocity trajectory within the controlled area is defined as the optimal control problem. The state equation of the controlled system is established, and it is assumed that f satisfies the existence of a locally convergent solution within the constraints of the equation:

[0240]

[0241] Where f represents the second-order linear dynamic equation;

[0242] d i′ (t) represents the position of the i′th vehicle at time t;

[0243] This represents the speed of the i′-th vehicle at time t;

[0244] Indicates the start time of control for the i′th vehicle;

[0245] Indicates the distance from the start of control to the stop line;

[0246] When E < 0, the green light duration is determined by the shortest time required to ensure the convoys can completely pass through the intersection and to avoid the green light being wasted. Due to limitations in vehicle and road safety, guidance strategies may have non-optimal solutions. Phase parameters need to be dynamically adjusted based on real-time fleet status, such as Figure 5 As shown;

[0247] When E > 0, at this time Due to the delayed phase activation, vehicles arriving early may queue at the stop line. Meanwhile, as subsequent vehicles arrive and the release queue expands, the mismatch between the position of the last vehicle and the passage time window will lead to an increase in the demand for green light duration. Therefore, phase parameters need to be dynamically adjusted based on the real-time queue status. Figure 6 and Figure 7 As shown;

[0248] When E≠0, based on the actual green light start time in each release phase output by the deep reinforcement learning model, motion planning is performed on the vehicles in each phase to obtain the vehicle motion planning results in each phase.

[0249] Based on the vehicle motion planning results in each phase, the predicted arrival time of the convoy at the stop line in each phase is obtained, and the predicted results are used to return to step three.

[0250] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0251] Without vehicle control, the vehicle will stop and wait at the stop line due to the loss of right-of-way. In a fleet control system, such as... Figure 4 As shown, vehicles can communicate with their surroundings (including traffic lights and other CAVs) via vehicle-to-vehicle (V2V) and internet-to-vehicle (I2V) technologies. All CAVs are equipped with onboard controllers that can control them at every time and location based on signal policies and vehicle information. To smooth vehicle trajectories for energy conservation and driving safety, CAVs are controlled to maintain a smooth trajectory before approaching signalized intersections without sudden accelerations and decelerations or excessive starts and stops within the road segment.

[0252] Experimental Section

[0253] In response to the complex and diverse urban road operation scenarios, and the dynamic and ever-changing road conditions and traffic states, fixed-time signal systems are ill-suited to adapt to real-time traffic flow changes and frequent vehicle starts and stops. This invention proposes a networked intersection ecological guidance and collaborative control method to achieve adaptive control of urban intersections and ecological guidance for vehicles. A simulated urban intersection operation scenario is built using VISSIM simulation software, and the method of this invention is applied to traffic flow states with different flow rates and arrival rates. The rationale for the method of this invention is further illustrated by the simulation scenario establishment process, the specific implementation process of which is as follows:

[0254] Step 1: First, based on the analysis of urban intersections, design a comparative experiment between the method of this invention and conventional intersections under traditional fixed timing conditions;

[0255] Step 2: Build an urban intersection traffic environment on the VISSIM platform, and set relevant traffic flow parameters, traffic light arrangements, and detector settings. While making the simulation environment as close to the real environment as possible, acquire risk perception data, such as... Figure 8 As shown.

[0256] Step 3: Using the traffic flow parameters of urban road sections as input, the simulation environment selects one-way traffic flow of 800 pcu / h and 1200 pcu / h respectively to design experiments. By connecting to the COM port through Python, the traffic flow data during the simulation environment can be detected and processed to carry out subsequent model building.

[0257] Step 4: Use the processed data to predict vehicle arrival behavior and make preliminary arrangements for their required passage phases. Optimize the best release sequence of the required passage phases for vehicles using a DQN network, and implement control by solving the optimal control problem online using an onboard controller, outputting a signal control scheme and a vehicle guidance scheme.

[0258] Step 5: Evaluate the operational status of urban intersections under the method of this invention. Average vehicle delay, travel time, and vehicle output power are selected as evaluation indicators for the operational effectiveness of this method. The resulting traffic evaluation indicators are shown in Table 1. Table 1 shows that the method of this invention improves the traffic efficiency and average vehicle delay of intersections, and reduces vehicle energy consumption (energy consumption calculation formula is:

[0259] Table 1

[0260]

[0261]

[0262] Finally, through simulation, it can be basically determined that the simulation results are consistent with the actual situation and have certain reference value and practical significance, thus verifying the effectiveness of the connected intersection ecological guidance and collaborative control method proposed in this invention.

[0263] This invention extracts real-time road information from connected vehicles. To respond to their traffic demands, it first pre-arranges the required traffic phases. Then, when performing real-time control at different urban intersections, it can adjust the allocation of right-of-way for each flow direction by releasing different vehicle platoon divisions. Secondly, this invention first uses a DQN network model to optimize the sequence of the pre-arranged phases. Then, the onboard controller guides the vehicles by combining signal strategies and vehicle conditions. Finally, iterative optimization of non-optimal control behaviors determines the cooperative control method. Since there are various neural network models, different network models can be selected for replacement in actual operation to complete the phase sequence optimization task. The onboard controller can also select different optimal control objectives to meet different operational requirements.

[0264] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A deep Q network based ecological guidance cooperative control method for a connected intersection, characterized in that, The method specifically comprises the following steps: Step one, deploying intelligent signal control equipment, road side units and vehicle-mounted units, the road side units send the collected information to the intelligent signal control equipment; And establish a communication link between the road side units and the vehicle-mounted units; Step two, for each approach lane in the current intersection, the intelligent signal control equipment predicts the vehicle arrival time at the stop line according to the received information; Step three, according to the prediction results of the vehicle arrival time at the stop line, the vehicles in the current intersection are dynamically divided into release queues and retention queues; Step four, according to the number of vehicles in the release queue of each flow direction, the phases of the vehicles in each flow direction are intensively processed, the state of the deep reinforcement learning model is constructed according to the intensive processing, and the actual phase release sequence and the green start time of each phase are output through the deep reinforcement learning model; The specific process of step four is: Step four i. Constructing a phase information matrix from the prediction results , let the phase information matrix be the first row of the matrix wherein: represents the number of vehicles in the i-th flow direction release tier; represents the number of vehicles in the i-th flow direction release tier; represents the time at which the first vehicle of the first stream reaches the stop line; represents the time when the first vehicle of the first stream leaves the stop line, = ;​ representing the end time of the vehicle phase of the flow direction; represents the green light duration for the th flow direction; Step four two, the phases of the vehicles in each flow direction are intensively processed, and the specific process is: Step 421: Establish dimensions as Merging decision matrix ,in, This represents the total number of vehicle flow directions at the current intersection, and the matrix... The Middle Line 1 Column elements The first condition is met if and only if conditions (1) to (3) are satisfied. The phase of the flow direction and the first The phases of each flow direction satisfy the merging condition, that is ; condition (1), the first vehicle trajectory of the first flow direction has no spatial conflict with the second vehicle trajectory of the second flow direction; condition (2), , is a threshold value, denotes the phase equivalent flow of the flow direction, denotes the phase equivalent flow of the flow direction; condition (3), , is a threshold value, denotes the phase requirement duration of the flow direction, denotes the phase requirement duration of the flow direction; Step four two two, matrix fusion is performed: wherein, denotes a Hadamard product; denotes the transpose of denotes the merged phase set, i.e. the result of the set intensive processing; the merged phase set for a flow direction satisfying a phase merging condition , a green light end time of the merged phase , , a green light duration of the merged phase ; represents a set of phases that are allowed to merge with the first flow direction phase and the second flow direction phase; representing a set of vehicles in the release tier of the merged phase after the merging of the phases Step four three, state space as the input of the deep reinforcement learning model, , representing the total number of combined phases, representing a phase release sequence, outputting an adjusted phase release sequence and the green light start time of each phase through a deep reinforcement learning model; Step five, according to the prediction results in step two and the output of the deep reinforcement learning model, it is judged whether to maintain the phase release scheme output by the deep reinforcement learning model currently; If the phase release scheme output by the deep reinforcement learning model currently is maintained, the vehicle motion planning of each phase is carried out according to the green start time of each phase output by the deep reinforcement learning model, the first phase in the phase release scheme is released, the vehicle arrival time at the stop line in each phase is obtained according to the vehicle motion planning result, and the prediction result is used to return to step three; If the phase release scheme output by the deep reinforcement learning model currently is not maintained, the vehicle motion planning of each phase is carried out according to the green start time of each phase output by the deep reinforcement learning model, the vehicle arrival time at the stop line in each phase is obtained according to the vehicle motion planning result, and the prediction result is used to return to step three.

2. The cooperative control method of the ecological guidance of the connected intersection based on the deep Q network according to claim 1, characterized in that, The multi-source perception device is arranged in the road side unit, and is used for collecting the instantaneous speed, distance from the approach lane stop line and running time of each connected vehicle in real time; and sending the real-time collected information and real-time traffic flow data to the intelligent signal control equipment.

3. The cooperative control method of the ecological guidance of the connected intersection based on the deep Q network according to claim 2, characterized in that, The specific process of step two is: For any one flow direction of vehicles in the current intersection, the vehicle driving out of the last intersection merging area is taken as the starting point of the control area, and the vehicle is allowed to enter the control area to drive at the maximum acceleration to the maximum speed, then the prediction value of the time when the vehicle reaches the stop line is: ​ wherein, represents the predicted value of the moment when the vehicle reaches the stop line; indicates the start time of the vehicle control, i.e. the moment when the vehicle enters the control zone; indicates the start time of the vehicle control, i.e. the moment when the vehicle enters the control zone; indicates the start time of the vehicle control, i representing the speed of the vehicle at the time instant representing the speed of the vehicle at the time instant representing the speed of the vehicle at the time instant Indicates the first The maximum permissible acceleration of a vehicle; representing the predicted value of the instant at which the vehicle reaches the stop line; representing the speed of the vehicle at the time instant representing the speed of the vehicle at the time instant representing the speed of the vehicle at the time instant representing the speed of the vehicle at the time instant representing the speed of the vehicle at the time instant representing the speed of the vehicle at the time instant the first vehicle and the second vehicle; and the time headway between the first vehicle and the second vehicle; For the first Vehicle and the preceding vehicle; the first vehicle is a front vehicle of the second vehicle; and the body length of the front vehicle of the second vehicle; D represents the distance of the vehicle relative to the stop line at the current time point vehicle relative to the stop line at the current time point When the first vehicle is travelling at maximum acceleration and accelerates to the maximum acceleration just before the stop line, the prediction of the time at which the first vehicle will reach the stop line is revised to be: When the first vehicle is travelling at maximum acceleration and accelerates to the maximum acceleration just before the stop line, the prediction of the time at which the first vehicle will reach the stop line is revised to be: in, express Time of the first The distance between the vehicle and the parking line; indicates the speed of the vehicle when entering the control zone. indicates the speed of the vehicle when entering the control zone.

4. The cooperative control method of the ecological guidance of the connected intersection based on the deep Q network according to claim 3, characterized in that, In step three, the specific process of dynamic queue division of vehicles is: (1) The time-space trajectory of the release queue vehicles satisfies: wherein, denotes the headway of the nth vehicle to the (n-1)th vehicle; represents the critical distance; the shortest green time representing the phase; maximum green time representing a phase; express The sum of the travel times of the nth vehicle and the nth vehicle within the intersection. Indicates the first The difference between the time a vehicle leaves the stop line and the time the first vehicle leaves the stop line; (2) a set of vehicles that are stranded in the platoon wherein, denotes a set of vehicles that are released from the platoon, denotes a set of all vehicles of the current flow direction.

5. The cooperative control method of ecological guidance of the connected intersection based on deep Q network according to claim 4, characterized in that, The first The constraints that are satisfied when calculating the predicted value of the instant at which the vehicle reaches the stop line are: ① Speed constraint: wherein, represents the speed of the vehicle at the time instant instant​ ② Acceleration constraint: wherein, represents the maximum acceleration that the vehicle is allowed to take up; represents the maximum deceleration the vehicle is allowed to take up; represents the acceleration of the vehicle at the time instant instant​ ③ Phase constraint: wherein, , denotes the end time of the current vehicle phase in the flow direction. Let , denote the time at which the first vehicle leaves the stop line.

6. The cooperative control method of ecological guidance of a connected intersection based on a deep Q network according to claim 5, characterized in that, the end time of the vehicle phase of the current flow direction The calculation method is: the time for the vehicle currently flowing through the intersection as the green light duration , the green light duration is: wherein, is the dissipation time for the current flow of vehicles; the travel time for the tail vehicle within the phase currently flowing to the vehicle within the intersection; calculating an end time of the current vehicle phase in the current flow direction is: 。 7. The deep Q network-based ecological guidance cooperative control method for a connected intersection according to claim 6, characterized in that, The reward function used in the training of the deep reinforcement learning model is: wherein, , denotes the predicted time of arrival of the head vehicle of the i-th flow at the stop line. represents the i-th stream of the green light start time of the release of the deep reinforcement learning model output; and represents the equilibrium constant; represents a reward function.

8. The cooperative control method of ecological guidance of a connected intersection based on a deep Q network according to claim 7, characterized in that, The specific process of step five is: Defining a phase adjustment error function : wherein, represents a vector of predicted times of arrival of the first vehicle at the stop line for each release phase; a vector representing the actual green light opening time points in each release phase output by the current deep reinforcement learning model; When the phase release scheme of the current output of the deep reinforcement learning model is maintained, the vehicles in each phase are motion planned according to the actual green light opening time in each release phase of the current output of the deep reinforcement learning model, and vehicle motion planning results in each phase are obtained. The intelligent signal control equipment sends the motion planning result to the vehicle-mounted unit through the road side unit, the vehicles in the first release phase are released according to the motion planning result, the prediction result of the vehicle arrival time at the stop line in each phase is obtained according to the vehicle motion planning result in each phase, and the prediction result is used to return to step three; The specific process of the motion planning is: wherein is a constant; representing the acceleration of the vehicle representing the acceleration of the vehicle representing the time at which the vehicle entered the control zone representing the time at which the vehicle entered the control zone representing the time at which the vehicle left the control zone vehicle left the control zone ； wherein represents a second order linear dynamics equation; representing the position of the vehicle at the time instant; representing the position of the vehicle at the time instant; representing the position of the vehicle at the time instant; representing the speed of the vehicle at the time instant instant​ Indicates the first Vehicle control start time; represents the distance from the control start to the stop line; When the actual green light opening time in each release phase output by the current depth reinforcement learning model, motion planning is performed for the vehicles in each phase to obtain vehicle motion planning results in each phase. And the prediction result of the vehicle arrival time at the stop line in each phase is obtained according to the vehicle motion planning result in each phase, and the prediction result is used to return to step three.

Images