A signal timing and networked automatic driving vehicle speed hierarchical cooperative control method based on reinforcement learning

Abstract

The application provides a signal timing and networked automatic driving vehicle speed hierarchical collaborative control method based on reinforcement learning, and relates to the technical field of intelligent traffic control and vehicle-road collaborative control, and comprises the following steps: obtaining the stop-and-drive vehicle state information on the import lane served by each phase at each signal cycle boundary, and generating a reference timing scheme of the next cycle by a cycle timing intelligent agent based on the stop-and-drive vehicle state information; obtaining a refined discharge window; and outputting a continuous longitudinal acceleration control instruction to a target networked automatic driving vehicle by a networked automatic driving vehicle speed intelligent agent according to the refined discharge window, so that the arrival time of the target networked automatic driving vehicle matches the discharge window, and the collaborative control of traffic signal timing and vehicle speed is realized.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent traffic control and vehicle-road cooperative control, and more particularly to a method for hierarchical cooperative control of signal timing and speed of connected autonomous vehicles based on reinforcement learning. Background Technology

[0002] Urban traffic congestion leads to significant increases in economic costs and environmental burden. Signalized intersections, as key bottlenecks in road networks, are particularly vulnerable; inappropriate signal timing at these intersections can easily induce frequent vehicle starts and stops, increasing delays, emissions, and energy consumption. With the development of connected autonomous vehicle technology and vehicle-to-infrastructure (V2I) communication technology, traffic signal control systems are now able to acquire more detailed vehicle operating status information and further optimize signal timing and vehicle trajectories to improve intersection efficiency and the overall quality of traffic system operation.

[0003] Deep reinforcement learning methods have been widely applied to adaptive traffic signal control, and have gradually expanded from single-intersection optimization to multi-intersection coordinated control and periodic timing optimization. Simultaneously, deep reinforcement learning has also been applied to the longitudinal motion control of connected autonomous vehicles to achieve energy-efficient driving, trajectory smoothing, and improved operational efficiency. However, existing single-sided control schemes that only target traffic signals or only target vehicle operation have limitations. The former typically treats the vehicle as a passively controlled object, making it difficult to fully exploit the vehicle's active control potential, while the latter lacks collaborative perception and unified decision-making regarding signal phase changes, resulting in limited vehicle control effectiveness.

[0004] In existing technologies, most traffic control schemes for signalized intersections only control the timing of traffic signals or only control the trajectory of connected autonomous vehicles, making it difficult to achieve unified coordination between the infrastructure and vehicle sides. Especially in mixed traffic scenarios involving both human-driven and connected autonomous vehicles, traffic signal timing is a discrete decision, while vehicle longitudinal control is a continuous decision, and both operate at different time scales. This makes it difficult for existing technologies to simultaneously achieve real-time performance, stability, and control effectiveness. Therefore, it is necessary to propose a method that can achieve coordinated control of signal timing and vehicle speed at different time scales. Summary of the Invention

[0005] To address the technical problem in existing technologies where traffic signal timing and speed control of connected autonomous vehicles are difficult to coordinate at different time scales, this invention provides a hierarchical collaborative control method for traffic signal timing and speed of connected autonomous vehicles based on reinforcement learning. This invention involves three intelligent agents interacting with the environment at multiple time scales. Through hierarchical collaboration of periodic timing control, phase-level online fine-tuning, and vehicle-level speed control, it generates, refines, and tracks intersection emission windows, enabling traffic signal control and vehicle operation control to work in tandem, thereby improving intersection traffic efficiency and reducing vehicle stopping and waiting.

[0006] The technical means employed in this invention are as follows:

[0007] A reinforcement learning-based hierarchical cooperative control method for signal timing and speed of connected autonomous vehicles is applied to a mixed traffic scenario at signalized intersections involving both connected autonomous vehicles and manually driven vehicles. The method includes the following steps: S1. At the boundary of each signal cycle, obtain the status information of stopped vehicles on the entrance lanes served by each phase, and generate the reference timing scheme for the next cycle based on the status information of the stopped vehicles. The reference timing scheme includes: total green light budget and green light allocation for each phase. The reference timing scheme satisfies the minimum green light duration constraint and the clearing time constraint. S2. When the remaining time of the current green light phase reaches the preset trigger threshold, a key vehicle is selected, and the phase fine-tuning agent adjusts the green light termination time of the current active phase in a discrete time manner based on the status of the key vehicle, the traffic pressure of the current green light phase, and the traffic pressure of the next phase. The emission window is refined under the premise of keeping the phase sequence unchanged and the total cycle budget unchanged, and the refined emission window is obtained. S3. Based on the refined emission window, the speed intelligent agent of the connected autonomous vehicle outputs continuous longitudinal acceleration control commands to the target connected autonomous vehicle, so that the arrival time of the target connected autonomous vehicle matches the emission window, thereby realizing the coordinated control of traffic signal timing and vehicle speed.

[0008] Further, S1 includes the following steps: The state vector of a periodically timed agent is constructed using the following formula:

[0009] in, to These represent the total number of vehicles stopped in the corresponding entrance lanes for north-south straight traffic, north-south left turns, east-west straight traffic, and east-west left turns, respectively. The periodically timed agent outputs the action vector for the next period at the boundary of each period, and the action vector is expressed by the following formula:

[0010] in, This indicates the total green light budget parameters. to These represent the green light ratio allocation parameters for each phase; after normalization, these green light ratio allocation parameters are mapped to feasible green light durations under constraints. The periodic reward of a periodically timed agent is calculated using the following formula:

[0011] in, Indicates the actual cycle duration. Represents the normalization constant. This represents the sum of the total number of vehicles momentarily stopped across all phases; to accommodate changes in cycle length, the discount factor is adjusted to... ; The periodic timing agent employs a bidirectional long short-term memory network to process recent data. The embedding vector of a periodic state sequence is obtained by encoding it using the following formula:

[0012] in, This indicates a splicing operation. Represents a fully connected layer; each step in the sequence The internal dynamics of the LSTM cell are determined by the following gating mechanism:

[0013] in, Represents the Sigmoid function; Represents the Hadamard product; These represent the forget gate, input gate, and output gate, respectively. Represents the cell state; the resulting embedding vector As input to the actor network and critic network, it is used to output the baseline timing scheme for the next cycle.

[0014] Further, step S2 includes the following steps: The state vector of the phase-tuning agent is constructed as follows:

[0015] in, These represent the normalized distance, speed, and acceleration of the critical vehicle from the stop line, respectively. Indicates the estimated arrival time of the key vehicle. Indicates the feasibility ratio. Indicates the remaining green light time, parameter pair This indicates the normalized number of stopped vehicles and cumulative delays on the approach lanes served by the current green light phase, while the parameters are related to... This indicates the normalized number of stopped vehicles and the cumulative delay in the next phase group; The actions of the phase-tuning agent are represented by the following formula:

[0016] The This represents the discrete-time adjustment amount applied to the green light duration of the currently active phase, and is based on... Update the green light duration for the currently active phase; The reward for the phase-fine-tuning agent is calculated using the following formula:

[0017] in, A binary indicator variable representing the presence of key vehicles. A binary indicator variable representing the accessibility of critical vehicles. Indicates the weighting coefficient. Represents the cutoff constant. This represents the incentive term for sparse results. This indicates alignment error.

[0018] Further, step S3 includes the following steps: The state vector of the speed agent of a connected autonomous vehicle is constructed as follows:

[0019] in, and These represent the normalized distance between the vehicle and the vehicle in front, and the relative speed, respectively; signal phase and timing-related terms include phase indication. and the remaining time until the next phase switch ; The range of values ​​for the continuous longitudinal acceleration control command output by the speed agent of the connected autonomous vehicle is: ; The total reward for the speed agent of a connected autonomous vehicle is calculated using the following formula:

[0020] in, This indicates the weight of the corresponding reward item.

[0021] Fuel penalty and ride comfort penalty are calculated using the following formulas respectively:

[0022] in, and Used for Normalize, Used to avoid division by zero. Used to adjust the impact penalty scale; Item Square impact Punishment will be carried out, among which This is the acceleration of the previous control step; Traffic efficiency bonus is calculated using the following formula:

[0023] in, Indicates the estimated arrival time. Indicates the normalized travel progress; when in the green light phase, When in the red light phase, the target window is Early arrival corresponds to Late arrival corresponds to When the window is hit ; Speed ​​alignment bonus is calculated as follows:

[0024] in, Indicates the target time. This represents the asymmetric coefficient when the light is in the green phase. When in the red light phase And pick it up when you arrive early. Pick it up when you arrive late ; The safety item is based on headway. The calculation, where the safe headway is obtained by dividing the distance to the preceding vehicle by the vehicle's speed; when At that time, set Otherwise set ,when When unavailable, set .

[0025] Compared with the prior art, the present invention has the following advantages: 1. The present invention provides a hierarchical collaborative control method for signal timing and speed of connected autonomous vehicles based on reinforcement learning. By unifying and coordinating periodic timing control, phase-level online fine-tuning and vehicle-level speed control, it realizes the generation, refinement and tracking of traffic signal emission windows, thereby enabling the linkage control between the infrastructure side and the vehicle side at different time scales.

[0026] 2. Under the premise of keeping the phase sequence unchanged and the total cycle budget constrained, the present invention uses a phase fine-tuning mechanism to refine the green light termination time online, and combines it with the continuous longitudinal acceleration control of the target connected autonomous vehicle to match the vehicle arrival time with the emission window, thereby reducing the parking, queuing and delays caused by the mismatch between the signal release period and the vehicle arrival time.

[0027] 3. This invention introduces a bidirectional long short-term memory network to encode the temporal evolution of traffic states and combines it with a phased course learning and training mechanism to improve the stability, executability and robustness of reinforcement learning strategies under mixed traffic scenarios and multi-timescale coupling conditions. This is beneficial to improving the traffic efficiency of intersections and also takes into account vehicle operation safety, smoothness and fuel economy.

[0028] In summary, by constructing a hierarchical collaborative mechanism of periodic timing, phase-level fine-tuning, and vehicle-level speed control, the technical solution of this invention achieves unified linkage between traffic signal timing and connected autonomous vehicle speed control at different time scales. This effectively alleviates problems in existing technologies caused by the disconnect between traffic signal timing control and connected autonomous vehicle speed control, such as mismatch between emission windows and vehicle arrival times, frequent vehicle starts and stops, queue accumulation, and increased waiting times. Therefore, the technical solution of this invention solves the problems in existing technologies where traffic signal timing control and connected autonomous vehicle speed control are difficult to coordinate decision-making at different time scales, difficult to balance real-time performance and stability, and difficult to simultaneously reduce vehicle stopping and waiting times.

[0029] Based on the above reasons, this invention can be widely applied in fields such as intelligent traffic control, vehicle-road cooperative control, and signalized intersection optimization control. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the overall control framework for signal timing and speed layered collaborative control of connected autonomous vehicles according to the present invention.

[0032] Figure 2 This diagram illustrates the training results of a periodically timed agent under two configurations: without BiLSTM and with BiLSTM.

[0033] Figure 3This diagram illustrates the training results for the phase fine-tuning agent and the speed agent of the connected autonomous vehicle.

[0034] Figure 4 This is a schematic diagram comparing the method of the present invention with a control method in terms of vehicle travel time.

[0035] Figure 5 This is a schematic diagram comparing the average number of parked vehicles with the method of the present invention and the control method.

[0036] Figure 6 This is a schematic diagram comparing the average waiting time index between the method of the present invention and the control method. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] like Figures 1-6 As shown, this invention provides a reinforcement learning-based signal timing and speed hierarchical cooperative control method for connected and automated vehicles, applicable to mixed traffic scenarios involving both connected and manually driven vehicles at signalized intersections, comprising the following steps: S1. At the boundary of each signal cycle, obtain the status information of stopped vehicles on the entrance lanes served by each phase, and generate a reference timing scheme for the next cycle based on the stopped vehicle status information; the reference timing scheme includes: total green light budget and green light allocation for each phase; and the reference timing scheme satisfies the minimum green light duration constraint and the clearing time constraint; S1 includes the following steps: The state vector of a periodically timed agent is constructed using the following formula:

[0040] in, to These represent the total number of vehicles stopped in the corresponding entrance lanes for north-south straight traffic, north-south left turns, east-west straight traffic, and east-west left turns, respectively. The periodically timed agent outputs the action vector for the next period at the boundary of each period, and the action vector is expressed by the following formula:

[0041] in, This indicates the total green light budget parameters. to These represent the green light ratio allocation parameters for each phase; after normalization, these green light ratio allocation parameters are mapped to feasible green light durations under constraints. The periodic reward of a periodically timed agent is calculated using the following formula:

[0042] in, Indicates the actual cycle duration. Represents the normalization constant. This represents the sum of the total number of vehicles momentarily stopped across all phases; to accommodate changes in cycle length, the discount factor is adjusted to... ; The periodic timing agent employs a bidirectional long short-term memory network to process recent data. The embedding vector of a periodic state sequence is obtained by encoding it using the following formula:

[0043] in, This indicates a splicing operation. Represents a fully connected layer; each step in the sequence The internal dynamics of the LSTM cell are determined by the following gating mechanism:

[0044] in, Represents the Sigmoid function; Represents the Hadamard product; These represent the forget gate, input gate, and output gate, respectively. Represents the cell state; the resulting embedding vector As input to the actor network and critic network, it is used to output the baseline timing scheme for the next cycle.

[0045] S2. When the remaining time of the current green light phase reaches a preset trigger threshold, a key vehicle is selected, and the phase fine-tuning agent, based on the status of the key vehicle, the traffic pressure of the current green light phase, and the traffic pressure of the next phase, performs discrete time adjustment on the green light termination time of the currently active phase. While maintaining the phase sequence and the total cycle budget, the emission window is refined to obtain the refined emission window. Step S2 includes the following steps: The state vector of the phase-tuning agent is constructed as follows:

[0046] in, These represent the normalized distance, speed, and acceleration of the critical vehicle from the stop line, respectively. Indicates the estimated arrival time of the key vehicle. Indicates the feasibility ratio. Indicates the remaining green light time, parameter pair This indicates the normalized number of stopped vehicles and cumulative delays on the approach lanes served by the current green light phase, while the parameters are related to... This indicates the normalized number of stopped vehicles and the cumulative delay in the next phase group; The actions of the phase-tuning agent are represented by the following formula:

[0047] The This represents the discrete-time adjustment amount applied to the green light duration of the currently active phase, and is based on... Update the green light duration for the currently active phase; The reward for the phase-fine-tuning agent is calculated using the following formula:

[0048] in, A binary indicator variable representing the presence of key vehicles. A binary indicator variable representing the accessibility of critical vehicles. Indicates the weighting coefficient. Represents the cutoff constant. This represents the incentive term for sparse results. This indicates alignment error.

[0049] S3. Based on the refined emission window, the speed agent of the connected autonomous vehicle outputs continuous longitudinal acceleration control commands to the target connected autonomous vehicle, so that the arrival time of the target connected autonomous vehicle matches the emission window, thereby achieving coordinated control of traffic signal timing and vehicle speed. Step S3 includes the following steps: The state vector of the speed agent of a connected autonomous vehicle is constructed as follows:

[0050] in, and These represent the normalized distance between the vehicle and the vehicle in front, and the relative speed, respectively; signal phase and timing-related terms include phase indication. and the remaining time until the next phase switch ; The range of values ​​for the continuous longitudinal acceleration control command output by the speed agent of the connected autonomous vehicle is: ; The total reward for the speed agent of a connected autonomous vehicle is calculated using the following formula:

[0051] in, This indicates the weight of the corresponding reward item.

[0052] Fuel penalty and ride comfort penalty are calculated using the following formulas respectively:

[0053] in, and Used for Normalize, Used to avoid division by zero. Used to adjust the impact penalty scale; Item Square impact Punishment will be carried out, among which This is the acceleration of the previous control step; Traffic efficiency bonus is calculated using the following formula:

[0054] in, Indicates the estimated arrival time. Indicates the normalized travel progress; when in the green light phase, When in the red light phase, the target window is Early arrival corresponds to Late arrival corresponds to When the window is hit ; Speed ​​alignment bonus is calculated as follows:

[0055] in, Indicates the target time. This represents the asymmetric coefficient when the light is in the green phase. When in the red light phase And pick it up when you arrive early. Pick it up when you arrive late ; The safety item is based on headway. The calculation, where the safe headway is obtained by dividing the distance to the preceding vehicle by the vehicle's speed; when At that time, set Otherwise set ,when When unavailable, set .

[0056] Example 1 This embodiment describes the implementation process of steps S1 to S3 in a single four-phase signal-controlled intersection scenario. Preferably, the intersection includes north-south straight-ahead phases, north-south left-turn phases, east-west straight-ahead phases, and east-west left-turn phases. Each approach direction has three approach lanes and three exit lanes. The rightmost approach lane is for right turns, the middle approach lane is for straight-ahead traffic, and the leftmost approach lane is for left turns. Right-turning vehicles are allowed to pass under conflict-free traffic conditions. The length of each approach lane is preferably 300 meters, the maximum allowed speed is preferably 50 km / h, and a 3-second yellow light and a 2-second all-red light clearing time are provided after each phase.

[0057] In step S1, the periodic timing agent makes a decision once at each period boundary. Its periodic state can be represented by equation (1), and its action can be represented by equation (2). To ensure that the agent balances congestion pressure and feedback consistency under different period lengths at the periodic level, the periodic reward is calculated using equation (3), i.e.

[0058] in, Indicates the actual cycle duration. Represents the normalization constant. This represents the sum of the total number of vehicles momentarily stopped in each phase. To improve the ability to characterize the temporal evolution of traffic demand, the periodic timing agent analyzes the most recent... The embedding vector of a periodic state sequence is obtained by bidirectional temporal coding according to equation (4), i.e.

[0059] The resulting embedding vectors are used as input to the actor network and the critic network to output the baseline timing scheme for the next cycle. Through the above steps, the baseline timing scheme for the next cycle is generated.

[0060] In step S2, the phase fine-tuning agent performs fine-grained online correction when the remaining green light time reaches the trigger threshold. Its state can be represented by equation (6), which includes the distance, speed, acceleration, passage feasibility ratio, remaining green light time, and the normalized number of stopped vehicles and cumulative delays for the current and next phase groups. The key vehicles are selected based on the proximity of their expected arrival time to the current emission window. The action of the phase fine-tuning agent can be represented by equation (7), where the updated green light duration satisfies… ,in, This represents the discrete-time adjustment amount for the duration of the green light in the currently active phase. When A value greater than 0 indicates that the current green light will be extended. When When the value is less than 0, it indicates that the current green light is terminated in advance. To ensure the executability of the control results, actions that violate the minimum green light duration constraint or exceed the cycle budget are masked before action selection. The reward of the phase fine-tuning agent is calculated according to formula (8) and adaptively switches according to three situations: empty discharge, inaccessible key vehicle, and accessible key vehicle, thereby realizing online fine-tuning of the emission window.

[0061] In step S3, the speed agent of the connected autonomous vehicle applies continuous longitudinal acceleration control to the target connected autonomous vehicle based on the emission window determined jointly by periodic timing and phase fine-tuning. Its state can be represented by equation (9), and the preferred action value range is... To balance traffic efficiency, smoothness, fuel economy, and safety, the total reward for the vehicle speed agent is calculated according to formula (10), i.e.

[0062] Among them, the fuel penalty and impact penalty are calculated according to formula (11), the efficiency and speed alignment are calculated according to formula (12) and formula (13) respectively, and the safety item gives a reward or penalty based on the headway. Through the above reward structure, the vehicle can pass the stop line within the emission window as much as possible while ensuring following safety and ride comfort, and reducing stopping and unnecessary speed fluctuations.

[0063] During training, a phased learning approach is preferred. First, a periodic timing agent is trained, and its parameters are fixed after convergence. Next, a phase fine-tuning agent is trained. Finally, based on the fixed parameters of the first two agents, a speed agent for the connected autonomous vehicle is trained. During online operation, the periodic timing agent generates a baseline timing scheme for the next cycle at the cycle boundary. The phase fine-tuning agent refines the green light end time based on the key vehicle status at the end of the green light period. The speed agent outputs longitudinal acceleration control to the target vehicle at fixed update intervals, thereby achieving coordinated optimization of signal timing and vehicle speed.

[0064] Example 2 This embodiment constructs a single-intersection mixed traffic flow scenario as described in Embodiment 1 on the SUMO simulation platform to illustrate the implementation process and operational effects of the present invention under time-varying traffic demand conditions. Preferably, the left-turn traffic demand is fixed at 100 vehicles / hour, and the straight-through traffic demand switches every 900 seconds, successively representing low flow, north-south peak, low flow, and east-west peak. During the low flow phase, the straight-through flow in each direction is 300 vehicles / hour. During the north-south peak phase, the north-south straight-through flow increases to 1800 vehicles / hour, while the east-west straight-through flow decreases to 200 vehicles / hour. The east-west peak phase employs the opposite flow configuration.

[0065] like Figure 2 As shown, after training, the training results of the periodic timing agent gradually stabilize, indicating that this implementation can form a stable periodic timing output. Furthermore, after introducing BiLSTM, the periodic timing agent can use the state history information of the most recent multiple cycles to capture the temporal evolution pattern of traffic demand, indicating that in this embodiment, the training results of the periodic timing agent are more stable after introducing temporal coding.

[0066] like Figure 3 As shown, both the phase fine-tuning agent and the speed agent of the connected autonomous vehicle can output stable control results after training. Specifically, the phase fine-tuning agent can stably output control results for extending, maintaining, and truncating green lights after training. Although the speed agent of the connected autonomous vehicle is affected by penalties such as fuel consumption, impact, and safety, and its reward value fluctuates near the negative range, it also tends to stabilize as training progresses. This indicates that under this implementation, the vehicle speed agent can output smooth and executable control commands within the emission window constraint. These results demonstrate that, in this embodiment, the output of each control module is more stable after adopting a phased training method.

[0067] The preferred parameter settings are shown in Table 1.

[0068] Table 1 Optimal Parameter Settings

[0069] Example 3 Under the simulation environment described in Example 2 and the parameter settings shown in Table 1, the method of the present invention is compared with a fixed timing method, a control method using only periodic timing control (HCTVRL-C), and a control method using both periodic timing and phase fine-tuning control (HCTVRL-CP). The fixed timing method uses a static four-phase timing scheme, with each phase having a green light time of 20 seconds, and a cycle length of 100 seconds including the yellow light and all-red light times.

[0070] This embodiment uses travel time, average number of parked vehicles, and average waiting time as performance evaluation metrics. Travel time characterizes the total travel time from departure to arrival. Average number of parked vehicles characterizes the average number of vehicles with speeds below 0.1 m / s during the simulation. Average waiting time characterizes the average of the cumulative waiting time for all vehicles during the simulation.

[0071] like Figure 4 As shown, the vehicle travel time distribution under the method of the present invention is more concentrated and the median is lower, indicating that the method of the present invention has better stability and robustness. Preferred experimental results show that the average travel time of the method of the present invention is 156.96 seconds, which is lower than that of the fixed timing method. Compared to the HCTVRL-C method, it reduces Compared to the HCTVRL-CP method, it reduces The results above show that by combining phase-level fine-tuning and vehicle-level speed control with periodic timing, the travel time of vehicles through intersections can be further shortened.

[0072] like Figure 5 As shown, during the time-varying traffic demand switching process, the average number of parked vehicles using the method of this invention remains at a low level. Compared with the fixed timing method, the HCTVRL-C method, and the HCTVRL-CP method, HCTVRL can reduce the average number of parked vehicles by [percentage missing]. , , The results above demonstrate that combining periodic green light budget allocation, phase-level online refinement, and vehicle-level speed control can reduce stoppages and queue accumulation caused by mismatch between vehicle arrival and signal emission windows.

[0073] like Figure 6 As shown, under the same conditions, the average waiting time of the method of the present invention is significantly lower than that of the comparative methods, and the waiting time curve increases more gradually and fluctuates less as demand shifts from low flow to north-south peak and then to east-west peak. Preferred experimental results show that, compared with the fixed timing method, the HCTVRL-C method, and the HCTVRL-CP method, the method of the present invention can reduce the average waiting time by […]. , , The above results demonstrate that, under conditions of dynamic changes in traffic demand, the method of this invention can reduce average waiting time and improve intersection operational efficiency.

[0074] In summary, under the conditions described in Examples 1 to 3, the present invention enables effective coupling of traffic signal timing decisions and speed control of connected autonomous vehicles at multiple time scales through periodic, phase, and vehicle-level coordinated control, thereby improving intersection traffic efficiency and reducing vehicle travel time, number of parked vehicles, and waiting time.

[0075] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning-based signal timing and speed hierarchical cooperative control method for connected and automated vehicles, applied to a mixed traffic scenario at signalized intersections involving both connected and manually driven vehicles, characterized in that... Includes the following steps: S1. At the boundary of each signal cycle, obtain the status information of the stopped vehicles on the entrance lane served by each phase, and generate the reference timing scheme for the next cycle based on the status information of the stopped vehicles. The benchmark timing scheme includes: total green light budget and green light allocation for each phase; and the benchmark timing scheme satisfies minimum green light duration constraint and clearing time constraint. S2. When the remaining time of the current green light phase reaches the preset trigger threshold, a key vehicle is selected, and the phase fine-tuning agent adjusts the green light termination time of the current active phase in a discrete time manner based on the status of the key vehicle, the traffic pressure of the current green light phase, and the traffic pressure of the next phase. The emission window is refined under the premise of keeping the phase sequence unchanged and the total cycle budget unchanged, and the refined emission window is obtained. S3. Based on the refined emission window, the speed intelligent agent of the connected autonomous vehicle outputs continuous longitudinal acceleration control commands to the target connected autonomous vehicle, so that the arrival time of the target connected autonomous vehicle matches the emission window, thereby realizing the coordinated control of traffic signal timing and vehicle speed.

2. The method for hierarchical cooperative control of signal timing and speed of connected autonomous vehicles based on reinforcement learning according to claim 1, characterized in that, S1 includes the following steps: The state vector of a periodically timed agent is constructed using the following formula: in, to These represent the total number of vehicles stopped in the corresponding entrance lanes for north-south straight traffic, north-south left turns, east-west straight traffic, and east-west left turns, respectively. The periodically timed agent outputs the action vector for the next period at the boundary of each period, and the action vector is expressed by the following formula: in, This indicates the total green light budget parameters. to These represent the green light ratio allocation parameters for each phase; after normalization, these green light ratio allocation parameters are mapped to feasible green light durations under constraints. The periodic reward of a periodically timed agent is calculated using the following formula: in, Indicates the actual cycle duration. Represents the normalization constant. This represents the sum of the total number of vehicles momentarily stopped across all phases; to accommodate changes in cycle length, the discount factor is adjusted to... ; The periodic timing agent employs a bidirectional long short-term memory network to process recent data. The embedding vector of a periodic state sequence is obtained by encoding it using the following formula: in, This indicates a splicing operation. Represents a fully connected layer; each step in the sequence The internal dynamics of the LSTM cell are determined by the following gating mechanism: in, Represents the Sigmoid function; Represents the Hadamard product; These represent the forget gate, input gate, and output gate, respectively. Represents the cell state; the resulting embedding vector As input to the actor network and critic network, it is used to output the baseline timing scheme for the next cycle.

3. The signal timing and speed hierarchical cooperative control method for connected autonomous vehicles based on reinforcement learning according to claim 1, characterized in that, Step S2 includes the following steps: The state vector of the phase-tuning agent is constructed as follows: in, These represent the normalized distance, speed, and acceleration of the critical vehicle from the stop line, respectively. Indicates the estimated arrival time of the key vehicle. Indicates the feasibility ratio. Indicates the remaining green light time, parameter pair This indicates the normalized number of stopped vehicles and cumulative delays on the approach lanes served by the current green light phase, while the parameters are related to... This indicates the normalized number of stopped vehicles and the cumulative delay in the next phase group; The actions of the phase-tuning agent are represented by the following formula: The This represents the discrete-time adjustment amount applied to the green light duration of the currently active phase, and is based on... Update the green light duration for the currently active phase; The reward for the phase-fine-tuning agent is calculated using the following formula: in, A binary indicator variable representing the presence of key vehicles. A binary indicator variable representing the accessibility of critical vehicles. Indicates the weighting coefficient. Represents the cutoff constant. This represents the incentive term for sparse results. This indicates alignment error.

4. The method for hierarchical cooperative control of signal timing and speed of connected autonomous vehicles based on reinforcement learning according to claim 1, characterized in that, Step S3 includes the following steps: The state vector of the speed agent of a connected autonomous vehicle is constructed as follows: in, and These represent the normalized distance between the vehicle and the vehicle in front, and the relative speed, respectively; signal phase and timing-related terms include phase indication. and the remaining time until the next phase switch ; The range of values ​​for the continuous longitudinal acceleration control command output by the speed agent of the connected autonomous vehicle is: ; The total reward for the speed agent of a connected autonomous vehicle is calculated using the following formula: in, This indicates the weight of the corresponding reward item; Fuel penalty and ride comfort penalty are calculated using the following formulas respectively: in, and Used for Normalize, Used to avoid division by zero. Used to adjust the impact penalty scale; Item Square impact Punishment will be carried out, among which This is the acceleration of the previous control step; Traffic efficiency bonus is calculated using the following formula: in, Indicates the estimated arrival time. Indicates the normalized travel progress; when in the green light phase, When in the red light phase, the target window is Early arrival corresponds to Late arrival corresponds to When the window is hit ; Speed ​​alignment bonus is calculated as follows: in, Indicates the target time. This represents the asymmetric coefficient when the light is in the green phase. When in the red light phase And pick it up when you arrive early. Pick it up when you arrive late ; The safety item is based on headway. The calculation, where the safe headway is obtained by dividing the distance to the vehicle in front by the vehicle's speed; when At that time, set Otherwise set ,when When unavailable, set .

Images