An adaptive lamp vehicle cooperative rolling control method based on deep reinforcement learning

Abstract

An adaptive traffic light-vehicle coordinated rolling control method based on deep reinforcement learning is disclosed, belonging to the field of intelligent transportation technology. This invention addresses the problems of low intersection traffic efficiency and high vehicle energy consumption in existing control methods. By using a single agent to control one traffic flow and multiple agents to control the intersection, this invention achieves more precise and flexible signal control, eliminating the need for two traffic flows to terminate simultaneously. Both signal timing and vehicle trajectory adjustment employ multi-agent deep reinforcement learning. In the signal timing phase, a long short-term memory neural network is used to fully exploit the temporal features of the traffic light agent's state information. In the vehicle trajectory adjustment phase, a multi-head attention mechanism is used as a feature extractor to extract key features from the state information, enhancing the agent's understanding and utilization of these features. The optimal control scheme is obtained through the interaction between the signal timing module and the vehicle trajectory adjustment module. This method can be applied to intelligent transportation systems.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, specifically relating to an adaptive light and vehicle cooperative rolling control method based on deep reinforcement learning. Background Technology

[0002] With rapidly increasing traffic demand, traffic congestion has become a prominent issue, causing numerous problems such as fuel waste and traffic delays. In recent years, connected and automated vehicle (CAV) technology has received considerable attention due to its real-time communication and controllability capabilities, which can be used to improve traffic conditions and address traffic problems caused by human-driven vehicles (HVs). However, it is unlikely that 100% of vehicles on the road will be CAVs within the next two decades. Therefore, a transition period where HVs and CAVs share road networks is inevitable, and traffic management strategies must be developed for this mixed traffic environment.

[0003] Optimizing signal timing alone or using CAVs for ecological trajectory control has limited effect on reducing energy consumption and improving intersection operating efficiency. Therefore, it is considered to incorporate vehicle trajectory and signal timing into a single framework for optimization to further improve intersection operating efficiency and reduce energy consumption. Current research employs deep reinforcement learning, which eliminates some complex modeling processes. The agent can determine the signal phase and duration based on real-time spatiotemporal traffic characteristics around the intersection and determine the optimal speed for CAVs, demonstrating good performance and control effects. However, most studies use fixed signal phases for coordinated control of signal timing and vehicle trajectory. The timing scheme lacks flexibility, and fixed phase switching results in both traffic flows ending simultaneously during traffic imbalances. The low-flow traffic flow may waste green time, while the high-flow traffic flow cannot be fully released, hindering further improvement in traffic efficiency. The effect on optimizing vehicle energy consumption is also limited, and there is insufficient feedback adjustment between traffic lights and intelligent connected vehicles. Secondly, for traffic problems with time-related characteristics such as morning and evening rush hours, directly applying deep reinforcement learning methods is insufficient to fully explore the features of state information, which reduces the practical application effect of deep reinforcement learning methods in signal timing and vehicle trajectory collaborative control, further resulting in low traffic efficiency and high vehicle energy consumption at intersections.

[0004] In summary, existing control methods still suffer from low intersection traffic efficiency and high vehicle energy consumption. Therefore, in order to further improve intersection operating efficiency and reduce energy consumption, it is an urgent need to propose a new light-vehicle coordinated control method. Summary of the Invention

[0005] The purpose of this invention is to address the problems of low traffic efficiency and high vehicle energy consumption at intersections in existing control methods, and to propose an adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is: an adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning, the method specifically including the following steps:

[0007] Step 1: Estimate the arrival time of vehicles at the stop line for each direction of traffic flow that needs to be controlled at the target intersection.

[0008] Step 2: Based on the arrival time of vehicles at the stop line in each traffic flow direction, obtain local observation information for each traffic flow direction, and use the local observation information for each traffic flow direction as the state information for each traffic flow direction. Then, pass the state information for each traffic flow direction through an LSTM network, and output the temporal features of the state information for each traffic flow direction through the LSTM network.

[0009] The temporal characteristics of the state information of each traffic flow direction are input into the corresponding traffic light agent, and the action set is generated by each traffic light agent.

[0010] Step 3: Determine a signal control cycle based on the action set generated by each traffic light agent. Each agent corresponding to each CAV in each traffic flow direction plans the CAV motion trajectory according to the phase allocation within the signal control cycle.

[0011] Step 4: The CAVs in each traffic flow direction drive according to the motion trajectory planning results. After the current signal control cycle ends, the arrival time of vehicles in each traffic flow direction to the stop line is estimated again based on the motion trajectory planning results.

[0012] Based on the estimated arrival times of vehicles at the stop line in each traffic flow direction, return to step two.

[0013] The beneficial effects of this invention are:

[0014] This invention achieves more precise and flexible signal control by using a single agent to control one traffic flow and multiple agents to control the intersection. Two traffic flows that are being allowed to proceed do not need to terminate simultaneously, resulting in higher traffic efficiency. Both signal timing and vehicle trajectory adjustment employ multi-agent deep reinforcement learning. In the signal timing phase, a long short-term memory neural network is used to fully exploit the temporal features of the traffic light agent's state information. In the vehicle trajectory adjustment phase, a multi-head attention mechanism is used as a feature extractor to extract key features from the state information, enhancing the agent's understanding and utilization of these features. During the training phase, priority experience replay is used to improve network learning efficiency. The optimal control scheme is obtained through the interaction between the signal timing module and the vehicle trajectory adjustment module. This invention does not require a fixed phase sequence during signal timing, allowing for flexible signal control. The adaptive light-vehicle cooperative rolling control method of this invention fully leverages the advantages of intelligent connected vehicles, guiding manually driven vehicles and coordinating with traffic lights to significantly alleviate traffic congestion, increase intersection throughput, and reduce vehicle energy consumption. Attached Figure Description

[0015] Figure 1 This is a flowchart of an adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning according to the present invention;

[0016] Figure 2 This is a schematic diagram of the internal structure of the collaborative control method of the present invention;

[0017] Figure 3 This is a schematic diagram of the simulation scenario of the present invention;

[0018] Figure 4 This is a schematic diagram of the rolling control strategy of the signal scheme of the present invention;

[0019] Figure 5a This is a schematic diagram showing the trajectory of vehicles turning left from the east entrance lane;

[0020] Figure 5b A schematic diagram showing the trajectory of vehicles traveling straight on the east entrance lane;

[0021] Figure 5c This is a schematic diagram of the trajectory of vehicles turning left from the west entrance.

[0022] Figure 5d This is a schematic diagram of the trajectory of vehicles traveling straight on the west entrance lane. Detailed Implementation

[0023] Specific Implementation Method 1: Combination Figure 1 and Figure 2 This embodiment describes an adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning. The method specifically includes the following steps:

[0024] Step 1: Estimate the arrival time of vehicles at the stop line for each direction of traffic flow that needs to be controlled at the target intersection.

[0025] Step 2: Based on the arrival time of vehicles at the stop line in each traffic flow direction, obtain local observation information for each traffic flow direction, and use the local observation information for each traffic flow direction as the state information for each traffic flow direction. Then, pass the state information for each traffic flow direction through an LSTM network, and output the temporal features of the state information for each traffic flow direction through the LSTM network.

[0026] The temporal characteristics of the traffic flow direction status information are input into the corresponding traffic light agent (the traffic flow directions of east entrance straight, east entrance left turn, west entrance straight, west entrance left turn, south entrance straight, south entrance left turn, north entrance straight, and north entrance left turn are numbered 1-8 respectively, and the traffic light is numbered the same as the traffic flow direction it controls. The right turn traffic flow of each entrance lane does not need to be controlled). The action set is generated by each traffic light agent.

[0027] Step 3: Determine a signal control cycle based on the action set generated by each traffic light agent (in this invention, the signal control cycle is defined as at least one complete green light phase being released in all eight traffic flow directions. For example, when the 6th element of the action vector output by the traffic light agent is reached, at least one complete green light phase is released in each direction of traffic flow, and the duration corresponding to the first 6 elements is one signal control cycle). Each agent corresponding to each CAV in each traffic flow direction (one agent for each CAV) performs CAV motion trajectory planning based on the phase allocation within the signal control cycle (the planning method is the same as the motion planning method in the process of estimating the arrival time of the leading vehicle when the leading vehicle is a CAV).

[0028] Step 4: CAVs in each traffic flow direction drive according to the motion trajectory planning results (they adjust their arrival time and driving trajectory reasonably through the planned signal scheme, and indirectly guide HVs, so as to minimize the situation of vehicles stopping in front of the intersection during red light time, so that the vehicles arrive at the intersection during green light time, and improve the green light time throughput). After the current signal control cycle ends, the arrival time of vehicles in each traffic flow direction to the stop line is estimated again based on the motion trajectory planning results.

[0029] Based on the estimated arrival times of vehicles at the stop line in each traffic flow direction, return to step two.

[0030] When traffic flow is unbalanced, a fixed phase switching method results in both traffic streams ending simultaneously. This can lead to wasted green time for the low-flow traffic stream and incomplete release for the high-flow traffic stream. To address this issue, this invention utilizes multi-agent control of multiple traffic streams, increasing the number of feasible solutions and resulting in more flexible signal schemes. This leads to a more efficient control scheme. Furthermore, a rolling control strategy ensures the accuracy of the execution phase. As new vehicles enter, the control scheme is redesigned upon completion of the first phase, making the control more effective.

[0031] To address the shortcomings of traditional methods in terms of interactivity and feedback, this invention utilizes a long short-term memory neural network to fully mine the temporal features of the traffic light agent's state information during the signal timing stage. During the vehicle trajectory adjustment stage, a multi-head attention mechanism is used as a feature extractor to extract key features from the state information, enhancing the agent's understanding and utilization of these features. This method enables full interaction between the traffic light and the intelligent connected vehicle, allowing for adjustments and optimization based on feedback, thereby improving the adaptability and optimization capabilities of the method.

[0032] The method of this invention can effectively increase the throughput of intersections, reduce the number of waiting vehicles and waiting time, and reduce emissions pollution, which can further enhance the application effect and practical value in the field of intelligent transportation.

[0033] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the target intersection is an intersection with four entrances. The traffic flow directions that need to be controlled at the intersection include the east entrance straight, the east entrance left turn, the west entrance straight, the west entrance left turn, the south entrance straight, the south entrance left turn, the north entrance straight, and the north entrance left turn.

[0034] The other steps and parameters are the same as in Specific Implementation Method 1.

[0035] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the specific process of step one is as follows:

[0036] For vehicles within traffic flow direction i that have not reached the stop line

[0037] Step 11: Divide the vehicles into two parts: the leading vehicle and the following vehicle. The arrival time of the leading vehicle at the stop line is estimated using Step 12, and the arrival time of the following vehicle at the stop line is estimated using Step 13.

[0038] If the current vehicle is a certain distance from the vehicle in front (the distance threshold can be set based on experience), the current vehicle's movement will not be affected by the vehicle in front, and the arrival time of the current vehicle can be estimated in the same way as the leading vehicle.

[0039] Steps 1 and 2: When the leading vehicle is a manually driven vehicle (HV) and its ecological speed is higher than the maximum speed limit on the road (the leading vehicle should travel at the maximum speed), the arrival time of the leading vehicle is... for:

[0040]

[0041] Among them, L k L represents the distance between the leading car and the stop line. k =x f -x0, where x0 represents the current position of the leading vehicle, x f This indicates the position of the stop line, v0 represents the current speed of the leading car, and v max Indicates the maximum speed limit for vehicles on the road, a max This indicates the maximum allowable acceleration for the leading vehicle;

[0042] When the leading vehicle is a manually driven vehicle (HV) and its eco-vehicle speed is lower than or equal to the maximum speed limit on the road (the leading vehicle should travel at its eco-vehicle speed), the arrival time of the leading vehicle is... for:

[0043]

[0044] Among them, v e Indicates the eco-speed of the leading vehicle;

[0045] When the leading vehicle is a CAV, the arrival time of the leading vehicle The estimation method is as follows:

[0046] Step 1: The leading vehicle obtains its own current position and current speed as well as the current position and speed of the vehicle in front. This then forms the state information observed by the leading vehicle at the current moment. for:

[0047]

[0048] In the formula, d p d represents the distance from the vehicle in front to the stop line. k v represents the distance from the leading car to the stop line. p The speed of the vehicle in front is represented by v. k μ represents the current speed of the leading vehicle. k This indicates the current acceleration of the leading vehicle. This indicates the current signal state of traffic flow direction i where the leading vehicle is located;

[0049] If the traffic flow direction i is currently on a green light, then The value is set to 1, where ξ represents the remaining time of the current green light signal for traffic flow direction i;

[0050] If the traffic flow direction i is currently facing a red light, then The value is set to 0, where ξ represents the time elapsed since the next green light begins to open, i is the distance from the current time in the direction of traffic flow i.

[0051] Step 2: Record the status information observed by the leading vehicle at the current moment. The input feature extractor is used as the input to the agent corresponding to the leading vehicle, and the agent outputs the action selected by the leading vehicle. μ represents the adjusted acceleration of the leading vehicle at the current moment. k (t)∈[-3,3];

[0052] The feature extractor is a multi-head attention layer that uses linear transformation to extract the input state information. The algorithm maps to multiple spaces, with each head independently calculating weights and applying them to its corresponding value vector to obtain a weighted average attention output. Finally, the outputs of all heads are concatenated, and the concatenated result is passed through a linear transformation layer to obtain the final multi-head attention output. These are the key features extracted from complex state information;

[0053] Step 3: After the leading vehicle performs the action selected in Step 2, it reaches a new state and then uses the new state to return to and execute Step 2 until the time for the leading vehicle to reach the parking line is planned.

[0054] Step 13: The arrival time of the following vehicle is:

[0055] t f-k+1 =t f-k +t h

[0056] Among them, t h This indicates the safe time interval (t) between the following vehicle and the vehicle in front. h This is derived from a safety-based following model. It assumes that the vehicle behind maintains a safe headway with the vehicle in front, and that if the vehicle in front suddenly brakes, the driver has time to react and slow down to a stop to avoid a collision. f-k t represents the estimated time it takes for the vehicle in front to reach the stop line. f-k+1 This indicates the time it takes for the following vehicle to reach the stop line.

[0057] Other steps and parameters are the same as in specific implementation method one or two.

[0058] When the leading vehicle is a CAV, the leading vehicle must follow the following constraints when adjusting its trajectory:

[0059]

[0060] x(t0)=x0

[0061] v(t0)=v0

[0062] x(t f )=x f

[0063] v(t f ) = v f

[0064] v min ≤v(t)≤v max

[0065] a min ≤μ(t)≤a max

[0066] The aforementioned constraints ensure smooth road operation and, on the other hand, limit the maximum speed of the road to guarantee that the trajectory adjustment process does not violate road regulations. When following another vehicle, this invention employs the Krauss model, a safety-based following model. The formula for the Krauss model is:

[0067]

[0068] In the formula, v k+1 τ represents the speed of the vehicle in front, v represents the driver's reaction time. k The following vehicle's current speed is represented by d, the distance between the two vehicles is represented by b, and the maximum braking deceleration is represented by v. safe This indicates a safe following speed.

[0069] The arrival times of all CAVs in traffic flow direction i are expressed as:

[0070]

[0071] The arrival times of all traffic vehicles (HVs) in traffic flow direction i are expressed as:

[0072]

[0073] The output of the vehicle arrival time estimation module for traffic flow direction i is:

[0074]

[0075] Similarly, the method described in this embodiment is used to estimate the arrival time of vehicles at the stop line in each traffic flow direction. Furthermore, during the initial control scheme planning, vehicle arrival times need to be estimated based on conventional signal phases.

[0076] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, in step 3, the time for the leading vehicle to reach the parking line is:

[0077] Calculate the time it takes for the leading car to reach the stop line based on the planned actions.

[0078]

[0079] Among them, f k (v k (t),μ k (t) represents instantaneous energy consumption, and t0 represents the current time.

[0080] (1) When the leading CAV is an electric vehicle, the formula for calculating instantaneous energy consumption is:

[0081]

[0082] Where η represents the vehicle powertrain efficiency, M represents the vehicle mass, g represents gravitational acceleration, u represents rolling resistance, and v k (t) represents the vehicle's speed, A represents the vehicle's frontal area, and C represents the vehicle's frontal area. d δ represents air resistance, and δ represents the rotational mass coefficient.

[0083] (2) When the leading CAV is a gasoline vehicle, the formula for calculating instantaneous energy consumption is:

[0084]

[0085] Where α, β1, β2, b1, b2, and b3 are the coefficients of the instantaneous energy consumption model, and R0 T G is the vehicle torque, and G is the road gradient.

[0086] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0087] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the specific process of step two is as follows:

[0088] Step 2: Record the number of vehicles queuing in the i-th direction of traffic flow in front of the stop line as q. i (t), the average waiting time w for vehicles queuing in the i-th direction of traffic flow in front of the stop line. i (t) is:

[0089]

[0090] Among them, w k Let l represent the waiting time of the kth vehicle in the i-th direction of traffic flow queuing in front of the stop line, and l represent the total number of vehicles in the i-th direction of traffic flow queuing in front of the stop line.

[0091] Vehicle waiting time is defined as the difference between the time it takes for a vehicle to start moving from the stop line and the time it takes for the vehicle to remain stationary before the stop line, w. k Specifically:

[0092] w k =t a,k -t f,k

[0093] Among them, t f,k t represents the time it takes for the k-th vehicle in the i-th direction of the traffic flow to come to a stop in front of the stop line. a,k This represents the time when the kth vehicle in the i-th direction of traffic queuing in front of the stop line begins to move due to the green light;

[0094]

[0095] Where, θ i Let be the time when the next green light begins for the i-th traffic flow direction;

[0096] Step 22: Calculate the smoothness V of vehicle movement in the i-th traffic flow direction. i (t):

[0097]

[0098] In the formula, v i (t) represents the average speed of all vehicles in the i-th traffic flow direction at time t, v i,k (t) represents the speed of the k-th vehicle in the i-th traffic flow direction at time t, m i This represents the total number of vehicles in the i-th traffic flow direction at time t;

[0099] Steps 2 and 3: Local observation information for the i-th traffic flow direction for:

[0100]

[0101] Step 24: Obtain the local observation information for the i-th traffic flow direction. The information is converted to the input format of the LSTM network. The converted information is then passed through the LSTM network. The temporal features output by the LSTM network are input into the traffic light agent corresponding to the i-th traffic flow direction. The traffic light agent corresponding to the i-th traffic flow direction outputs an action vector containing F elements.

[0102] The constraints that the action vector output by the traffic light agent must satisfy are:

[0103]

[0104]

[0105] Among them, t s To plan the start time, C n' This represents the duration of the n'th control cycle. This represents the duration of the green light for the i-th traffic flow direction in the n-th control cycle. This indicates the start time of the green light for the i-th traffic flow direction in the n-th control cycle;

[0106] To prevent frequent switching and excessively long green light times for a particular traffic flow direction, there are constraints on green light duration:

[0107]

[0108] Among them, g min g represents the shortest green light duration for a phase. max This indicates the longest green light duration for a given phase.

[0109] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0110] In the action vector output by the agent, each element has a value of either 0 or 1. 1 indicates a green light for passage, and 0 indicates a red light for passage. Taking an action vector containing 10 elements as an example, the corresponding times for each element are 10s, 20s, 30s, 40s, 50s, 60s, 70s, 80s, 90s, and 100s. When an element is 1, if the previous element is also 1, it means that the green light will continue for the next 10 seconds. If the previous element is 0, it means that the signal turns green at the time corresponding to element 1. The set of actions of each traffic light agent at a given moment is A = {1,1,0,0,0,0,0,0}, indicating that the traffic flow going straight and turning left at the east entrance is in a green light state, while the traffic lights for all other traffic flows are red. In this invention, the signal planning duration T can be set appropriately long, ensuring that the planned signal duration includes at least two signal control cycles, meaning that each traffic flow direction's phase is released at least twice. The CAV agent plans its movement based on the planned signal release phases. After the CAV agent completes its planning, each CAV will only move according to the movement planning result corresponding to the phase of the first control cycle. After the phase of one control cycle ends, the vehicle arrival time is re-estimated based on the previously planned phase of the second control cycle. Based on the arrival time estimation result, a second signal release planning and CAV movement planning are then performed.

[0111] Define a set of conflicting flows Φ, which contains conflicting traffic flow pairs (i, j), as shown in Table 1. 1 indicates that the two traffic flows do not conflict, and 0 indicates that the two traffic flows conflict.

[0112] Table 1

[0113]

[0114]

[0115] Introducing Passage Status For the i-th traffic flow direction where the traffic light is green... The value of is 1; for the i-th traffic flow direction where the traffic light is red, The value of is 0.

[0116] Conflicting traffic pairs have the following relationship:

[0117]

[0118] During model training, the agent outputs the action that yields the highest reward among non-conflicting actions.

[0119] The method described in this embodiment can be used to obtain the action vector output by each traffic light agent.

[0120] Specific Implementation Method Six: Combination Figure 1 This embodiment is described below. The difference between this embodiment and one of specific embodiments one to five is that the traffic light agent and the agent corresponding to the CAV are jointly trained. The specific training method is as follows:

[0121] Step 1: Initialize the Actor network parameters and Critic network parameters of the agent.

[0122] For traffic light agents, the Actor network is used to generate signal control actions based on the current traffic conditions; for agents corresponding to traffic avian vehicles (CAVs), the Actor network is used to output the acceleration of the CAVs based on the current signal conditions (adjusting the CAV's trajectory); and the Critic network for each agent is used to evaluate the expected reward under a given state and action.

[0123] The target Actor network and the target Critic network are set as reference targets for the training process, respectively.

[0124] Step 2: Initialize the training batch size as M', the training step size for each batch as B, and the pre-training step size as t. pre Experience replay pool The training sample size is Y, and the signal planning duration is T, meaning that the duration corresponding to the action output by each agent is T.

[0125] And estimate the time it takes for vehicles in each traffic flow direction to reach the stop line;

[0126] Step 3: Initialize the training batch m = 1;

[0127] Step 4: Initialize the training step size b = 1;

[0128] Step 5: Initialize the pre-training steps t' = 1;

[0129] Step 6: The traffic light agent for the i-th traffic flow direction obtains initial observation information. The initial observation information of all traffic flow directions is used to construct the state information of the traffic light agent in the i-th traffic flow direction at the current step. i = 1, 2, ..., N L ;

[0130] This invention employs a centralized training and distributed execution paradigm. Each agent has its own complete algorithm flow, but during training, data sharing is used, and the decisions of other agents are considered during policy adjustments. The agents interact with the environment; the traffic light agent obtains initial observation information. Each traffic light agent shares traffic observation data with each other to generate state information for the current time step.

[0131] status information The input format is converted to that of an LSTM network, and then the converted state is used as the input to the LSTM network to obtain the output features of the LSTM network.

[0132] Features As the input to the Actor network of the traffic light agent in the i-th traffic flow direction, the Actor network of the traffic light agent in the i-th traffic flow direction outputs action a. i ;

[0133] Step 7: Transfer global status information The output of LSTM and the actions of each agent The input to the Critic network of the traffic light agent in the i-th traffic direction is used to output the joint action value function.

[0134] Step 8: Obtain initial observation information for the k-th CAV in the i-th traffic flow direction. Will The input of the feature extractor is used as the input of the feature extractor, and the output of the feature extractor is used as the input of the Actor network corresponding to the k-th CAV in the i-th traffic flow direction. The Actor network then outputs action a'. i,k And the actions output by the Actor network for each CAV corresponding to the intelligent agent in each traffic flow direction are recorded as follows: Where, N Veh This represents the total number of CAVs in each traffic flow direction;

[0135] Global state information is constructed using initial observation information of all CAVs in all traffic flow directions. Then transfer the global state information The output of the feature extractor and Input the Critic network corresponding to the k-th CAV in the i-th traffic flow direction, and output the joint action value function through the Critic network.

[0136] Step 9: The agent continues to interact with the environment to obtain the local observation data of the traffic light agent at the next moment. and local observation data of the CAV corresponding agent. Rewards are calculated for each traffic light agent and the agent corresponding to each CAV;

[0137] Global state data is constructed by using the local observation data of each traffic light agent at the next moment. global state data The LSTM model's output is then used as the input to the target Actor network of the traffic light agent in the i-th traffic direction. The target Critic network of the traffic light agent in the i-th traffic direction then uses the output of the target Actor network as the basis for its operation. and global state data Calculate the target Q value;

[0138] Global state data is constructed using local observation data from each CAV corresponding agent. global state data The feature extractor's output is then used as the input to the target Actor network for each CAV agent. The target Critic network for each CAV agent outputs the target Actor network's output based on the CAV agent's target Actor network. With global state data Output the target Q value;

[0139] Step 10: Sample the empirical data Store in experience pool D L In the middle, the empirical data sample Store in experience pool D Veh In the process, the states of each traffic light agent are transferred to... The state transition of the agents corresponding to each CAV to

[0140] Step 11: Determine if the pre-training step size t has been reached. pre ;

[0141] If the condition is met, proceed to step 12.

[0142] If the target is not reached, then set t' = t' + 1 and return to step 6.

[0143] Step 12, from experience pool D L Y pieces of empirical data are extracted from the pool to update the parameters of the Critic and Actor networks of the traffic light agent. Then, the parameters of the target Critic and Actor networks of the traffic light agent are updated based on the parameters of the Critic and Actor networks. From the experience pool D... Veh Y empirical data points are extracted to update the parameters of the Critic and Actor networks of the CAV agent, and then the parameters of the target Critic and Actor networks of the CAV agent are updated based on the parameters of the Critic and Actor networks.

[0144] Determine whether the training step size B has been reached. If it has, continue to step 13. If it has not been reached, let b = b + 1 and return to step 5.

[0145] Step 13: Determine whether the training batch M' has been reached;

[0146] If this is achieved, then the trained traffic light agent and CAV agent will be obtained;

[0147] If the target is not reached, set m = m + 1 and return to step 4.

[0148] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0149] Only through coordinated interaction among multiple traffic light agents can the optimal signal phase scheme be determined to improve the overall efficiency of the intersection and reduce delays. Multiple CAVs cooperate to increase green light throughput when energy depletion is unlikely. The Actor network for the CAV agent consists of four layers: one input layer, two hidden layers, and one output layer. The hidden layers use the ReLU activation function, and the output layer uses the tanh activation function, multiplied by three to obtain the action range [-3,3]. The first half of the traffic light agent's Actor network structure is consistent with that of the CAV agent, the difference being that the CAV agent's Actor network is followed by multiple fully connected layers, and the output of the traffic light agent's Actor network is the score of two actions. The score is mapped to the probability distribution of the actions using the Softmax function, and the probability is then converted into a One-Hot vector to determine the final action. Actions with higher probability values ​​are represented as 1 in the vector, and the rest as 0. The Critic networks for both the traffic light agent and the CAV agent consist of one input layer (state vector and action input together), one output layer, and two hidden layers. The target Actor network structure and the target Critic network structure of each agent are identical. Theoretically, after training, the network parameters of each traffic light agent are the same, equivalent to obtaining a set of network parameters for all traffic light agents; similarly, the network parameters of each CAV agent are the same, equivalent to obtaining a set of network parameters for all CAV agents. In subsequent practical applications for control, each traffic light agent and each CAV agent is loaded with the same set of network parameters.

[0150] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the step of extracting from the experience pool D... L Y pieces of empirical data are extracted from the pool to update the parameters of the Critic and Actor networks of the traffic light agent. Then, the parameters of the target Critic and Actor networks of the traffic light agent are updated based on the parameters of the Critic and Actor networks. From the experience pool D... Veh Y empirical data points are extracted to update the parameters of the Critic and Actor networks of the CAV agent. Then, the parameters of the target Critic and Actor networks of the CAV agent are updated based on the parameters of the Critic and Actor networks. The specific process is as follows:

[0151] Step (1): For the Critic network of the traffic light agent:

[0152]

[0153] In the formula: Let represent the joint action value function output by the target Critic network for the i-th traffic light agent. Let r represent the policy of the target Actor network for the i-th traffic light agent. i (t) represents the current reward of the i-th traffic light agent, γ represents the discount factor, γ∈[0,1], L(θ) i Let represent the loss function of the i-th traffic light agent, and E represent the expectation.

[0154] Then minimize the loss function L(θ) i The parameters of the Critic network for the i-th traffic light agent are updated in reverse, and then the parameters of the Actor network for the i-th traffic light agent are updated using the policy gradient method, so that the action selected by the Actor network can obtain the highest Q value estimated by the Critic network. The policy gradient method is as follows:

[0155]

[0156] Where, θ i π represents the Actor network parameters of the i-th traffic light agent. i D represents the policy of the Actor network for the i-th traffic light agent. i The action space of the Actor network representing the traffic light agent;

[0157] Step (2), for the Critic network of the CAV agent:

[0158]

[0159] In the formula: This represents the joint action value function output by the target Critic network for the k-th CAV agent. Let r' represent the policy of the target Actor network for the k-th CAV agent. k (t) represents the current reward of the k-th CAV agent, L(θ') k Let ) denote the loss function of the k-th CAV agent, and E denote the expectation;

[0160] Then minimize the loss function L(θ') k The parameters of the Critic network for the k-th CAV agent are updated in reverse, and then the parameters of the Actor network for the k-th CAV agent are updated using the policy gradient method, so that the action selected by the Actor network can obtain the highest Q value estimated by the Critic network. The policy gradient method is as follows:

[0161]

[0162] Where, θ k This represents the Actor network parameters of the k-th CAV agent. D represents the calculation of the gradient. k This represents the action space of the Actor network for the k-th CAV agent;

[0163] Step (3): The target Actor network parameters of the traffic light agent are soft-updated based on the Actor network parameters, and the target Critic network parameters of the traffic light agent are soft-updated based on the Critic network parameters.

[0164] The target Actor network parameters of the CAV agent are soft-updated based on the Actor network parameters, and the target Critic network parameters of the CAV agent are soft-updated based on the Critic network parameters.

[0165] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0166] Taking the target Actor network of the i-th traffic light agent as an example, the soft update method for the target Actor network parameters and the target Critic network parameters is as follows:

[0167] θ' i ←δθ i +(1-δ)θ' i

[0168] θ' i,π ←δθ i,π +(1-δ)θ' i,π

[0169] Where, θ' i Let θ represent the parameters of the target Actor network for the i-th traffic light agent. i,π Let θ' represent the Critic network parameters of the i-th traffic light agent. i,π Let represent the target Critic network parameters of the i-th traffic light agent.

[0170] Specific Implementation Method Eight: This implementation method differs from one of Specific Implementation Methods One to Seven in that the probability of extracting experience data samples from the experience pool is:

[0171]

[0172] Where TD(n) represents the TD error of the nth empirical data sample in the experience pool, |·| represents the calculation of the absolute value, P(n) represents the probability of the nth empirical data sample being drawn from the experience pool, N represents the total number of samples in the experience pool, and ζ is a hyperparameter that controls the degree of priority.

[0173] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0174] This invention improves the experience playback by determining the experience priority based on the difference between the target Q value and the current Q value (TD error). The larger the TD error, the greater the impact of backpropagation and the stronger the learning guidance.

[0175] When storing empirical sequences, the method for calculating the sample TD error is as follows:

[0176]

[0177] The larger the absolute value of the TD error of a sample, the higher the probability of the sample being selected. When ζ = 0, it is uniform random sampling. At the same time, to prevent some high-priority samples from being selected too frequently, the absolute value of the TD error is halved after a sample is selected once.

[0178] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that the empirical data sample... The reward R'(t) is:

[0179]

[0180] In the formula: △Γ represents the additional number of vehicles passing through after adjustments based on the actions output by each CAV agent. ω1 and ω2 represent the additional energy consumption of the CAV vehicle after adjusting its trajectory based on the action output by the kth CAV agent, where ω1 and ω2 are weighting coefficients.

[0181]

[0182] in, This represents the estimated arrival time of the k-th CAV. This represents the arrival time after adjustment based on the action output by the k-th CAV agent;

[0183] △Γ(t)=Γ (λ+1) (t)-Γ (λ) (t)

[0184] Among them, Γ (λ) (t) represents the number of vehicles passing through during the green light period before adjustments were made based on the actions output by each CAV agent; Γ (λ+1) This indicates the number of vehicles that pass through during the green light period after adjustments are made based on the actions output by each CAV agent.

[0185] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0186] The CAV (Carrier Ability) agent outputs the vehicle's acceleration sequence through its Actor network—that is, the acceleration of the vehicle at each moment along the distance from its initial position to the stop line. Ultimately, the CAV derives an adjustment scheme that minimizes additional energy consumption and maximizes the number of vehicles passing through during the green light period. After this control phase is completed, the vehicle arrival time is recalculated based on the vehicle's current position, and the next phase's traffic light and vehicle acceleration control scheme is obtained.

[0187] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One to Nine in that the empirical data sample... The reward R(t) is:

[0188]

[0189] Where, r i (y) represents the reward for the i-th traffic light agent. N represents the number of times the i-th traffic flow direction has been allowed to proceed within the current control cycle. N = 0 when the signal schemes output by all traffic light agents satisfy the constraints; otherwise, N is infinite.

[0190] r i (t)=ρ1V i (t)-ρ2w i (t)-ρ3q i (t)

[0191] Where ρ1, ρ2, and ρ3 are all weighting coefficients of the indicators.

[0192] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0193] The reward function measures the impact of an agent's actions on its own gains. The agent perceives the environment, selects and executes actions based on the current environmental conditions and its own strategy. The agent's actions affect the environment, and the environment's response is fed back to the agent as a reward, helping it adjust its strategy. The reward function for a traffic light agent consists of three parts: the number of vehicles in queue, vehicle waiting time, and speed utilization coefficient. The agent needs to adjust its strategy to maximize the reward, thus taking a negative sign for the waiting time and queue size metrics. In traffic light scheme planning, within a control cycle, a certain lane may have high traffic volume. To maximize the overall reward, the agent may delay releasing vehicles in lanes with lower traffic volume; therefore, a coefficient is added. Within a control cycle, repeatedly releasing lanes will not increase the reward further; when a control cycle ends, the coefficient... It will be reset to 0.

[0194] Experimental Section

[0195] The experimental scenario is a signalized intersection set up in a traffic simulation platform. Each approach lane has three lanes: left turn, straight ahead, and right turn. Under uncontrolled conditions, fixed signal timing is used, with a signal cycle of 110 seconds. The green light duration for north-south straight ahead is 30 seconds, and for north-south left turns, it is 20 seconds. The green light duration for east-west straight ahead is 30 seconds, and for east-west left turns, it is 20 seconds. The yellow light duration is 10 seconds. The intersection scenario is as follows: Figure 3 As shown in Table 2, the simulation experiment is divided into two parts: balanced and unbalanced flow. The flow distribution is illustrated in Table 2. The proposed adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning is implemented in Python using the Traci interface of the SUMO simulation platform.

[0196] Table 2 Traffic volume of each lane under balanced and unbalanced conditions

[0197]

[0198] The planning starts at time t. s The arrival time estimation module estimates the arrival times of vehicles within the intersection area. The calculation results are then transmitted to the signal scheme planning module and the vehicle trajectory adjustment module. In the signal planning module, the traffic light agent acquires and calculates traffic conditions in real time (number of waiting vehicles, waiting time, speed utilization coefficient), outputs agent actions through the Actor network, and finally outputs a sequence scheme containing n control periods of duration T (i.e., one control cycle is T in length), which is then sent to the CAV agent. Based on the signal scheme and vehicle arrival times, the CAV outputs a sequence of vehicle accelerations through the Actor network. Ultimately, the CAV derives an adjustment scheme that minimizes additional energy consumption and maximizes the number of vehicles passing through during the green light period. The traffic light and CAV continuously interact and adjust, such as... Figure 4 As shown, when this control cycle ends, the planning starts again, and so on, until the simulation ends and exits.

[0199] Simulations were conducted under both balanced and unbalanced traffic flow conditions, with the northbound through lane as the target lane. The traffic volume in this lane remained unchanged under both conditions. Under balanced traffic flow conditions, experiments were conducted with all vehicles using gasoline engines (uncontrolled), all vehicles using gasoline engines (controlled by the method of this invention), all vehicles using electric vehicles (uncontrolled), and all vehicles using electric vehicles (controlled by the method of this invention). The same procedures were followed under unbalanced traffic flow conditions. The CAV penetration rate was set to 0.5 in the simulation. Energy consumption, number of waiting vehicles, average delay, and average lane speed were recorded during the simulation. Vehicles were released 400m from the intersection stop line, and the results were analyzed after 500 seconds of simulation. The analysis results are shown in Table 3.

[0200] Table 3 Comparison of various indicators between the proposed method and the uncontrolled method.

[0201]

[0202] As shown in Table 3, the method proposed in this invention reduces the number of waiting vehicles, decreases delays, lowers energy consumption, and increases average speed. Specifically, under balanced traffic conditions, both gasoline and electric vehicles show improvement in all indicators when using the method of this invention, although the improvement is relatively small. Under unbalanced conditions, the improvement effect of the method of this invention is more significant. This indicates that under balanced traffic conditions, the optimized signal scheme is not much different from the fixed phase, and the advantage of flexible phase sequence is compressed. However, under unbalanced traffic conditions, the flexible phase sequence signal scheme plays a significant role, further improving the intersection's traffic efficiency. Regardless of the traffic situation, in terms of efficiency indicators, when the CAV is gasoline vehicles, the improvement effect is slightly higher than that of electric vehicles. This is because gasoline vehicles still consume a considerable amount of energy when idling, making them more inclined to guide vehicles to reduce stopping. Electric vehicles, on the other hand, consume zero energy when stopped and have an energy recovery mechanism, resulting in a greater improvement in energy consumption.

[0203] Figures 5a to 5d The diagram shows the trajectory of a vehicle (fuel truck) under the control of the method provided in this invention for a portion of the time, where the gray curve represents the HV and the blue curve represents the CAV. As can be seen from the positions marked by the blue lines, the vehicle traveling at a constant speed will stop at the stop line due to the red light. The CAV adjusts its trajectory, appropriately reduces its speed, and successfully passes through during the green light period, guiding the HV behind it and improving the traffic efficiency of the intersection.

[0204] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning, characterized in that, The method specifically includes the following steps: Step 1: Estimate the arrival time of vehicles at the stop line for each direction of traffic flow that needs to be controlled at the target intersection. The specific process of step one is as follows: Regarding traffic flow direction Vehicles that have not reached the stop line Step 11: Divide the vehicles into two parts: the leading vehicle and the following vehicle. The arrival time of the leading vehicle at the stop line is estimated using Step 12, and the arrival time of the following vehicle at the stop line is estimated using Step 13. Steps 1 and 2: When the leading vehicle is manually driven and its ecological speed is higher than the maximum speed limit on the road, the arrival time of the leading vehicle is... for: in, Indicates the distance between the leading car and the stop line. , Indicates the current position of the leading vehicle. Indicates the location of the parking line. Indicates the current speed of the leading vehicle. Indicates the maximum speed limit for vehicles on the road. Indicates the maximum allowable acceleration of the leading vehicle; When the leading vehicle is manually driven and its ecological speed is lower than or equal to the maximum speed limit on the road, the arrival time of the leading vehicle is... for: in, Indicates the eco-speed of the leading vehicle; When the leading vehicle is a CAV, the arrival time of the leading vehicle The estimation method is as follows: Step 1: The leading vehicle obtains its own current position and current speed as well as the current position and speed of the vehicle in front. This then forms the state information observed by the leading vehicle at the current moment. for: In the formula, Indicates the distance from the vehicle in front to the stop line. Indicates the distance from the leading car to the stop line. Indicates the current speed of the vehicle in front. This indicates the current speed of the leading vehicle. This indicates the current acceleration of the leading vehicle. Indicates the direction of traffic flow where the leading vehicle is located. Current signal status; If the direction of traffic flow If the current light is green, then The value is set to 1. Indicates traffic flow direction The remaining time of the current green light signal; If the direction of traffic flow The current light is red. The value is 0. Indicates the distance from the direction of traffic flow at the current moment. The duration for the next green light to open; Step 2: Record the status information observed by the leading vehicle at the current moment. The input feature extractor is used as the input to the agent corresponding to the leading vehicle, and the agent outputs the action selected by the leading vehicle. , This indicates the acceleration of the leading car after adjustment at the current moment. ; Step 3: After the leading vehicle performs the action selected in Step 2, it reaches a new state and then uses the new state to return to and execute Step 2 until the time for the leading vehicle to reach the parking line is planned. In step 3, the time it takes for the leading vehicle to reach the parking line is: Calculate the time it takes for the leading car to reach the stop line based on the planned actions. : in, Indicates instantaneous energy consumption. Indicates the current time; (1) When the leading CAV is an electric vehicle, the formula for calculating instantaneous energy consumption is: in, This indicates the efficiency of the vehicle's powertrain. Indicates vehicle mass. Represents gravitational acceleration. Indicates rolling resistance force, Indicates the speed of the vehicle. This indicates the vehicle's frontal area. Indicates air resistance, Indicates the rotational mass coefficient; (2) When the leading CAV is a gasoline vehicle, the formula for calculating instantaneous energy consumption is: in, , , , , and These are the coefficients of the instantaneous energy consumption model. It is the vehicle's torque. Road slope Step 13: The arrival time of the following vehicle is: in, Indicates the safe time interval between the following vehicle and the vehicle in front. This indicates the estimated time it will take for the vehicle in front to reach the stop line. Indicates the time it takes for the following vehicle to reach the stop line; Step 2: Based on the arrival time of vehicles at the stop line in each traffic flow direction, obtain local observation information for each traffic flow direction, and use the local observation information for each traffic flow direction as the state information for each traffic flow direction. Then, pass the state information for each traffic flow direction through an LSTM network, and output the temporal features of the state information for each traffic flow direction through the LSTM network. The temporal characteristics of the state information of each traffic flow direction are input into the corresponding traffic light agent, and the action set is generated by each traffic light agent. Step 3: Determine a signal control cycle based on the action set generated by each traffic light agent. Each agent corresponding to each CAV in each traffic flow direction plans the CAV motion trajectory according to the phase allocation within the signal control cycle. Step 4: The CAVs in each traffic flow direction drive according to the motion trajectory planning results. After the current signal control cycle ends, the arrival time of vehicles in each traffic flow direction to the stop line is estimated again based on the motion trajectory planning results. Based on the estimated arrival times of vehicles at the stop line in each traffic flow direction, return to step two.

2. The adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning according to claim 1, characterized in that, The target intersection is an intersection with four entrances. The traffic flow directions that need to be controlled at the intersection include the east entrance for straight traffic, the east entrance for left turns, the west entrance for straight traffic, the west entrance for left turns, the south entrance for straight traffic, the south entrance for left turns, the north entrance for straight traffic, and the north entrance for left turns.

3. The adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning according to claim 2, characterized in that, The specific process of step two is as follows: Step Two One: The first person to queue in front of the parking line The number of vehicles in each traffic flow direction is denoted as The first person in line in front of the parking line Average waiting time for vehicles in each traffic direction for: in, This indicates the number of people queuing in front of the stop line. The first traffic flow direction The waiting time for the vehicle This indicates the number of people queuing in front of the stop line. The total number of vehicles in each traffic flow direction; Specifically: in, This indicates the number of people queuing in front of the stop line. The first traffic flow direction The time it takes for a vehicle to come to a complete stop before reaching the stop line. This indicates the number of people queuing in front of the stop line. The first traffic flow direction The time it takes for a car to begin moving due to a green light; in, For the first The time when the next green light begins in each direction of traffic flow; Step 22, Calculate the first Smoothness of vehicle movement in each traffic direction : In the formula, express At the moment The average speed of all vehicles in each traffic flow direction express At the moment The first traffic flow direction speed, express At the moment The total number of vehicles in each traffic flow direction; Steps two and three, the first Local observation information for each traffic flow direction for: Step Two Four, the first Local observation information for each traffic flow direction The format-converted information is passed through an LSTM network, and the time-series features output by the LSTM network are input into the first... The intelligent traffic light agent corresponding to the traffic flow direction, through the first... The traffic light agent corresponding to each traffic flow direction outputs an action vector containing F elements; The constraints that the action vector output by the traffic light agent must satisfy are: in, To plan the start time, Indicates the first The duration of each control cycle Indicates the first The traffic flow direction is at the first The duration of the green light in each control cycle. Indicates the first The traffic flow direction is at the first The start time of the green light for each control cycle; The duration of the green light is subject to constraints: in, This represents the shortest duration of a green light for a given phase. This indicates the longest green light duration for a given phase.

4. The adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning according to claim 3, characterized in that, The traffic light agent and the CAV corresponding agent are jointly trained. The specific training method is as follows: Step 1: Initialize the Actor network parameters and Critic network parameters of the agent. For the traffic light agent, the Actor network is used to generate signal control actions based on the current traffic state; for the agent corresponding to the CAV, the Actor network is used to output the acceleration of the CAV based on the current signal state; and the Critic network of each agent is used to evaluate the expected reward under a given state and action. Step 2: Initialize the training batch. The training step length for each batch is The pre-training step size is Experience replay pool , The training sample size is The signal planning duration is That is, the duration corresponding to the action output by each agent is ; And estimate the time it takes for vehicles in each traffic flow direction to reach the stop line; Step 3: Initialize training batches ; Step 4: Initialize training step size ; Step 5: Initialize the number of pre-training steps ; Step 6, the The traffic light agent for each traffic flow direction obtains initial observation information. The first phase is constructed using initial observation information from all traffic flow directions. The global state information of the traffic light agent for each traffic direction in the current step. , ; global state information The input format is converted to that of an LSTM network, and then the converted state is used as the input to the LSTM network to obtain the output features of the LSTM network. ; Features As the first The input to the Actor network of the traffic light agent in each traffic flow direction is obtained through the first... The Actor network outputs actions for the traffic light agents in each traffic flow direction. ; Step 7: Transfer global status information The output of LSTM and the actions of each agent As the first The input to the Critic network of the traffic flow direction signal agent is obtained through the first... The Critic network outputs a joint action value function for the traffic light agents in each traffic flow direction. ; Step 8, the The first traffic flow direction Each CAV obtains initial observation information. ,Will As input to the feature extractor, and then as the output of the feature extractor, the second feature extractor is used as the first feature extractor. The first traffic flow direction Each CAV corresponds to the input of the agent's Actor network, and the Actor network outputs actions. And the actions output by the Actor network for each CAV corresponding to the intelligent agent in each traffic flow direction are recorded as follows: ,in, This represents the total number of CAVs in each traffic flow direction; Global state information is constructed using initial observation information of all CAVs in all traffic flow directions. Then transfer the global state information The output of the feature extractor and Enter the first The first traffic flow direction Each CAV corresponds to a Critic network for the agent, and the Critic network outputs a joint action value function. ; Step 9: The agent continues to interact with the environment to obtain the local observation data of the traffic light agent at the next moment. and local observation data of the CAV corresponding agent Each traffic light agent and each agent corresponding to a CAV calculates a reward; Global state data is constructed by using the local observation data of each traffic light agent at the next moment. global state data As input to the LSTM model, the output of the LSTM model is then used as the first... The input to the target Actor network of the traffic light agent in each traffic flow direction, the first The target Critic network of the traffic light agents in each traffic flow direction is based on the output of the target Actor network. and global state data Calculate the target Q value; Global state data is constructed using local observation data from each CAV corresponding agent. global state data The feature extractor's output is then used as the input to the target Actor network for each CAV agent. The target Critic network for each CAV agent outputs the target Actor network's output based on the CAV agent's target Actor network. With global state data Output the target Q value; Step 10: Sample the empirical data , to take empirical data samples Store in experience pool In the process, the states of each traffic light agent are transferred to The state transition of the agents corresponding to each CAV to ; , ; Step 11: Determine if the pre-training step size has been reached. ; If the condition is met, proceed to step 12. If it is not achieved, then let Return to step 6; Step 12, from the experience pool Extraction The parameters of the Critic and Actor networks of the traffic light agent are updated using empirical data, and then the parameters of the target Critic and Actor networks of the traffic light agent are updated based on the parameters of the Critic and Actor networks; from the experience pool... Extraction The parameters of the CAV agent's Critic and Actor networks are updated using empirical data, and then the parameters of the CAV agent's target Critic and Actor networks are updated based on the parameters of the Critic and Actor networks. Determine if the training step length has been reached If the condition is met, proceed to step 13; otherwise, set... Return to step 5; Step 13: Determine if the training batch has been reached. ; If this is achieved, then the trained traffic light agent and CAV agent will be obtained; If not achieved, then Return to step 4.

5. The adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning according to claim 4, characterized in that, The experience pool Extraction The parameters of the Critic and Actor networks of the traffic light agent are updated using empirical data, and then the parameters of the target Critic and Actor networks of the traffic light agent are updated based on the parameters of the Critic and Actor networks; from the experience pool... Extraction The parameters of the CAV agent's Critic and Actor networks are updated using empirical data, and then the parameters of the CAV agent's target Critic and Actor networks are updated based on the parameters of the Critic and Actor networks; the specific process is as follows: Step (1): For the Critic network of the traffic light agent: In the formula: Indicates the first The joint action value function output by the target Critic network of each traffic light agent. Indicates the first The strategy of the target Actor network for each traffic light agent. Indicates the first The current reward for each traffic light agent. Indicates the discount factor. , Indicates the first The loss function of a traffic light agent This indicates a demand for expectation; Then minimize the loss function To update the reverse version The Critic network parameters of the first traffic light agent are then updated using the policy gradient method. The Actor network parameters for each traffic light agent, and the policy gradient method are as follows: in, Indicates the first Actor network parameters for each traffic light agent. Indicates the first The strategy of the Actor network for each traffic light agent. The action space of the Actor network representing the traffic light agent; Step (2), for the Critic network of the CAV agent: In the formula: Indicates the first The joint action value function output by the target Critic network of each CAV agent. Indicates the first The strategy of the target Actor network for each CAV agent. Indicates the first The current reward for each CAV agent. Indicates the first The loss function of a CAV agent This indicates a demand for expectation; Then minimize the loss function To update the reverse version The Critic network parameters of the first CAV agent are then updated using the policy gradient method. The Actor network parameters for each CAV agent, and the policy gradient method are as follows: = in, Indicates the first Actor network parameters for each CAV agent, This indicates the calculation of the gradient. Indicates the first The action space of an Actor network of CAV agents; Step (3): The target Actor network parameters of the traffic light agent are soft-updated based on the Actor network parameters, and the target Critic network parameters of the traffic light agent are soft-updated based on the Critic network parameters. The target Actor network parameters of the CAV agent are soft-updated based on the Actor network parameters, and the target Critic network parameters of the CAV agent are soft-updated based on the Critic network parameters.

6. The adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning according to claim 5, characterized in that, The of The probability of a sample being selected is: in, express The indivual TD error of the sample This indicates the calculation of absolute value. express The indivual The probability of a sample being selected. express The total number of samples, It's a hyperparameter.

7. The adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning according to claim 6, characterized in that, The empirical data sample Rewards for: In the formula: This represents the additional number of vehicles that can pass through after adjustments are made based on the actions output by each CAV agent. Indicates CAV vehicle according to the first The additional energy consumption after adjusting the trajectory of the actions output by each CAV agent. and These are the weighting coefficients; in, The estimated first Arrival time of each CAV Indicates according to the first The arrival time after adjusting the actions output by each CAV agent; in, This indicates the number of vehicles that passed through during the green light period before adjustments were made based on the actions output by each CAV agent. This indicates the number of vehicles that pass through during the green light period after adjustments are made based on the actions output by each CAV agent.

8. The adaptive light-vehicle cooperative rolling control method based on deep reinforcement learning according to claim 7, characterized in that, The empirical data sample Rewards for: in, Indicates the first Rewards for each traffic light agent. Indicates the first The number of times each traffic flow direction has been allowed to proceed within the current control cycle, and the condition that the signal schemes output by all traffic light agents meet the constraints. ;otherwise, It is infinitely large; in, These are all weighting coefficients for the indicators.

Images