A method for planning acceleration trajectory of interconnected vehicles at signal light intersections

Abstract

The present invention relates to a method for planning acceleration trajectory of an interconnected car at a signal light intersection based on deep reinforcement learning, comprising the following steps: constructing a vehicle dynamics model and a signal light control model; when the interconnected car enters the control area, judging whether the current vehicle can pass through the intersection in the nearest green light interval and under the road speed limit according to vehicle information and road information, and if so, executing the next step; constructing a deep reinforcement learning framework, defining a Markov process and setting a neural network; training the neural network according to the vehicle dynamics model, the signal light control model and the deep reinforcement learning algorithm until the preset training termination condition is reached, completing the training and obtaining an acceleration control strategy; sending the acceleration control strategy to the interconnected car, and the interconnected car executes the acceleration control strategy and passes through the intersection according to the set target. Compared with the prior art, the present invention has the advantages of good safety, short travel time, and wide application range.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for planning acceleration trajectories of interconnected vehicles at signal light intersections based on deep reinforcement learning. Background Art

[0002] In recent years, intelligent control has been introduced in ground transportation, which provides new ideas for mitigating the impact of automobiles on traffic, energy and the environment. Among them, connected vehicle (Connected Vehicle) technology is increasingly being considered for research and development, and has been absorbed by real-world transportation systems due to the potential benefits of traffic throughput and energy consumption management. In connected vehicles (CVs), real-time information about the environment, including other road users and traffic signals, is provided to vehicle operators through vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications. Therefore, there is a great opportunity to effectively utilize shared or perceived information and reduce human errors.

[0003] Mahler G (Mahler G, Vahidi A. An optimal velocity-planning scheme for vehicle energy efficiency through probabilistic prediction of traffic-signal timing [J]. IEEE Transactions on Intelligent Transportation Systems, 2014) developed a predictive optimal velocity planning algorithm that introduced a signal phase prediction model that uses historical average time data and real-time phase data to determine the probability of the upcoming traffic light being green. In an optimal control framework, the optimal velocity trajectory is calculated to maximize the chance of passing the green light. Asadi B (Asadi B, Vahidi A. Predictive cruise control: Utilizing upcoming traffic signal information for improving fuel economy and reducing trip time [J]. IEEE Transactions on Control Systems Technology, 2010) not only uses traffic signal information for prediction, but also uses a short-range radar for identification to obtain the optimal velocity trajectory of the vehicle, which greatly improves the probability of passing the green light interval. In addition, Sun C (Sun C, Guanetti J, Borrelli F, et al. Robust eco-driving control of autonomous vehicles connected to traffic lights [J]. 2018) applied the idea of ​​robustness to the optimal driving problem and proposed a robust optimal driving control strategy for multi-signal intersections with uncertain traffic signal time. At the same time, taking driving time as a state variable, a vehicle speed optimization formula aimed at reducing fuel consumption was proposed. The concept of "Effective Red Light Duration" was introduced and represented as a random variable to describe the feasible transfer time through the signal intersection. A chance constraint was attached to the optimal control problem to incorporate the robustness to uncertain signal time, and the optimal control problem was finally solved by dynamic programming.

[0004] The traditional method for solving the optimal control problem is to determine the necessary conditions for optimal control based on variational calculus and Pontryagin's minimum principle. Then, the iterative numerical method is used to solve the two-point boundary value problem to obtain the optimal control. Generally speaking, due to the nature of traffic light constraints, this optimization problem is non-convex. At the same time, the real-world vehicle system is nonlinear and highly complex. Too many restrictions and the expansion of the problem scale will cause the dimensionality curse problem, which will lead to problems such as low solution accuracy and difficulty in solving the problem. Summary of the invention

[0005] The purpose of the present invention is to provide a method for planning the acceleration trajectory of connected vehicles at signal light intersections based on deep reinforcement learning, aiming at the problem of how connected vehicles can control acceleration to achieve safe driving under specific goals in road traffic at signal light intersections. The method avoids the dimensionality disaster problem caused by too many restrictions and the expansion of the problem scale. After acquiring data from the cloud and performing offline training, an acceleration control strategy is obtained and deployed in real time on individual vehicles, thereby achieving control of the vehicle braking process under specific goals (such as shortest time passing, minimum energy consumption passing, and mixed goals).

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for planning acceleration trajectories of interconnected vehicles at signal light intersections based on deep reinforcement learning, comprising the following steps:

[0008] Step 1) constructing a vehicle dynamics model and a signal light control model;

[0009] Step 2) When the connected car enters the control area, the cloud determines whether the current vehicle can pass through the intersection within the nearest green light interval and at the road speed limit based on the vehicle information fed back by the connected car and the road information fed back by the intelligent transportation system. If so, execute step 3);

[0010] Step 3) Build a deep reinforcement learning framework in the cloud, define the Markov process and set up the neural network;

[0011] Step 4) The cloud trains the neural network according to the vehicle dynamics model, the signal light control model and the deep reinforcement learning algorithm until the preset training termination condition is reached, the training is completed, and the acceleration control strategy is obtained;

[0012] Step 5) The cloud sends the acceleration control strategy to the connected car, and the connected car executes the acceleration control strategy and passes the intersection according to the established target.

[0013] The vehicle dynamics model is:

[0014]

[0015] Where s(t) is the distance between the vehicle’s current position and the starting point of the control area, v(t) is the vehicle’s current speed, a(t) is the vehicle’s current acceleration, and the superscript · indicates the derivative.

[0016] The traffic light control model processes the yellow light as a red light, the traffic light switching complies with the "red, green, red" rule, and the red and green light times have fixed durations.

[0017] In step 2), when the connected car enters the starting point of the set control area, the current time is recorded and the timing starts. The connected car communicates with the cloud and sends the current vehicle information to the cloud. At the same time, the intelligent transportation system uploads the current road information to the cloud.

[0018] The vehicle information includes the current speed, maximum acceleration, and minimum acceleration of the connected vehicle;

[0019] The road information includes the current traffic light status, the current traffic light duration, the cycles of different color lights of the traffic lights at this intersection, the current maximum speed of the road, the current minimum speed of the road, and the length of the control area of ​​this intersection.

[0020] In step 2), the cloud determines whether the current vehicle can pass through the intersection within the nearest green light interval and at the road speed limit based on the vehicle information fed back by the connected car and the road information fed back by the intelligent transportation system, including two situations:

[0021] If the current traffic light is green, determine whether the vehicle can pass the intersection within the current green light interval and meet the road speed limit conditions based on the vehicle information and road information;

[0022] If the current traffic light is red, determine whether the vehicle can pass the intersection within the green light interval after the current red light and meet the road speed limit conditions based on the vehicle information and road information;

[0023] If yes, proceed to the next step.

[0024] The Markov process is specifically defined as:

[0025] Defining states is {the position of the vehicle from the starting point of the control area, the current speed of the vehicle, the time from entering the control area, the current state of the traffic light, the duration of the current traffic light} = {s(t), v(t), t, ts(t), tp(t)};

[0026] Defining actions is the current acceleration of the vehicle = a(t);

[0027] For the objective function Defining Rewards is r=r1+r2+r3, where

[0028]

[0029]

[0030]

[0031] Among them, ω1, ω2 are weight parameters, t f is the time when the vehicle passes through the intersection, t0 is the time when the vehicle arrives at the starting point of the control area, E fuel is the energy consumption, For E fuel The derivative of represents the energy consumption rate, T s is the sampling time interval, v min 、v max are the minimum and maximum speeds of the current road respectively, v(t) is the current speed of the connected car, s(t) is the distance between the current position of the vehicle and the starting point of the control area, L is the length of the control area of ​​this intersection, ts is the current traffic light status, ts=-1 represents a red light, ts=1 represents a green light, and p is the penalty term.

[0032] Different objectives are determined according to the value of the weight parameter in the objective function. If ω1=0, it is the minimum energy objective, and if ω2=0, it is the shortest time objective.

[0033] In the deep reinforcement learning framework, variables are normalized when state values ​​are input into the neural network.

[0034] The deep reinforcement learning algorithm uses a deep deterministic policy gradient reinforcement learning method to construct an evaluation network and action network Among them, θ Q and θ π are the parameters of the neural network, and introduce the target networks corresponding to the evaluation network and the action network respectively and Among them, θ Q′ With θ Q With the same initial value, θ π′ and θ π have the same initial value, For status, For actions, the action selection method is: is a random process.

[0035] In step 4), the training process is specifically as follows:

[0036] Evaluation network parameters θ Q The update method is:

[0037] Define the loss function:

[0038]

[0039] Where N is the minimum batch, γ is the discount rate, and r is the reward; the parameter θ of the evaluation network is updated by minimizing the loss function L Q ;

[0040] Action network parameters θ π The update method is:

[0041]

[0042] The target network is updated as follows:

[0043] θ Q′ =τθ Q +(1-τ)θ Q′

[0044] θ π′ =τθ π +(1-τ)θ π′

[0045] Where τ is the target smoothing factor, and 0≤τ≤1.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. Aiming at road traffic at intersections with traffic lights, the present invention obtains an acceleration control strategy by using a deep reinforcement learning method, thereby achieving the problem of safe driving of connected cars under specific goals. By shortening the travel time, the passing rate of road traffic is improved, and energy consumption and exhaust pollutant emissions are reduced;

[0048] 2. The deep reinforcement learning method adopted by the present invention does not depend on a specific model and is applicable to different vehicles and roads. At the same time, this method is applicable to problems with many constraints and large scales and is not limited by the dimension of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the method of the present invention;

[0050] Figure 2 A road traffic schematic diagram of the present invention;

[0051] Figure 3 A schematic diagram of the changes of the traffic light of the present invention;

[0052] Figure 4 It is the DDPG algorithm framework of the present invention;

[0053] Figure 5It is a structural schematic diagram of the evaluation network of the present invention;

[0054] Figure 6 It is a structural schematic diagram of the action network of the present invention;

[0055] Figure 7 An acceleration trajectory diagram obtained through training in an embodiment;

[0056] Figure 8 is a velocity-time diagram corresponding to an acceleration trajectory in one embodiment;

[0057] Fig. 9 1 is a displacement-time diagram corresponding to the acceleration trajectory in one embodiment. DETAILED DESCRIPTION

[0058] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0059] like Figure 1 As shown, this embodiment is directed to Figure 2 The road traffic scenario shown in the figure provides a method for planning the acceleration trajectory of a connected car at a traffic light intersection based on deep reinforcement learning, including the following steps:

[0060] Step 1) Construct vehicle dynamics model and signal light control model.

[0061] This embodiment adopts a simplified vehicle dynamics model:

[0062]

[0063] Where s(t) is the distance between the vehicle’s current position and the starting point of the control area, v(t) is the vehicle’s current speed, a(t) is the vehicle’s current acceleration, and the superscript · indicates the derivative.

[0064] At the same time, a simplified traffic light control model is introduced, in which the yellow light is treated as a red light. The traffic light switching follows the "red, green, and red" rule, and the red and green light times have fixed durations, denoted as T and T respectively. R and T G In this embodiment, the traffic light change diagram is as follows: Figure 3 As shown, the red light and the green light are alternately switched, and both have a fixed duration.

[0065] Step 2) When the connected car enters the control area, the cloud determines whether the current vehicle can pass through the intersection within the nearest green light interval and at the road speed limit based on the vehicle information fed back by the connected car and the road information fed back by the intelligent transportation system. If so, execute step 3).

[0066] The judgment conditions include the following two situations:

[0067] If the current traffic light is green, determine whether the vehicle can pass the intersection within the current green light interval and meet the road speed limit conditions based on the vehicle information and road information;

[0068] If the current traffic light is red, it is determined based on the vehicle information and road information whether the vehicle can pass through the intersection within the green light interval after the current red light and meet the road speed limit conditions.

[0069] In this embodiment, when the connected car enters the starting point of the set control area, the current time t0 is recorded and the timing starts. The connected car communicates with the cloud and sends the current vehicle information to the cloud. At the same time, the intelligent transportation system uploads the current road information to the cloud.

[0070] Vehicle information includes the connected car’s current speed v0, maximum acceleration a max , minimum acceleration a min ;

[0071] Road information includes the current traffic light state ts, the current traffic light duration tp, and the cycle of different color lights at this intersection T R and T G 、The current maximum road speed v max 、The current minimum road speed v min , the length L of the control area of ​​this intersection.

[0072] Step 3) Build a deep reinforcement learning framework in the cloud, define the Markov process and set up the neural network;

[0073] Deep reinforcement learning algorithm framework such as Figure 4 As shown in the figure, the intelligent agent and the simulation environment interact continuously. The intelligent agent uses the deep reinforcement learning algorithm to update its strategy based on the observation values ​​given by the environment, executes actions, and feeds back to the simulation environment. At the same time, the simulation environment feeds back rewards to the intelligent agent based on the actions of the intelligent agent and updates the observation values.

[0074] Step 3) comprises the following steps:

[0075] Step 3-1) Define the Markov process

[0076] ① Status

[0077] Defining states {the position of the vehicle from the starting point of the control area, the current speed of the vehicle, the time from entering the control area, the state of the current traffic light, the duration of the current traffic light} = {s(t), v(t), t, ts(t), tp(t)}.

[0078] In a preferred embodiment, the variables are normalized when the state values ​​are input into the neural network.

[0079] ②Action

[0080] Defining actions is the current acceleration of the vehicle = a(t).

[0081] ③Rewards

[0082] Construct the objective function:

[0083]

[0084] Among them, ω1 and ω2 are weight parameters. If ω1 = 0, it is the minimum energy target. If ω2 = 0, it is the shortest time target. If both are not 0, it is a mixed target. f is the time when the vehicle passes through the intersection, t0 is the time when the vehicle arrives at the starting point of the control area, E fuel is the energy consumption, For E fuel The derivative of , represents the energy consumption rate.

[0085] Defining Rewards is r=r1+r2+r3, where

[0086]

[0087]

[0088]

[0089] Among them, T s is the sampling time interval, v min 、v max are the minimum and maximum speeds of the current road respectively, v(t) is the current speed of the connected car, s(t) is the distance between the current position of the vehicle and the starting point of the control area, L is the length of the control area of ​​this intersection, ts is the current traffic light status, ts=-1 represents a red light, ts=1 represents a green light, and p is the penalty term.

[0090] Here, the reward item r2 is a reflection of the speed limit condition. If the current speed exceeds the speed limit, a penalty value -p is imposed. Similarly, the reward item r3 is a reflection of the traffic light restriction condition. If the current traffic light is red when the vehicle arrives at the intersection, a penalty value -p is imposed.

[0091] Step 3-2) Build a neural network

[0092] The states and actions of this type of problem are both continuous variables, so this embodiment selects the Deep Deterministic Policy Gradient reinforcement learning method.

[0093] Introducing the Evaluation Network and action network Here θ Q and θ π are the parameters of the neural network, and then the target networks corresponding to the evaluation network and the action network are introduced respectively. and where θ Q′ With θ Q With the same initial value, θ π′ and θ π Also has the same initial value. Here, the action is selected as follows: is a random process. In this embodiment, the network structure of the evaluation network and the network structure of the action network are respectively as follows: Figure 5 and Figure 6 As shown, the evaluation network takes state and action as input, and the action network takes state as input.

[0094] In another embodiment, other deep reinforcement learning methods can also solve such problems, such as the Policy Gradient method, the Twin-Delayed Deep Deterministic Policy Gradient method, etc. In addition, if the continuous action is discretized, the Deep Q-Network method is also applicable to this method, and these solutions should also be considered to be based on the principle of this invention. At the same time, the definition of the Markov process proposed in this solution can also be improved, and these improvements should also be considered as the scope of protection of the present invention.

[0095] Step 4) The cloud trains the neural network according to the vehicle dynamics model, the traffic light control model and the deep reinforcement learning algorithm until the preset training termination condition is reached, the training is completed, and the acceleration control strategy is obtained.

[0096] During the training process, the parameters θ of the evaluation network are Q The update method is:

[0097] Define the loss function:

[0098]

[0099] Where N is the minimum batch, γ is the discount rate, and r is the reward; the parameter θ of the evaluation network is updated by minimizing the loss function LQ .

[0100] Action network parameters θ π The update method is:

[0101]

[0102] The target network is updated as follows:

[0103] θ Q′ =τθ Q +(1-τ)θ Q′

[0104] θ π′ =τθ π +(1-τ)θ π′

[0105] Where τ is the target smoothing factor, and 0≤τ≤1.

[0106] Starting from the time the connected car enters the control area, the movement of the vehicle on the road is simulated according to the current vehicle dynamics model, traffic light changes and deep reinforcement learning methods to obtain the reward value; then, according to the pre-defined Markov process and deep reinforcement learning framework, the neural network is trained and updated using the above update formula until the set training termination condition is reached, and the acceleration control strategy is obtained.

[0107] Step 5) The cloud sends the acceleration control strategy to the connected car, and the connected car executes the acceleration control strategy and passes the intersection according to the established target.

[0108] The focus of this invention is to solve the optimal driving problem of a connected car approaching a signalized intersection using a reinforcement learning (RL) framework. When the vehicle enters a certain range of a traffic signal, using V2X communication, it receives the phase and timing (SPaT) information of the traffic signal. Then, as the vehicle approaches the signalized intersection, it attempts to minimize a specific objective. This optimization problem can be formulated by minimizing the engine idling time of the connected vehicle as it approaches the intersection, subject to satisfying constraints. The problem can also be formulated as an optimal control problem, dealing with finding a control strategy to minimize a specified objective function.

[0109] The present invention mainly has the following innovative features:

[0110] 1. Taking advantage of the fact that reinforcement learning methods do not rely on models, we creatively apply reinforcement learning methods to the problem of acceleration trajectory optimization;

[0111] 2. After the introduction of neural networks, the use of deep reinforcement learning methods has expanded the scope of application of this solution and reduced the difficulty of solving the problem;

[0112] 3. Introducing the Markov process, the vehicle dynamics model and the traffic light control model are transformed into the Markov process, and the objective function and constraints are transformed into reward values, thus avoiding the difficult-to-solve problems caused by too many constraints;

[0113] 4. The solution designed by the present invention only needs to repeat the training process and learning process for different vehicles and different road conditions, and does not need to modify other contents;

[0114] 5. The solution designed by the present invention can achieve the optimization of specific goals by only modifying the corresponding reward value part for different optimization goals.

[0115] Based on the specific description of the above technical solution, this embodiment performs experimental design and simulation testing based on the Reinforcement Learning Toolbox in Matlab. The specific process is as follows:

[0116] Step 1) construct a vehicle dynamics model and a signal light control model; wherein the fixed durations of the red light and green light in the signal light control model are set to T R =15s and T G =15s;

[0117] Step 2) When the connected car enters the control area, it communicates with the cloud and sends the current vehicle information (current speed v0 = 5m / s, maximum acceleration a max =2m / s 2 , minimum acceleration a min =-2m / s 2 ) is sent to the cloud. At the same time, the intelligent transportation system sends the current road information (current traffic light status ts0 = -1, current traffic light duration tp o = 5s, the cycle of the different color lights of this intersection signal light T R =15s and T G =15s, current maximum road speed v max =10m / s, current road minimum speed v min =0m / s, the length of the control area of ​​this intersection L = 100m) is uploaded to the cloud. In addition, the timing starts from this moment, and t(0) = t0 = 0 is set; thereafter, the cloud determines whether the current vehicle can pass the intersection within [10s, 25s] under the road speed limit based on the above vehicle information and road information. If yes, step 3) is executed.

[0118] During the judgment process, the most extreme case is considered first. The current vehicle can move forward at the maximum acceleration. It takes (10-5) / 2=2.5s to reach the maximum speed limit. The travel distance is 5*2.5+0.5*2*2.5^2=18.75m. Then continue to move forward at the maximum speed limit. The remaining 100-18.75=81.25m takes 81.25 / 10=8.125s, and the total time is 2.5+8.125=10.625s. In this extreme case, the shortest time calculated to pass the intersection and meet the speed limit conditions is within the time interval [10s, 25s]. Therefore, the current vehicle can pass the intersection in the green light interval after the current red light, and execute step 3).

[0119] Step 3) Build a deep reinforcement learning framework in the cloud, define the Markov process and set up the neural network;

[0120] In this embodiment, ω1=1, ω2=0 is set, that is, the shortest time-consuming target; reward is r=r1+r2+r3, where

[0121] r1=T s , where the sampling time interval is T s =0.1s, The penalty term p=30.

[0122] The specific parameter selection of the Deep Deterministic Policy Gradient reinforcement learning method is shown in Table 1.

[0123] Table 1 Algorithm parameter selection

[0124]

[0125] Step 4) The cloud trains the neural network according to the vehicle dynamics model, the traffic light control model and the deep reinforcement learning algorithm until the preset training termination condition is reached, the training is completed, and the acceleration control strategy is obtained.

[0126] In this embodiment, the training process parameters are shown in Table 2.

[0127] Table 2 Training process parameters

[0128] Parameter name Numeric Parameter name Numeric Maximum number of screens 10000 Algorithm termination criteria Total number of scenes Maximum single screen length 500 End value 10000

[0129] Step 5) The cloud sends the acceleration control strategy to the connected car, and the connected car executes the acceleration control strategy and passes the intersection according to the established target.

[0130] The acceleration trajectory obtained in this embodiment is shown in FIG. Figure 7 As shown, the time-speed diagram is Figure 8 As shown, the time-displacement diagram is Fig. 9 shown.

[0131] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for planning acceleration trajectories of interconnected vehicles at traffic light intersections based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1) constructing a vehicle dynamics model and a signal light control model; Step 2) When the connected car enters the control area, the cloud determines whether the current vehicle can pass through the intersection within the nearest green light interval and at the road speed limit based on the vehicle information fed back by the connected car and the road information fed back by the intelligent transportation system. If so, execute step 3); Step 3) Build a deep reinforcement learning framework in the cloud, define the Markov process and set up the neural network; Step 4) The cloud trains the neural network according to the vehicle dynamics model, the signal light control model and the deep reinforcement learning algorithm until the preset training termination condition is reached, the training is completed, and the acceleration control strategy is obtained; Step 5) The cloud sends the acceleration control strategy to the connected car, and the connected car executes the acceleration control strategy and passes the intersection according to the established target; The Markov process is specifically defined as: Defining states is {the position of the vehicle from the starting point of the control area, the current speed of the vehicle, the time from entering the control area, the current state of the traffic light, the duration of the current traffic light} = {s(t), v(t), t, ts(t), tp(t)}; Defining actions is the current acceleration of the vehicle = a(t); For the objective function Defining Rewards is r=r1+r2+r3, where Among them, ω1, ω2 are weight parameters, t f is the time when the vehicle passes through the intersection, t0 is the time when the vehicle arrives at the starting point of the control area, E fuel is the energy consumption, For E fuel The derivative of represents the energy consumption rate, T s is the sampling time interval, v min 、v max are the minimum speed and maximum speed of the current road, v(t) is the current speed of the connected car, s(t) is the distance between the current position of the vehicle and the starting point of the control area, L is the length of the control area of ​​the intersection, ts is the current traffic light status, ts=-1 represents a red light, ts=1 represents a green light, and p is a penalty term; Different objectives are determined according to the value of the weight parameter in the objective function. If ω1=0, it is the minimum energy objective, and if ω2=0, it is the shortest time objective.

2. According to the method of claim 1, the acceleration trajectory planning method of the interconnected car at the signal light intersection based on deep reinforcement learning is characterized in that: The vehicle dynamics model is: Where s(t) is the distance between the vehicle’s current position and the starting point of the control area, v(t) is the vehicle’s current speed, a(t) is the vehicle’s current acceleration, and the superscript · indicates the derivative.

3. According to the method of claim 1, the acceleration trajectory planning method of the interconnected car at the signal light intersection based on deep reinforcement learning is characterized in that: The traffic light control model processes the yellow light as a red light, the traffic light switching follows the "red, green, red" rule, and the red and green light times have fixed durations.

4. According to the method of claim 1, the acceleration trajectory planning method of the interconnected car at the signal light intersection based on deep reinforcement learning is characterized in that: In step 2), when the connected car enters the starting point of the set control area, the current time is recorded and the timing starts. The connected car communicates with the cloud and sends the current vehicle information to the cloud. At the same time, the intelligent transportation system uploads the current road information to the cloud. The vehicle information includes the current speed, maximum acceleration, and minimum acceleration of the connected vehicle; The road information includes the current traffic light status, the current traffic light duration, the cycles of different color lights of the traffic lights at this intersection, the current maximum speed of the road, the current minimum speed of the road, and the length of the control area of ​​this intersection.

5. According to the method of claim 1, the acceleration trajectory planning method of the interconnected car at the signal light intersection based on deep reinforcement learning is characterized in that: In step 2), the cloud determines whether the current vehicle can pass through the intersection within the nearest green light interval and at the road speed limit based on the vehicle information fed back by the connected car and the road information fed back by the intelligent transportation system, including two situations: If the current traffic light is green, determine whether the vehicle can pass the intersection within the current green light interval and meet the road speed limit conditions based on the vehicle information and road information; If the current traffic light is red, determine whether the vehicle can pass the intersection within the green light interval after the current red light and meet the road speed limit conditions based on the vehicle information and road information; If yes, proceed to the next step.

6. The method for planning acceleration trajectory of interconnected vehicles at signal light intersections based on deep reinforcement learning according to claim 1, characterized in that: In the deep reinforcement learning framework, variables are normalized when state values ​​are input into the neural network.

7. The method for planning acceleration trajectory of interconnected vehicles at signal light intersections based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning algorithm uses a deep deterministic policy gradient reinforcement learning method to construct an evaluation network and action network Among them, θ Q and θ π are the parameters of the neural network, and introduce the target networks corresponding to the evaluation network and the action network respectively and Among them, θ Q′ With θ Q With the same initial value, θ π′ and θ π have the same initial value, For status, For actions, the action selection method is: is a random process.

8. The method for planning acceleration trajectories of interconnected vehicles at signal light intersections based on deep reinforcement learning according to claim 7, characterized in that: In step 4), the training process is specifically as follows: Evaluation network parameters θ Q The update method is: Define the loss function: Where N is the minimum batch, γ is the discount rate, and r is the reward; the parameter θ of the evaluation network is updated by minimizing the loss function L Q ; Action network parameters θ π The update method is: The target network is updated as follows: i Q′ =tθ Q +(1-τ)θ Q′ i π′ =tθ π +(1-τ)θ π′ Where τ is the target smoothing factor, and 0≤τ≤1.

Images