A traffic light intersection passing method based on proximal strategy optimization

Abstract

The present invention is applicable to the technical field of automatic driving control of road vehicles, and provides a method for passing through a signal light intersection based on proximal strategy optimization, comprising the following steps: building a deep reinforcement learning network model based on the PPO algorithm, and collecting real-time road information as the state space of the economical signal light intersection passing model; step 2, proposing an economical signal light intersection passing reward function based on limiting factors; step 3, interacting with the environment in real time based on the action variables generated by the economical signal light intersection passing model, evaluating the actions taken by the controlled vehicle in the current state according to the reward function, and optimizing the strategy parameters in the deep reinforcement learning, and finally obtaining an economical signal light intersection passing strategy. This method can not only avoid idling at intersections due to red lights and increase road traffic efficiency, but also the economical driving behavior during the entire driving process can further reduce the vehicle's energy consumption.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic driving control of road vehicles, and in particular relates to a method for passing through a signal light intersection based on proximal strategy optimization. Background Art

[0002] Traffic light intersections are a crucial component of urban traffic flow and play an irreplaceable role in maintaining traffic order. However, with the increasing number of vehicles in cities and the changing traffic planning practices of various cities, vehicles often stop and wait at traffic light intersections. Because the powertrain operates in an inefficient zone when idling, this results in a significant amount of unnecessary energy consumption and waiting time.

[0003] Recent advances in communication technology have made vehicle-to-infrastructure (V2I) communication a reality. Furthermore, with the continued maturity of artificial intelligence algorithms, such as deep reinforcement learning, deep reinforcement learning has proven to possess powerful learning capabilities even in complex environments, without requiring humans to develop complex control models or rules. Optimizing autonomous vehicles using traffic information from traffic lights combined with deep reinforcement learning has become a hot topic in the field of autonomous driving.

[0004] With the advancement of connected vehicle technology over the past few years, the problem of optimizing the economic driving of autonomous vehicles has been a hot topic of research. This involves optimizing the economic driving behavior of a vehicle using traffic information collected during driving. This problem is typically addressed using the concept of optimal control, where the vehicle's energy consumption is considered the optimization objective, and traffic regulations, driving safety, and the vehicle's dynamics model are considered the problem constraints. However, as the complexity of driving tasks increases, the number of state variables and control variables in the optimal driving control problem increases. Therefore, the computational difficulty of solving model-based driving strategies increases with the dimensionality of the optimal problem. For example, while model predictive control and dynamic programming algorithms offer good interpretability, the significant uncertainty inherent in traffic systems complicates the design of rule-based solutions and weakens their optimization capabilities. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a method for passing through a signalized intersection based on proximal strategy optimization, aiming to solve the problems raised in the above background technology.

[0006] The embodiment of the present invention is implemented as follows: a method for passing through a signal light intersection based on proximal strategy optimization, comprising the following steps:

[0007] Step 1: Build a deep reinforcement learning network model based on the proximal policy optimization (PPO) algorithm. Collect real-time vehicle speed information, the distance between the vehicle's current position and the target signal intersection, and the time difference between the current position and the passable green light phase, and other road information as the state space of the economic signal intersection traffic model;

[0008] Step 2: Propose an economical signalized intersection passing reward function based on factors such as vehicle energy consumption, vehicle speed, whether the vehicle can pass the target signalized intersection under the green light timing, and the vehicle speed limit under urban conditions;

[0009] Step 3: The controlled vehicle interacts with the environment in real time in the open-source vehicle simulator Carla based on the action variables generated by the economical traffic light intersection model. The actions taken by the controlled vehicle in the current state are evaluated based on the reward function, and the policy parameters in deep reinforcement learning are optimized to ultimately obtain an economical traffic light intersection strategy.

[0010] Further technical solution, said step 1 comprises the following specific steps:

[0011] Step 1.1: The driving condition is designed so that the initial vehicle generation position of each training round is the same distance from the traffic light intersection. The vehicle can have three different initial speeds at the initial generation position: low, medium and high. ,The signal lights can have different timing and phase sequences, and the construction of the ,above driving conditions is implemented in the Carla / Python ,co-simulation platform;

[0012] Step 1.2: Set the action space in the applied proximal strategy optimization algorithm to the accelerator pedal opening, and set the range of the accelerator pedal opening to [0, 1].

[0013] Step 1.3: In order to make the agent perceive the changes in the environment in real time, the state space of the PPO algorithm is set to: the distance between the current position of the vehicle and the target traffic light intersection, recorded as ; The real-time longitudinal speed of the vehicle is recorded as The time difference between the current moment and the passable green light phase is recorded as ; Then the proximal strategy optimization algorithm has a three-dimensional state space, that is .

[0014] According to a further technical solution, in step 1.2, the vehicle dynamics model driven by the accelerator pedal is provided by a Carla simulator.

[0015] Further technical solution, said step 2 includes the following specific steps:

[0016] Step 2.1: Real-time speed reward for integrated vehicles , the instantaneous energy consumption level reward of the vehicle during driving and rewards for passing within the green light timeline Setting; setting the maximum expected passing speed and minimum expected passing speed The calculation formula is as follows:

[0017]

[0018]

[0019] In the above formula is the distance between the current position of the vehicle and the target traffic light intersection, is the time difference from the current moment to the passable green light phase, The duration of the green phase;

[0020] Step 2.2: Vehicle's real-time speed reward The specific definitions are as follows:

[0021]

[0022] In the above formula Represents the real-time longitudinal speed of the vehicle; Represents the time difference between the initial moment of each training round and the passable green light phase; Represents the driving time of the vehicle in a training round; Represents a truncation function that limits the value of the first argument in the following parentheses to between the second and third arguments; represents the upper speed limit in the driving condition, which is set to 50 km / h in this invention; the speed reward in the reward function The reward value that should be given to the vehicle at different speeds and driving times when driving towards a signalized intersection is fully considered, and considering that signalized intersections mostly occur in urban driving conditions, a certain limit is imposed on the maximum speed of the vehicle. Have the ability to guide the vehicle through the signalized intersection at an appropriate speed under the green light sequence;

[0023] Step 2.3: Vehicle instantaneous energy consumption level reward The definition of is as follows:

[0024]

[0025] In the above formula Represents the difference in acceleration of the controlled vehicle collected in two adjacent time steps; Represents the difference in the action output of the PPO algorithm between two adjacent time steps, that is, the difference in the accelerator pedal opening; Represents the instantaneous energy consumption of the vehicle, which is calculated based on the VT-Micro fuel consumption model using the vehicle's speed and acceleration. The calculation formula of the VT-Micro fuel consumption model is as follows:

[0026]

[0027] in, It is a parameter related to vehicle acceleration and is related to factors such as the positive and negative acceleration of the vehicle during driving. and are the acceleration and velocity values of the vehicle at any moment;

[0028] Vehicle driving energy consumption level reward Comprehensively consider the vehicle's energy consumption level. It not only intuitively considers the vehicle's energy consumption, but also has the ability to further reduce the vehicle's driving energy consumption by limiting the vehicle's acceleration changes and the vehicle's accelerator pedal changes, and can improve the vehicle's driving comfort.

[0029] Step 2.4: Rewards for passing within the green light sequence is defined as follows:

[0030]

[0031] When the main vehicle passes through the signal light intersection at the green light timing, A larger reward will be given to enable the algorithm to obtain the green light faster;

[0032] Step 2.5: Reward Function for 、 and The sum is: .

[0033] Further technical solution, said step 3 includes the following specific steps:

[0034] The network results of deep reinforcement learning based on the PPO algorithm use an Actor-Critic structure. The Actor network is responsible for collecting the required current state of the controlled vehicle from the established Carla / Python joint simulation platform according to the existing strategy as input. The current state of the controlled vehicle includes longitudinal speed, the distance between the current position and the target traffic light intersection, and the time difference from the current moment to the passable green light phase. The output is the probability distribution of the executable action, namely the degree of opening of the vehicle accelerator pedal. The network is updated using policy gradients with small samples, and the old Actor network is retained to minimize the gap between the new and old strategies, ensuring the stability of the model update.

[0035] Interaction data between the Actor network and the environment , that is, the current state , the action taken at the current moment , the reward obtained by taking the current action in the current state and the state at the next moment , will be stored in the experience pool, and certain sample data will be extracted from the experience pool when the network is updated;

[0036] The advantage function can be used by the generalized advantage estimation method to estimate the value of the critic network output It is calculated that the Critic network is responsible for evaluating the interaction between the Actor network and the environment based on the input state of the controlled vehicle, so that the Actor network strategy is updated and the controlled vehicle learns an economical traffic strategy at a signalized intersection.

[0037] The proximal strategy optimization algorithm based on the truncated objective function method is adopted, and the importance sampling method is used at the same time, so that the network can be updated using the "different strategy" method, which will speed up the update speed of the network. The importance sampling weight is the ratio of the new and old strategies. The calculation method is as follows:

[0038]

[0039] in, yes The probability distribution of the strategy after the update, is the probability distribution of the strategy before the update. Then the objective function of the proximal strategy optimization algorithm based on the truncated objective function method is The calculation method is as follows:

[0040]

[0041] In the above formula, is the network's policy parameter; represents the expected estimate; Represents the truncation term, that is, the importance sampling weight Restricted to area middle, is the empirical hyperparameter, The value of is 0.2; the min function is to select the smaller of the two parameters in the brackets; for The moment advantage function is obtained using the Generalized Advantage Estimator (GAE) method. The specific expression of this method is as follows:

[0042]

[0043] In the above formula is the discount factor; is the weight coefficient; is the number of time steps selected; yes The moment time difference error is defined as follows:

[0044]

[0045] In the above formula is the reward at that time step, for The state valuation given by the Critic network at that moment.

[0046] An embodiment of the present invention provides a method for navigating signalized intersections based on proximal strategy optimization. This method enables controlled vehicles to navigate signalized intersections within green light sequences at varying initial speeds and signal phases, while also performing economical driving. Passing through signalized intersections within green light sequences not only avoids idling at red lights at intersections, thereby increasing road efficiency, but also further reduces vehicle energy consumption through economical driving throughout the entire driving process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A system flow chart of a method for passing through a signalized intersection based on proximal strategy optimization provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a driving scenario provided by an embodiment of the present invention;

[0049] Figure 3 A structural diagram of a deep reinforcement learning network model based on PPO provided in an embodiment of the present invention;

[0050] Figure 4This is the reward function change diagram for the controlled vehicle at an initial speed of 27km / h under long time sequence;

[0051] Figure 5 This is the speed change diagram of the controlled vehicle under the condition of initial speed of 27km / h under long time sequence;

[0052] Figure 6 This is the position change diagram of the controlled vehicle under the condition of initial speed 27km / h in long time sequence;

[0053] Figure 7 This is a graph showing the change in the average instantaneous energy consumption of the controlled vehicle with an initial velocity of 27 km / h as a function of the number of training rounds under long time series;

[0054] Figure 8 This is the reward function change diagram for the controlled vehicle at an initial speed of 14km / h in a short time sequence;

[0055] Figure 9 This is the speed change diagram of the controlled vehicle under the condition of an initial speed of 14km / h in a short time sequence;

[0056] Figure 10 This is the position change diagram of the controlled vehicle under the condition of initial speed 14km / h in short time sequence;

[0057] Figure 11 This is a graph showing the change in the average instantaneous energy consumption of the controlled vehicle with an initial velocity of 14km / h under short-term conditions as a function of the number of training rounds. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0060] like Figure 1 As shown, a method for passing through a signalized intersection based on proximal strategy optimization is provided in accordance with an embodiment of the present invention, comprising the following steps:

[0061] Step 1: Build a deep reinforcement learning network model based on the proximal policy optimization (PPO) algorithm. Collect real-time vehicle speed information, the distance between the vehicle's current position and the target signal intersection, the time difference between the current position and the passable green light phase, and other road information as the state space of the economic signal intersection traffic model;

[0062] Step 2: Propose an economical signalized intersection passing reward function based on factors such as vehicle energy consumption, vehicle speed, whether the vehicle can pass the target signalized intersection under the green light timing, and the vehicle speed limit under urban conditions;

[0063] Step 3: The controlled vehicle interacts with the environment in real time in the open-source vehicle simulator Carla based on the action variables generated by the economical traffic light intersection model. The actions taken by the controlled vehicle in the current state are evaluated based on the reward function, and the policy parameters in deep reinforcement learning are optimized to ultimately obtain an economical traffic light intersection strategy.

[0064] As a preferred embodiment of the present invention, step 1 includes the following specific steps:

[0065] Step 1.1: The driving condition is designed so that the initial vehicle generation position of each training round is the same distance from the traffic light intersection. The vehicle can have three different initial speeds at the initial generation position: low, medium and high. The traffic lights can have different timing and phase sequences. The above driving conditions are implemented in the Carla / Python joint simulation platform. The schematic diagram of the driving scenario is as follows Figure 2 As shown;

[0066] Step 1.2: Considering that vehicle braking will result in additional vehicle energy consumption, in order to achieve an economical signal light passing strategy, the action space in the applied proximal strategy optimization algorithm is set to the accelerator pedal opening, and the accelerator pedal opening range is set to [0,1]. The vehicle dynamics model driven by the accelerator pedal is provided by the Carla simulator.

[0067] Step 1.3: In order to enable the intelligent agent to perceive the changes in the environment in real time, the state space of the PPO algorithm is set to: the distance between the current position of the vehicle and the target traffic light intersection, recorded as ; The real-time longitudinal speed of the vehicle is recorded as The time difference between the current moment and the passable green light phase is recorded as Then the PPO algorithm has a three-dimensional state space, namely .

[0068] As a preferred embodiment of the present invention, step 2 includes the following specific steps:

[0069] Real-time speed bonus for comprehensive vehicles , the instantaneous energy consumption level reward of the vehicle during driving and rewards for passing within the green light timeline Settings. Speed Bonus It is related to the expected speed and the time from the current moment to the green light phase where the target can pass. The maximum expected passing speed is set and minimum expected passing speed The calculation formula is as follows:

[0070]

[0071]

[0072] In the above formula is the distance between the current position of the vehicle and the target traffic light intersection, is the time difference from the current moment to the passable green light phase, The duration of the green light phase. The vehicle's real-time speed bonus The specific definitions are as follows:

[0073]

[0074] In the above formula Represents the real-time longitudinal speed of the vehicle; Represents the time difference between the initial moment of each training round and the passable green light phase; Represents the driving time of the vehicle in a training round; Represents a truncation function that limits the value of the first argument in the following parentheses to between the second and third arguments; Represents the upper speed limit in the driving condition, which is set to 50 km / h in this invention. Speed reward in the reward function The reward value that should be given to the vehicle at different speeds and driving times when driving towards a signalized intersection is fully considered, and considering that signalized intersections mostly occur in urban driving conditions, a certain limit is imposed on the maximum speed of the vehicle. Have the ability to guide the vehicle through the signalized intersection at an appropriate speed under the green light timing.

[0075] Vehicle instantaneous energy consumption level reward The definition of is as follows:

[0076]

[0077] In the above formula Represents the difference in acceleration of the controlled vehicle collected in two adjacent time steps; Represents the difference in the action output of the PPO algorithm between two adjacent time steps, that is, the difference in the accelerator pedal opening; Represents the instantaneous energy consumption of the vehicle, which is calculated based on the VT-Micro fuel consumption model using the vehicle's speed and acceleration. The calculation formula of the VT-Micro fuel consumption model is as follows:

[0078]

[0079] in, It is a parameter related to vehicle acceleration and is related to factors such as the positive and negative acceleration of the vehicle during driving. and are the acceleration and velocity values of the vehicle at any moment;

[0080] Vehicle driving energy consumption level reward Taking the vehicle's energy consumption level into comprehensive consideration, it not only intuitively considers the vehicle's energy consumption, but also has the ability to further reduce the vehicle's driving energy consumption by limiting the vehicle's acceleration changes and the vehicle's accelerator pedal changes, and can improve the vehicle's driving comfort.

[0081] Rewards for passing within the green light timeline is defined as follows:

[0082]

[0083] When the main vehicle passes through the signal light intersection at the green light timing, A larger reward will be given to enable the algorithm to gain the ability to pass the green light more quickly.

[0084] Reward Function for 、 and The sum is:

[0085] .

[0086] As a preferred embodiment of the present invention, step 3 includes the following specific steps:

[0087] The network results of deep reinforcement learning based on the PPO algorithm adopt an Actor-Critic structure. The Actor network is responsible for collecting the required current state of the controlled vehicle (longitudinal speed, distance between the current position and the target traffic light intersection, and time difference from the current moment to the passable green light phase) from the established Carla / Python joint simulation platform according to the existing strategy. It outputs the probability distribution of executable actions (the opening degree of the vehicle accelerator pedal) and uses policy gradient to update the network with a small sample. The old Actor network is retained to prevent the gap between the new and old strategies from being too large, thus ensuring the stability of the model update. The interaction data between the Actor network and the environment , that is, the current state , the action taken at the current moment , the reward obtained by taking the current action in the current state and the state at the next moment , will be stored in the experience pool. When the network is updated, a certain amount of sample data will be extracted from the experience pool. The advantage function can be estimated by the Generalized Advantage Estimator (GAE) method using the value output by the Critic network. The Critic network is responsible for evaluating the interaction between the Actor network and the environment based on the input state of the controlled vehicle, so that the strategy of the Actor network is updated and the controlled vehicle learns the economical traffic strategy at the intersection. The specific network structure is as follows Figure 3 shown.

[0088] The proximal strategy optimization algorithm based on the truncated objective function method is adopted, and the importance sampling method is used at the same time, so that the network can be updated using the "different strategy" method, which will speed up the update speed of the network. The importance sampling weight is the ratio of the new and old strategies. The calculation method is as follows:

[0089]

[0090] in, yes The probability distribution of the strategy after the update, is the probability distribution of the strategy before the update. Then the objective function of the proximal strategy optimization algorithm based on the truncated objective function method is The calculation method is as follows:

[0091]

[0092] In the above formula, is the policy parameter of the network; represents the expected estimate; Represents the truncation term, that is, the importance sampling weight Restricted to area middle, is the empirical hyperparameter, The value of is 0.2; the function of min is to select the smaller of the two parameters in the brackets; for The moment advantage function is obtained using the Generalized Advantage Estimator (GAE) method. The specific expression of this method is as follows:

[0093]

[0094] In the above formula is the discount factor; is the weight coefficient; is the number of time steps selected; yes The moment time difference error is defined as follows:

[0095]

[0096] In the above formula is the reward at that time step, for The state valuation given by the Critic network at that moment.

[0097] The pseudo code of the proposed economical traffic strategy at signalized intersections is as follows:

[0098] Table 1 Pseudo code of the economical signal light intersection traffic strategy PPO proposed by the present invention

[0099]

[0100] The reward function changes as follows: Figure 4 As shown, the speed changes as Figure 5 As shown, the position changes as Figure 6 As shown in the figure, the average instantaneous energy consumption changes with the number of training rounds as shown in the figure. Figure 7 As shown in the figure, a long time sequence means that at the beginning of each training round, the traffic light phase sequence is green-red-green, with a green light duration of 10 seconds and a red light duration of 15 seconds. The initial speed of the controlled vehicle is set at 27 km / h. As the number of training rounds increases, the controlled vehicle learns to drive economically through the second green light phase (if it passes the first green light phase, it will exceed the speed limit in urban driving conditions). The controlled vehicle's speed can be maintained stable or increase smoothly, and its average instantaneous energy consumption also decreases with the number of training rounds.

[0101] In the short time sequence, the initial speed of the controlled vehicle is 14km / h. The reward function changes as follows: Figure 8 As shown, the speed changes as Figure 9 As shown, the position changes as Figure 10 As shown in the figure, the average instantaneous energy consumption changes with the number of training rounds as shown in the figure. Figure 11As shown in the figure, short-term timing refers to the phase sequence of the traffic lights being red-green at the beginning of each training round, with a green light duration of 10 seconds and a red light duration of 15 seconds. The initial speed of the controlled vehicle is set at 14 km / h. As the number of training rounds increases, the controlled vehicle learns to drive economically through the green light phase. The longitudinal speed of the controlled vehicle increases smoothly within a training round, and its average instantaneous energy consumption decreases with the number of training rounds.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A traffic light intersection passing method based on proximal strategy optimization, characterized in that: The following steps are involved: Step 1: Build a deep reinforcement learning network model based on the proximal policy optimization algorithm. The collected real-time vehicle speed information, the distance between the vehicle's current position and the target traffic light intersection, and the time difference from the current position to the passable green light phase are used as the state space of the economic traffic light intersection model. Step 2: Based on the vehicle's energy consumption, vehicle speed, whether it can pass the target signalized intersection under the green light timing, and the vehicle's speed limit under urban conditions, an economical signalized intersection passing reward function is proposed; Step 3: The controlled vehicle interacts with the environment in real time in the open-source vehicle simulator Carla based on the action variables generated by the economical signal light intersection traffic model. The actions taken by the controlled vehicle in the current state are evaluated based on the reward function, and the policy parameters in the deep reinforcement learning are optimized to ultimately obtain an economical signal light intersection traffic strategy. The step 2 includes the following specific steps: Step 2.1: Real-time speed reward for integrated vehicles , the instantaneous energy consumption level reward of the vehicle during driving and rewards for passing within the green light timeline Setting; setting the maximum expected passing speed and minimum expected passing speed The calculation formula is as follows: ； ； In the above formula is the distance between the current position of the vehicle and the target traffic light intersection, is the time difference from the current moment to the passable green light phase, The duration of the green phase; Step 2.2: Vehicle's real-time speed reward The specific definitions are as follows: ； In the above formula Represents the real-time longitudinal speed of the vehicle; represents the initial velocity; Represents the time difference between the initial moment of each training round and the passable green light phase; Represents the driving time of the vehicle in a training round; Represents a truncation function that limits the value of the first argument in the following parentheses to between the second and third arguments; Represents the upper speed limit in the driving condition, which is set to 50 km / h; Step 2.3: Vehicle instantaneous energy consumption level reward The definition of is as follows: ； In the above formula Represents the difference in acceleration of the controlled vehicle collected in two adjacent time steps; Represents the difference in action output of the proximal strategy optimization algorithm between two adjacent time steps, that is, the difference in accelerator pedal opening; Represents the instantaneous energy consumption of the vehicle, which is calculated based on the VT-Micro fuel consumption model using the vehicle's speed and acceleration. The calculation formula of the VT-Micro fuel consumption model is as follows: ； in, It is a parameter related to vehicle acceleration and is related to the positive or negative acceleration of the vehicle during driving; and are the acceleration and velocity values of the vehicle at any moment; Step 2.4: Rewards for passing within the green light sequence is defined as follows: ； When the main vehicle passes through the signal light intersection at the green light timing, Rewards will be given to improve the algorithm's ability to pass the green light; Step 2.5: Reward Function for 、 and The sum is: .

2. The signal light intersection passing method based on proximal strategy optimization according to claim 1 is characterized in that: The step 1 includes the following specific steps: Step 1.1: The driving condition is designed so that the initial vehicle generation position of each training round is the same distance from the traffic light intersection. The vehicle can have three different initial speeds at the initial generation position: low, medium and high. ,The traffic lights have different timing and phase sequences.,The construction of the above driving conditions is implemented in the Carla / Python ,co-simulation platform; Step 1.2: Set the action space in the applied proximal strategy optimization algorithm to the accelerator pedal opening, and set the range of the accelerator pedal opening to [0, 1]. Step 1.3: To enable the agent to perceive changes in the environment in real time, the state space of the proximal strategy optimization algorithm is set to: the distance between the current position of the vehicle and the target traffic light intersection, denoted as ; The real-time longitudinal speed of the vehicle is recorded as The time difference between the current moment and the passable green light phase is recorded as ; Then the proximal strategy optimization algorithm has a three-dimensional state space, that is .

3. The signal light intersection passing method based on proximal strategy optimization according to claim 2 is characterized in that: In step 1.2, the vehicle dynamics model driven by the accelerator pedal is provided by the Carla simulator.

4. The signal light intersection passing method based on proximal strategy optimization according to claim 3 is characterized in that: The step 3 includes the following specific steps: The network results of deep reinforcement learning based on the proximal policy optimization algorithm adopt an actor-critic structure. In this structure, the actor network is responsible for collecting the required current state of the controlled vehicle from the established Carla / Python joint simulation platform according to the existing strategy as input. The current state of the controlled vehicle includes longitudinal speed, distance between the current position and the target traffic light intersection, and the time difference between the current position and the passable green light phase. The output is the probability distribution of the vehicle's accelerator pedal opening, and the policy gradient is used to update the network with a small sample size. The old actor network is retained to ensure the stability of the model update. Interaction data between the Actor network and the environment , that is, the current state , the action taken at the current moment , the reward obtained by taking the current action in the current state and the state at the next moment , will be stored in the experience pool, and sample data will be extracted from the experience pool when the network is updated; The advantage function is estimated by the generalized advantage estimation method using the value output of the critic network It is calculated that the Critic network is responsible for evaluating the interaction between the Actor network and the environment based on the input state of the controlled vehicle, so that the Actor network strategy is updated and the controlled vehicle is assisted in learning an economical traffic strategy at a signal light intersection.

5. The signal light intersection passing method based on proximal strategy optimization according to claim 4 is characterized in that: In step 3, the proximal strategy optimization algorithm based on the truncated objective function method is adopted, and the importance sampling method is used at the same time, so that the network is updated using the "different strategy" method, and the importance sampling weight is the ratio of the new and old strategies. The calculation method is as follows: ； in, yes The probability distribution of the strategy after the update, is the probability distribution of the strategy before the update, then the objective function of the proximal strategy optimization algorithm based on the truncated objective function method is The calculation method is as follows: ； In the above formula, is the network's policy parameter; represents the expected estimate; Represents the truncation term, that is, the importance sampling weight Restricted to area middle, is the empirical hyperparameter, The value of is 0.2; the min function is to select the smaller of the two parameters in the brackets; for The moment advantage function is obtained using the generalized advantage calculation method. The specific expression of this method is as follows: ； In the above formula is the discount factor; is the weight coefficient; is the number of time steps selected; yes The moment time difference error is defined as follows: ； In the above formula is the reward at that time step, for The state valuation given by the Critic network at that moment.

Images