An automatic road intersection management method based on a projection constraint strategy optimization

Abstract

The application is suitable for the technical field of intelligent transportation system, and provides an automatic road intersection management method based on a projection type constraint strategy optimization, introduces a risk network, divides a safety level when updating a strategy, maximizes a reward value on the premise of meeting safety, and solves the problem that an enhanced learning algorithm will violate safety when pursuing traffic efficiency and comfort. In addition, a strategy neural network applies an LSTM neural network to capture the relationship between each vehicle and centrally regulate all vehicles in a complex intersection to obtain a strategy with better performance. Through experimental testing, the performance of the method is compared with that of a PPO and a PPO-SC based on non-risk perception, an MPC based vehicle intersection coordination scheme algorithm and an MICA algorithm based on MIP, and experiments show that the method has better performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent transportation systems, and particularly relates to an automatic road intersection management method based on a Multi-Agent Projection-Based Constrained Policy Optimization (MAPCPO). BACKGROUND

[0002] In recent years, the rapid development of connected automatic vehicles (CAVs) has brought new opportunities for controlling and managing the flow of vehicles, people and goods within and outside cities, and traffic control is rapidly changing. The challenge facing the new intelligent transportation system is to provide new methods to control autonomous vehicles to improve the safety, efficiency and comfort of vehicle traffic in cities. Therefore, improving and introducing new control strategies are crucial for effective traffic management decisions. Among them, the complex road intersection is an extremely important scene in urban road sections. According to statistics, traffic accidents at complex road intersections account for about 30% of all traffic accidents in China, so it is of great significance to improve the safety of complex road intersection traffic. The autonomous intersection management (AIM) system aims to effectively manage autonomous vehicles at complex road intersections in cities, eliminate collisions and optimize overall traffic flow.

[0003] Most of the AIMs currently use rule-based methods, such as first-come-first-served, which still has a lot of waiting delay, and the performance at non-equilibrium intersections is even lower than that of traditional traffic lights; or formula-based traditional methods, but the traffic state at real road complex intersections is constantly changing, and traditional methods based on formulas such as model predictive control (MPC) and mixed integer programming (MIP) have the problem of large calculation amount and high delay when encountering more complex scenarios. In recent years, deep neural networks have emerged, and deep neural networks require large data sets for training, but after the network is trained, it only takes a very short time to control vehicles according to the current traffic state, solving the problem of too much calculation time required by traditional formula-based methods. However, two problems have arisen: 1) deep neural networks are black box technologies, and people cannot intuitively see the working principle of deep neural networks, which poses a safety risk when applied in the field of intelligent driving, and safety is an extremely important part of intelligent driving, which limits the application of deep neural networks in real traffic scenarios; 2) most deep neural network training is supervised training, and after training in the data set, the strategy is directly applied, lacking the ability to continuously learn from real traffic scenarios.

[0004] The emergence of reinforcement learning (RL) solves the second problem, which is an unsupervised learning with four elements of policy, value, reward and model. The agent explores different behaviors in the model, the value function evaluates the benefits of the current behavior of the agent, and the algorithm updates the current policy of the agent according to the benefits evaluated by the value function. RL has the ability of continuous learning, which can continuously update the policy from the traffic scene, solving the problem that the deep neural network cannot be continuously updated. However, the reinforcement learning algorithm does not constrain the exploration behavior of the agent in the learning process, and the agent may explore dangerous behaviors due to inappropriate rewards. This is extremely deadly in the field of autonomous driving, and it is not an easy problem to assign a proper reward function. In addition, the trained strategy can only maximize the reward and cannot guarantee the safety of the strategy. For example, the current more mainstream reinforcement learning algorithms include DDPG (Deep Deterministic Policy Optimization) and A3C (Asynchronous Advantage Actor-Critic). These algorithms update the strategy by designing a reward function. Since the traffic efficiency, comfort and safety are in a contradictory relationship to some extent, the strategy cannot guarantee to improve the traffic efficiency and comfort on the premise of meeting the safety in the process of pursuing the maximization of the reward. The algorithm will violate the safety condition which has a higher priority when pursuing higher efficiency and comfort. SUMMARY

[0005] The embodiment of the application aims to provide an automatic road intersection management method based on projection constraint policy optimization, which aims to solve the problems in the background technology.

[0006] The embodiment of the application is implemented as follows: an automatic road intersection management method based on projection constraint policy optimization, comprising the following steps:

[0007] Step 1, projection constraint policy optimization:

[0008] The six elements include a policy network, a model, a reward function, a risk function, a value estimation network, and a risk estimation network; the policy network uses an LSTM neural network to determine the next behavior of the agent according to the current state of the agent; the model is a scenario in which the agent interacts with the environment, and the agent explores in the model to learn the optimal policy; the reward function is used to guide the agent to learn to achieve the expected goal; the value estimation network is used to evaluate the current policy of the agent to update the policy network; the risk function is used to ensure the safety of the exploration process of the agent; and the risk estimation network is used to evaluate the risk degree of the current behavior of the agent to limit the update of the policy network to a certain extent;

[0009] When the policy network is updated, the safety risk of the current policy is first determined, and the safety risk is divided into three levels: safe, relatively safe, and dangerous; if the current policy is in the safe range, the risk function is not considered when the policy is updated, and the policy will be updated in the direction of maximizing the reward; if the current policy is in the relatively safe range, the policy network is first updated to maximize the reward, and then the policy is projected on the risk constraint set to make the policy return to the safe range; if the current policy is in the dangerous range, the parameters suitable for network update are found by backtracking;

[0010] Step 2, automatic intersection management based on projection constraint policy optimization:

[0011] A complex intersection scene is built in the Carla environment as a model, and a reward function and a risk function are designed; the reward function takes into account the traffic efficiency, comfort, collision safety, and lateral control stability of the vehicle, and the risk function takes into account the collision risk and the risk of the vehicle distance being less than the safe distance;

[0012] The policy training process is as follows: the vehicle explores the behavior within a certain time, the policy network controls the speed of all vehicles in the environment by inputting the distance of the vehicle from the target location and the current speed of the vehicle, and the route of the vehicle is designed by the simulator; if a collision occurs or all vehicles pass through, the scene is reset; the input of the policy network, the output of the policy network, the score, the risk value, the value estimation value, and the risk estimation value are collected within this period of time to update the policy network, the value estimation network, and the risk estimation network.

[0013] The embodiment of the application provides an automatic road intersection management method based on a projection type constraint strategy optimization, the method introduces a risk network, divides a safety level when updating a strategy, maximizes a reward value on the premise of meeting safety, and solves the problem that an reinforcement learning algorithm will violate safety when pursuing traffic efficiency and comfort. In addition, the strategy neural network applies an LSTM neural network to capture the relationship between each vehicle and centrally regulate all vehicles in a complex intersection to obtain a strategy with better performance. The performance of the method is compared with the performance of a PPO and a MAPPO-SC based on non-risk perception, a vehicle intersection coordination scheme (VICS) algorithm based on MPC, and a MICA (Mixed integer programming based Intersection Coordination Algorithm, MICA) algorithm based on MIP, and experiments show that the method has better performance, compared with the PPO algorithm, zero collision is realized for the first time, the comfort is improved by 46.15%, and compared with a traditional method, the traffic efficiency is improved by 34.54%. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 FIG. 1 is a schematic diagram of an automatic road intersection management method based on a projection type constraint strategy optimization provided by the embodiment of the application;

[0015] Figure 2 FIG. 2 is a structure diagram of a strategy neural network;

[0016] Figure 3 FIG. 3 is a schematic diagram of a complex intersection structure;

[0017] Figure 4 FIG. 4 is a round average reward value of MAPCPO, MAPPO-SC and MAPPO in a training stage;

[0018] Figure 5 FIG. 5 is a round average risk value of MAPCPO, MAPPO-SC and MAPPO in a training stage;

[0019] Figure 6 FIG. 6 is a round average collision rate of MAPCPO, MAPPO-SC and MAPPO in a training stage;

[0020] Figure 7 FIG. 7 is a TTC violation frequency of MAPCPO, MAPPO-SC and MAPPO in a training stage;

[0021] Figure 8 FIG. 8 is a change trend of acceleration of MAPCPO, MAPPO-SC and MAPPO in a training stage;

[0022] Figure 9 average round collision rate plot for each method when the traffic flow density is

[0023] Figure 10 TTC violation frequency plot for each method when the traffic flow density is

[0024] Figure 11 average round acceleration plot for each method when the traffic flow density is

[0025] Figure 12 average round length plot for each method when the traffic flow density is

[0026] Figure 13 average round collision rate plot for each method when the traffic flow density is

[0027] Figure 14 TTC violation frequency plot for each method when the traffic flow density is

[0028] Figure 15 average round acceleration plot for each method when the traffic flow density is

[0029] Figure 16 average round length plot for each method when the traffic flow density is

[0030] Figure 17 average round collision rate plot for each method when the traffic flow density is

[0031] Figure 18 TTC violation frequency plot for each method when the traffic flow density is

[0032] Figure 19 average round acceleration plot for each method when the traffic flow density is

[0033] Figure 20 average round length plot for each method when the traffic flow density is DETAILED DESCRIPTION

[0034] ​​​​​​​​​​​​To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0036] like Figure 1 As shown, an automatic road intersection management method based on projection constraint strategy optimization is provided in one embodiment of the present invention, including the following steps:

[0037] Step 1: Optimization of Projective Constraint Strategy

[0038] It comprises six elements: a policy network, a model, a reward function, a risk function, a value estimation network, and a risk estimation network. The policy network uses an LSTM neural network and is responsible for determining the agent's next action based on its current state. The model represents the interaction scenario between the agent and its environment, allowing the agent to explore and learn the optimal policy. The reward function guides the agent's learning, enabling it to achieve the expected goal. The value estimation network evaluates the agent's current policy to update the policy network. The risk function ensures the safety of the agent's exploration process. The risk estimation network assesses the danger level of the agent's current behavior to limit the policy network's updates to some extent.

[0039] When updating, the policy network first assesses the safety risk of the current policy, which is categorized into three levels: safe, relatively safe, and dangerous. If the current policy is within the safe range, the risk function is not considered during the policy update, and the policy will be updated in the direction of maximizing reward. If the current policy is within the relatively safe range, it means that the policy network can return to or remain within the safe range through updates. In this case, the policy network first performs a reward-maximizing update, and then projects the policy onto the risk constraint set to bring the policy back to the safe range. If the current policy is within the dangerous range, it means that the policy cannot return to the safe range through updates. In this case, backtracking is used to find suitable parameters for the network update.

[0040] Step 2: Automatic intersection management based on projection constraint strategy optimization:

[0041] A complex intersection scenario was built in the Carla environment as a model, and a scientific reward function and risk function were designed. The reward function considers the vehicle's traffic efficiency, comfort, collision safety and lateral control stability, while the risk function considers the vehicle's collision risk and the risk of the vehicle distance being less than the safe distance.

[0042] The strategy training process is that the vehicle explores the behavior in a certain time, the strategy network controls the speed of all vehicles in the environment by inputting the distance of the vehicle from the target place and the current speed of the vehicle, the route of the vehicle is designed by the simulator, and the scene is reset if a collision occurs or all vehicles pass through, and the input of the strategy network, the output of the strategy network, the score, the risk value, the value estimation value and the risk estimation value in this period of time are collected to update the strategy network, the value estimation network and the risk estimation network.

[0043] As a preferred embodiment of the present application, the step 1 comprises the following specific steps:

[0044] Step 1.1, algorithm definition: the projection constraint strategy optimization problem is described as a constraint Markov game (CMG), on the basis of the traditional constraint Markov game, the set of allowed strategies is limited by adding constraints; a CMG for running an agent is defined as a tuple , wherein represents the global state space, defined as , represents the set of joint action spaces of all agents; represents the joint reward function, which describes the state taking joint action to the next state immediate reward, represents the set of risk functions defined by the safety constraints of the specific environment (each agent has risk function), transition tuple maps to risk cost threshold , represents the probability of transition from to state , represents the initial distribution of three neural network parameters, represents the discount factor;

[0045] Step 1.2, constructing a strategy neural network: the input of the strategy neural network is the joint state of all agents at the current time, and the output is the behavior of all agents at the next time; the structure of the strategy neural network is as shown in Figure 2 , which contains three Linear layers and one LSTM layer; assuming that there are agents in a complex intersection scenario, the input of the strategy neural network is the information collected by each agent at the time step, represented as , the state of all agents collected at each time step is a matrix, which is first changed to and then activated by a Relu function, followed by an LSTM layer whose output dimension becomes The output of the LSTM is adjusted in range (RA) to ensure that the output of the LSTM network is suitable for a real scenario, and the result after the adaptation operation is sent to the Linear layer and the Relu activation operation is performed, and finally the dimension of the output result is changed to and Sigmoid activation is performed as the final output result of the policy network;

[0046] Step 1.3, constructing a value estimation and risk estimation neural network: the function of the value estimation and risk estimation neural network is to determine whether the updated policy neural network has higher rewards and lower risks according to the current collected trajectory information, and a fully connected layer neural network model is used as the value estimation and risk estimation neural network. First, the state input of the network is the same as the policy network model, and the distance of each vehicle from the target location and the current speed of the vehicle are input, with a dimension of After passing through three connection layers and performing Relu activation operation, the final output result is obtained, and the dimension changes in turn The update principle of the value estimation and risk estimation neural network is as follows:

[0047] The value estimation and risk estimation neural network is updated by the difference between the predicted value and the true value, and the purpose of the update is to reduce the difference between the predicted value and the true value, which is represented as:

[0048] ;

[0049] (10).

[0050] As a preferred embodiment of the present application, in step 1.2, in order to ensure the safety of the policy neural network update, the policy neural network update is divided into three cases: the KL divergence trust domain is within the risk trust domain, the KL divergence trust domain intersects with the risk trust domain, and the KL divergence trust domain has no intersection with the risk trust domain. First, define the value function for the reward value , which represents the expected reward value corresponding to the state , wherein represents the trajectory dependent on the policy , and represents the discounted reward value corresponding to the trajectory . Define the behavior value function as , which represents the expected reward value corresponding to the state and the behavior , and define the advantage function as , which represents the state Under the premise of consistency, behavior The difference between the corresponding reward and the expected reward, if A positive value indicates that the action... The overall reward value can be higher, if A negative value indicates that the action was performed. The overall reward value obtained is lower than the average reward, defined with neural network parameters. Policy Neural Network The corresponding expected return is Similarly, the value function, behavioral function, and advantage function are defined for risk as follows: Relative to the risk function The expected value of the discounted risk is Then the set of feasible strategies for the CMGs model can be represented as: ,in Indicates in Time of the first Risk threshold of an agent.

[0051] pass The update method is determined comprehensively, among which Representing the risk function gradient, This indicates how close the agent's risk value is to the risk threshold under the current policy. (Definition) ,but Its function is to determine whether the KL divergence trust region and the risk trust region have an overlapping relationship, whereby... The Hessian matrix represents the KL divergence between the old and new strategies.

[0052] when Extremely small ( This indicates that the current policy and its surrounding neighborhood are in a state of low danger potential, and updating the policy in any direction will not pose a danger; when and The statement indicates that the KL divergence trust domain of the current policy is within the risk trust domain, meaning that updating the policy in any direction will not be dangerous. This invention categorizes these two cases as Case 3. When the policy is in Case 3, the trust domain policy optimization algorithm is applied to update the policy network parameters:

[0053] (1);

[0054] ;

[0055] Based on the second-order approximation of the KL divergence constraint... The objective function at point is linearized, resulting in the following update:

[0056] (2);

[0057] ;

[0058] where denotes the gradient of the reward advantage function, the network update formula is obtained by solving equation (2) using convex optimization method:

[0059] (3);

[0060] When , it means that the risk trust region and the KL divergence trust region are intersected, and at this time, if the expected reward value is maximized, the policy may enter a dangerous area. This situation is classified as case 2. If the policy neural network is in case 2, the policy network is updated using the projection constraint policy optimization, which is divided into two steps: the reward promotion step and the projection step. The reward promotion step maximizes the reward value through the trust region policy optimization (TRPO), and the projection step projects the policy network after the reward promotion step into the risk trust region. First, maximize the advantage function :

[0061] (4);

[0062] ;

[0063] Secondly, by minimizing the distance between the intermediate policy and the risk trust region, the policy is projected into the risk trust region:

[0064] (5);

[0065] ;

[0066] The second-order approximate KL divergence constraint is linearized to the objective function at , and the reward promotion step update approximation is obtained:

[0067] (6);

[0068] ;

[0069] The distance between and the risk trust region is measured by the KL divergence, and the second-order approximate KL divergence is also linearized to the cost constraint at :

[0070] (7);

[0071] ;

[0072] Solve equations (6) and (7) using convex optimization methods:

[0073] (8);

[0074] like and If the current policy is in a relatively dangerous state and the KL divergence trust domain does not overlap with the risk trust domain, then updating the policy in any direction will not result in a safe state. This situation is classified as Case 1. If the policy is in Case 1, a linear backtracking method is used to update the policy neural network to minimize the risk value.

[0075] (9);

[0076] In a preferred embodiment of the present invention, step 2 includes the following specific steps:

[0077] Step 2.1, Design of State Space and Behavior Space:

[0078] Define the state space as , This represents the distance of an autonomous vehicle entering a complex intersection area from its departure point. This represents the driving direction of the autonomous vehicle, specifically indicating right turn, straight ahead, and left turn.

[0079] like Figure 3 As shown, due to the irregularity of the intersection, The calculation involves two scenarios: vehicles entering from the straight road from the east, south, and north, and vehicles entering from the curve on the west side. When a vehicle enters from the straight road, such as... Figure 3 As shown in .a, there are three types of traffic actions: right turn, straight ahead, and left turn. The calculation formulas for these three actions are:

[0080] (11);

[0081] When a vehicle enters from a curve, such as Figure 3 As shown in .b, there are also three types of traffic: right turn, straight ahead, and left turn. The calculation formula is as follows:

[0082] (12);

[0083] Represents intelligent driving vehicles Real-time speed at current time step. Each vehicle has only one driving direction when passing through the complex intersection, the state space of the remaining two driving directions is filled with zero vectors, for example, intelligent driving cars right-turn behavior . The strategy can give intelligent driving cars steering information, combined with The strategy can determine the position of intelligent driving cars at the complex intersection, and the state space includes the current speed information of each intelligent driving car. By integrating all the information in the state space, a reasonable complex intersection coordination strategy can coordinate all intelligent driving cars to pass through the complex intersection more comfortably and efficiently without collision.

[0084] The action space is defined as Since the lateral control of intelligent driving cars is managed by the path planning module in the simulator, and the simulator can use the PID algorithm to get reasonable throttle opening and brake force to control the intelligent driving car to smoothly reach the desired speed, in order to make the strategy focus more easily on the key information of the problem, the joint action space of the model only selects the future joint desired speed of all intelligent driving cars in the complex intersection area.

[0085] Step 2.2, define risk function and reward function:

[0086] The defined complex intersection coordination problem is a multi-attribute problem, including the safety, efficiency and comfort of intelligent driving cars passing through the complex intersection. Among them, safety is a prerequisite for efficiency and comfort, so the risk function is responsible for monitoring the safety of complex intersection coordination, and the reward function should include the three attributes of safety, efficiency and comfort.

[0087] First, define the risk function. The potential safety risks of intelligent driving cars passing through the complex intersection include collision and skidding, so the risk function is designed for these two safety risks.

[0088] Collision: Collision is more dangerous than planning, and will pose a great threat to personal safety and property safety, so the risk score should account for a large proportion:

[0089] (13);

[0090] Among them represents the risk value given when a collision occurs, which aims to prevent intelligent driving cars from colliding; represents the risk value given when an intelligent driving car violates the safe distance when there is a risk of collision at time step ​The number of intelligent driving vehicles that represent the risk of collision and violate the safety distance, the purpose is to prevent the distance between the intelligent driving vehicles that have the risk of collision is small, to ensure that the intelligent driving vehicles have enough brake distance when the collision is about to occur.

[0091] Side slip: when the intelligent driving vehicle occurs left turn or right turn behavior, the high speed or longitudinal acceleration will cause the friction provided by the ground to be insufficient, thus causing a side slip accident, therefore the design of the risk function should prevent the intelligent driving vehicle from having too high speed or longitudinal acceleration when turning:

[0092] (14);

[0093] wherein represents the instantaneous acceleration of the intelligent driving vehicle at time step , represents the gravitational acceleration, represents the friction coefficient of the complex intersection road surface. In summary, the risk function is defined as:

[0094] (15);

[0095] Then define the reward function, safety: the reward of the safety of the intelligent driving vehicle is divided into guiding reward and decisive reward. The guiding reward is intended to reduce the difficulty of training and speed up the training, and the decisive reward is intended to let the strategy understand the optimal solution expected by people and prevent the aggressive behavior caused by the greedy guiding reward:

[0096] (16);

[0097] wherein represents the guiding reward, represents the decisive reward.

[0098] Efficiency: under the premise of ensuring safety, it is crucial to improve the road traffic efficiency, and the algorithm defines the reward of the efficiency attribute as:

[0099] (17);

[0100] wherein is the discount factor, represents the difference between the speed of the intelligent driving vehicle at time step and the expected speed, represents the reward loss caused by time accumulation.

[0101] Smoothness: smoothness is closely related to the comfort of passengers, and the algorithm defines the reward of the smooth attribute as:

[0102] (18);

[0103] wherein is a discount factor, represents the acceleration of the intelligent driving car at a time step. In summary, the reward function is defined as:

[0104] (19).

[0105] As a preferred embodiment of the present application, in each time step of the experiment, the static information of the road in the given scene is given, including the length, width and position coordinates of each point on the road, the dynamic information of all vehicles in the scene is transmitted through V2 communication technology, including the position coordinates, vehicle speed, behavior mode of the vehicle and the distance from the vehicle to the departure point at this time step, and the AIM controls the expected speed of the vehicle at the next time step through these known information. The motion control layer of the vehicle calculates the throttle opening and brake force size of the vehicle at the next time step through the expected speed given by the AIM system, and the steering wheel angle of the vehicle is autonomously controlled by the path planning module in the simulator. The working of the present application aims to regulate the expected speed of the vehicle through the AIM system to make all vehicles pass through efficiently and comfortably under the premise of no collision.

[0106] In order to better show the superiority of the MAPCPO algorithm, the training process of MAPCPO is compared with that of multi-agent Proximal Policy Optimization (MAPPO) and Multi-Agent Proximal Policy Optimization-Safety Constraints (MAPPO-SC) algorithm, wherein MAPPO-SC has the same reward function setting as MAPCPO, indicating that the MAPCPO algorithm has better overall performance compared with the MAPPO algorithm. In addition, the optimal strategy trained by MAPCPO is compared with the optimal strategies trained by MAPPO and MAPPO-SC, the VICS method based on MPC and the MICA method based on MIP in terms of safety, comfort, traffic efficiency and method reasoning time. The following will be introduced in detail from the experimental settings, training performance comparison and AIM system performance comparison.

[0107] ​(1) Experimental setup: All experiments were conducted in Carla 0.9.11 simulation environment, and the 'TOWN5' map was used to build a complex intersection traffic scene, and the MAPCPO model was built using the PyTorch-based framework. The dynamic information of the vehicle depends on the built-in sensor of Carla, and the BasicAgent class is used to plan the trajectory of the vehicle and control the steering angle of the vehicle. During the experiment, the expected speed of the vehicle at each time step is output by the AIM system, which is converted into the size of the throttle opening and brake torque through the built-in PID algorithm of Carla to control the operation of the vehicle. The operating system is Ubuntu 18.04, and the GPU model is NVIDIA GeForce RTX 3090.

[0108] A four-way two-lane signal-free complex intersection in 'TOWN5' was selected as the training and testing environment for the RL model. The characteristics of this complex intersection are as follows: the road width is 14.2 meters, the length of the east-west lane (departure area) is 65 meters, and the length of the south-north lane (departure area) is 50 meters. Considering the road characteristics in Carla map and the coverage range of V2I communication, the length of the east-west control area is set to 70 meters, and the length of the south-north control area is set to 60 meters.

[0109] In order to simulate the real traffic flow as much as possible, a variety of vehicle types were selected for the experiment in the simulation environment. The length of these vehicles is between 3.6 and 5.4 meters, the width is between 1.8 and 2.2 meters, and the height is between 1.5 and 2 meters. It is assumed that the arrival of vehicles obeys the Poisson distribution. According to the set average traffic flow per hour , the invention uses the Poisson function in the numpy package to generate a random number . This random number is used to determine the time interval for vehicles to enter the complex intersection. Further, the free running speed of the vehicle before entering the control area is considered to calculate the distance between adjacent vehicles. Then, the position coordinates of the vehicles are continuously generated to create a continuous traffic flow that conforms to the Poisson distribution in Carla. To simulate the control of real traffic vehicles, the invention selects a time step of 0.1s in the simulator.

[0110] For the strategy neural network of the MAPCPO algorithm, the LSTM network structure is adopted, in addition to which the strategy and value neural networks of MAPCPO, MAPPO and MAPPO-SC all have two hidden layers of Fourier neural networks. For each policy iteration, 2048 time steps of trajectory data will be collected, and the strategy neural network will be updated using the MAPCPO algorithm, and the value neural network will be updated using the Adam optimizer. In addition, the parameter settings of the MPC-based VICS algorithm and the MIP-based MICA algorithm are the same as the original method.

[0111] (2) Training performance comparison: This part shows the difference of training performance between MAPCPO, MAPPO and MAPPO-SC. MAPPO algorithm uses PPO algorithm for complex intersection vehicle cooperative control, and MAPPO-SC algorithm replaces the reward function with the reward function of MAPCPO method (including related safety constraints) on the basis of the original MAPPO algorithm.

[0112] Although MAPPO-SC shares the same safety-related constraints with MAPCPO, it uses the strategy update mechanism and parameters used by MAPPO. That is, these constraints affect the update of the strategy by being incorporated into the penalty term of the reward function, rather than as independent constraints that need to be explicitly obeyed. Figures 4-8 The comparative analysis of the performance indicators of MAPCPO, MAPPO-SC and MAPPO in the training phase is shown. The line represents the average value of the training trajectory, and the shaded area represents the related variance.

[0113] The MAPCPO algorithm uses progressive training, first using medium traffic scenarios for training, and then using high traffic scenarios for the second stage of training when the collision rate of vehicles stabilizes at 0. The MAPPO and MAPPO-SC algorithms are directly trained using high traffic scenarios, and each algorithm iterates 2000 times.

[0114] Figure 4 The difference in average reward values for each round of the three algorithms is shown. Since the reward function of the MAPPO algorithm is the same as the original text, the numerical value of the reward function is lower than that of MAPCPO and MAPPO-SC, so the reward value of the MAPPO algorithm is generally lower. In terms of the optimal performance achieved by the strategy, it can be seen that after the curve converges, the reward value of the MAPCPO algorithm is higher than that of the MAPPO-SC algorithm. The main reason is that the MAPCPO algorithm satisfies the risk constraint when updating the strategy, and can finally keep the risk value near 0, while the MAPPO-SC algorithm cannot guarantee to minimize the risk value when pursuing maximum reward during updating. In terms of the stability of the strategy, it can be seen that since the MAPCPO algorithm has strict risk constraints when updating the strategy, the curve has little fluctuation after convergence, while the MAPPO and MAPPO-SC algorithms have large fluctuations. From these two aspects, the safety and stability of the basic algorithm of reinforcement learning do not meet the requirements of autonomous driving.

[0115] Figure 5The differences of the average risk values of the three algorithms are shown. The dark horizontal dotted line at the bottom of the figure represents the risk threshold. From the final training results, only the MAPCPO algorithm with risk constraints added during policy updating can maintain the risk value below the risk threshold. Since the MAPPO-SC algorithm adds risk constraints in the reward function compared with the MAPPO algorithm, the risk value is lower than that of the MAPPO algorithm.

[0116] The collision rate is the most direct indicator for evaluating the safety of the algorithm, as shown in Figure 6 The collision rate curve corresponds to the risk value curve. A lower risk value means a lower collision rate. In the design of the reinforcement learning reward function, the greater the punishment of collision on the reward value, the safer the strategy learned by the algorithm, but it is impossible to keep the collision rate at 0. For example, the MAPPO algorithm and the MAPPO-SC algorithm, the MAPPO-SC algorithm adds the safety constraints of the MAPCPO algorithm in the reward function, so it has a lower collision rate, but both algorithms cannot maintain the collision rate at 0, but the method of the present application can do this.

[0117] The TTC (Time to Collision) violation frequency represents the size of the potential risk during vehicle driving, as shown in Figure 7 The TTC violation frequency is not directly related to the collision rate, but a high TTC violation frequency means a greater probability of collision, so it has the same trend as Figure 5 and 6 , which again shows that the MAPCPO algorithm has higher safety.

[0118] Figure 8 The change trend of acceleration during the training process of the three algorithms is shown. Acceleration is a key indicator for representing ride comfort. Since the design of the reward function in the original MAPPO algorithm does not include ride comfort, the training cannot reduce the acceleration and the curve fluctuates greatly. The MAPCPO algorithm designs a scientific reward function, which includes ride comfort, so the training can improve this performance. Compared with the MAPPO-SC algorithm with the same reward function, the MAPCPO algorithm has a lower average acceleration per round after the curve converges, which represents the superiority of the MAPCPO algorithm.

[0119] (3) AIM system performance comparison: The strategies trained by MAPCPO, MAPPO and MAPPO-SC algorithm, and the AIM system performance of VICS based on MPC and MICA based on MIP under three different traffic densities are compared, including the average collision rate (MCR), the average TTC violation frequency (TTCVF), the average episode length (MEL) and the average acceleration (MEA). Each method conducts ten experiments, each experiment including ten traffic attempts, all vehicles passing safely or colliding as a traffic attempt, and the above evaluation indexes are obtained by taking the average of ten experiments. The low, medium and high traffic densities are , , respectively. The experimental results are shown in Figures 9-20 . The upper and lower limits of the box plot represent the maximum and minimum values of the test data, and the upper and lower boundaries of the box represent the data fluctuation.

[0120] In terms of safety, from the average and fluctuation of MCR and TTCVF, no matter what the traffic density is, the strategy trained by MAPCPO method, the strategy based on MPC method and the strategy based on MIP method have absolute advantages compared with the other two methods, and the strategies trained by MAPPO and MAPPO-SC algorithms hinder the application of real vehicles due to their high risk. The passing strategies based on MPC and MIP have the same safety level as the strategy trained by MAPCPO method, but the cost is high MEL caused by high reasoning time, and sometimes even cannot get the optimal solution because of too large space. From the trend of traffic density, smaller traffic means smaller collision risk.

[0121] In terms of smoothness, MAPCPO algorithm has the smallest acceleration and the smallest acceleration fluctuation, and MAPPO-SC method has smaller fluctuation than MAPPO algorithm because of the comfort constraint in the reward function, but overall MAPCPO algorithm shows the best performance. MPC algorithm presents the initial ride comfort because of too many vehicles waiting and accelerating to drive off. From the trend of traffic density, smaller traffic has better smoothness, because when the traffic increases, the space increases exponentially, and it is difficult for the strategy to explore good control methods.

[0122] In terms of traffic efficiency, the MAPPO algorithm shows the best performance, because the reward function of the MAPPO method does not add the safety constraint of TTC violation, so there are more times of TTC violation, the MEL of the three kinds of reinforcement learning algorithms is less different, and the performances are basically flat, and the other two traditional control methods have too high MEL, which means that the traffic efficiency is greatly different from that of the reinforcement learning method. From the change trend of the vehicle flow, the small vehicle flow has higher traffic efficiency.

[0123] The above merely describes preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An automatic road intersection management method based on a projection-based constraint policy optimization, characterized by, Comprising the following steps: Step 1, projection constraint strategy optimization: Including six elements: policy network, model, reward function, risk function, value estimation network and risk estimation network; the policy network uses LSTM neural network to determine the next behavior of the agent according to the current state of the agent; The model is used to provide the scene for the interaction between the agent and the environment; the reward function is used to guide the agent learning, so that the agent can achieve the expected goal; the value estimation network is used to evaluate the current policy of the agent to update the policy network; The risk function is used to ensure the safety of the exploration process of the agent; the risk estimation network is used to evaluate the risk level of the current behavior of the agent to limit the update of the policy network; When the policy network is updated, the safety risk of the current policy is first judged, and the safety risk is divided into three levels: safe, relatively safe and dangerous; if the current policy is in the safe range, the risk function will not be considered when the policy is updated, and the policy will be updated in the direction of maximizing the reward; if the current policy is in the relatively safe range, the policy network will first update the reward maximization, and then project the policy on the risk constraint set to make the policy return to the safe range; if the current policy is in the dangerous range, the appropriate network update parameters are found by backtracking; Step 2, automatic intersection management based on projection constraint strategy optimization: A complex intersection scene is built as a model, and reward function and risk function are designed; the reward function takes into account the traffic efficiency, comfort, collision safety and lateral control stability of the vehicle, and the risk function takes into account the collision risk and the risk of vehicle distance less than safe distance; The strategy training process is: let the vehicle explore behavior within a predetermined time, the policy network controls the speed of all vehicles in the environment by inputting the distance of the vehicle from the target place and the current speed of the vehicle, the route of the vehicle is designed by the simulator, and if a collision occurs or all vehicles pass through, the scene is reset, the input of the policy network, the output of the policy network, the score, the risk value, the value estimation value and the risk estimation value are collected within this period of time, which are used to update the policy network, the value estimation network and the risk estimation network; The step 2 includes the following specific steps: Step 2.1, design of state space and action space: The state space is defined as , represents the distance of the intelligent driving vehicle entering the complex intersection area from the departure point, represents the driving direction of the intelligent driving vehicle, respectively indicating the right turn, straight and left turn behaviors of the intelligent driving vehicle; The calculation of the vehicle's speed is divided into two cases: the vehicle enters from a straight road and the vehicle enters from a curve; when the vehicle enters from a straight road, it is divided into three behaviors: right turn, straight and left turn, and the calculation formula of the three behaviors is: （11）； When the vehicle enters from the curve, it is also divided into three behaviors: right turn, straight and left turn, and the calculation formula is: （12）； representing intelligent driving cars at the current time step, the real-time speed, each car has only one driving direction when passing through the complex intersection, and the state space of the remaining two driving directions is filled with zero vectors; give the steering information of the strategy intelligent driving car, combined with the strategy determines the current position of the intelligent driving car at the complex intersection, and the state space includes the current speed information of each intelligent driving car; The behavior space is defined as The joint behavior space of the model only selects the future joint expected speed of all intelligent vehicles in the complex intersection area; Step 2.2, define risk function and reward function: First, define the risk function, the potential safety risks of the intelligent driving car when passing through the complex intersection include collision and side slip: Collision: （13）； wherein represents a risk value assigned when a collision occurs; , represents a risk value assigned when a collision risk exists at a time step , represents a risk value assigned when a collision risk exists at a time step , represents a logarithm of a collision risk and a safe distance violation of a smart vehicle Side slip: （14）； wherein , representing an intelligent driving car at a time step instantaneous acceleration, representing the gravitational acceleration, representing the friction coefficient of the complex intersection road surface; the risk function is defined as: （15）； Then define the reward function, safety: the reward of the safety of the intelligent driving car is divided into guiding reward and decisive reward: （16）； wherein represents a guiding reward, represents a decisive reward; Efficiency: （17）； wherein is a discount factor, represents the difference between the speed of the intelligent driving car at the time step and the expected speed of the intelligent driving car, represents the reward loss caused by time accumulation; Smoothness: （18）； wherein, representing an acceleration of the intelligent driving car at a time step ; and in summary, the reward function is defined as: （19）。 2. The method of claim 1, wherein the method is based on a projection-based constraint policy optimization. The step 1 includes the following specific steps: Step 1.1, Algorithm Definition: Formulate the projected constrained policy optimization problem as a CMG, a CMG for running an agent is defined as a tuple where represents the global state space, defined as , represents the set of joint action spaces for all agents; represents the joint reward function, used to describe the state taking joint action to the next state the immediate reward, represents the set of risk functions defined by the safety constraints of the specific environment, each agent has a risk function, the transition tuple maps to a risk cost threshold , represents the probability of transitioning from to state , represents the initial distribution of the three neural network parameters, represents the discount factor; Step 1.2, constructing strategy neural network: the input of strategy neural network is the joint state of all agents at the current time, and the output is the behavior of all agents at the next time; the strategy neural network contains three Linear layers and an LSTM layer, assuming that there are agents in the complex intersection scenario, the input of the strategy neural network is the information collected by each agent at the time step, denoted as , the state of all agents collected at each time step is a matrix, which is first changed to a matrix by inputting a Linear layer, then activated by a Relu function, and then the output dimension of the LSTM layer is changed to , the output of the LSTM is adjusted in value range to ensure that the output of the LSTM network is suitable for the real scene, and the result after the adaptive operation is sent to the Linear layer and the Relu activation operation is performed, and finally the dimension of the output result is changed to by the Linear layer and the Sigmoid activation is performed as the final output result of the strategy network; Step 1.3, constructing the value estimation and risk estimation neural network: the value estimation and risk estimation neural network judges whether the updated policy neural network has higher rewards and lower risks through the current collected trajectory information, and adopts a fully connected layer neural network model as the value estimation and risk estimation neural network; first, the state input of the network is the same as the policy network model, and the distance of each vehicle from the target location and the current speed of the vehicle are input, with a dimension of , after passing through three connection layers and performing a Relu activation operation, the final output result is obtained, and the dimension changes in turn .

3. The method of claim 2, wherein the method is based on a projection-based constraint policy optimization. In step 1.2, the update of the policy neural network is divided into three cases: KL divergence trust domain is in risk trust domain, KL divergence trust domain intersects with risk trust domain, and KL divergence trust domain has no intersection with risk trust domain; First, define a value function for the reward value. , indicating state The corresponding expected reward value, of which Dependency and Policy The trajectory Representing the trajectory The corresponding discounted reward value is defined as the behavioral value function. , indicating state With behavior The corresponding expected reward value, the advantage function is defined as follows: , indicating the state Under the premise of consistency, behavior The difference between the corresponding reward and the expected reward, if A positive value indicates that the action... The overall reward value is higher, if A negative value indicates that the action was performed. The overall reward value obtained is lower than the average reward, defined with neural network parameters. Policy Neural Network The corresponding expected return is Similarly, the value function, behavior function, and advantage function are defined for risk as follows: Relative to the risk function The expected value of the discounted risk is Then the set of feasible strategies for the CMGs model is represented as ,in Indicates in Time of the first Risk threshold of an individual agent; pass The update method is determined comprehensively, among which Representing the risk function gradient, This indicates how close the agent's risk value is to the risk threshold under the current policy; [Definition] ,but This is used to determine whether the KL divergence trust domain and the risk trust domain have an overlapping relationship. The Hessian matrix representing the KL divergence between the old and new strategies; When , there is no danger to update the policy in any direction at this time; when and , there is also no danger to update the policy in any direction at this time, these two cases are summarized as case 3, when the policy is in case 3, the trust domain policy optimization algorithm is applied to update the policy network parameters: （1）； ； According to the linearization of the objective function at the second-order approximation of the KL divergence constraint, the following update is obtained: According to the linearization of the objective function at the second-order approximation of the KL divergence constraint, the following update is obtained: （2）； ； where The gradient of the reward advantage function is denoted by, and the network update formula is obtained by solving equation (2) using a convex optimization method: （3）； When , if the maximum expected reward value update may lead to the policy entering a more dangerous region, this case is attributed to case 2; if the policy neural network is in case 2, the policy network is updated using a projected constraint policy optimization: a reward promotion step and a projection step; the reward promotion step maximizes the reward value by using the trust region policy optimization algorithm, and the projection step projects the policy network after the reward promotion step into the risk trust region; first, the advantage function is maximized in the KL divergence trust region : （4）； ； Second, by minimizing the intermediate policy The distance to the risk trust domain projects the policy into the risk trust domain: （5）； ； linearizing the KL divergence constraint at the second order approximation of the target function, resulting in a reward step update approximation: linearizing the KL divergence constraint at the second order approximation of the target function, resulting in a reward step update approximation: （6）； ； measured by the KL divergence the distance to the risk trust region, again applying the second order approximation of the KL divergence to linearize the cost constraint at （7）； ； Use convex optimization method to solve equations (6) and (7): （8）； If and In this case, the strategy is not in a safe state regardless of the direction of the update, so we classify this case as Case 1. If the strategy is in Case 1, we update the strategy neural network using the linear back-up method to minimize the risk value: （9）。 4. The method of claim 3, wherein the method is based on a projection-based constraint policy optimization. In step 1.3, the update principle of value estimation and risk estimation neural network is as follows: The value estimation and risk estimation neural networks are updated by the difference between the predicted value and the true value, denoted as: ； （10）。

Images