A method for intersection management based on intelligent connected vehicles

Abstract

The present invention relates to an intersection management method based on intelligent networked vehicles, comprising the following steps: constructing a multi-agent reinforcement learning model, where an agent is defined as each intelligent vehicle entering an intersection; setting up collaborative data communication between intelligent vehicles and networked vehicles, and defining a local state space, a local action space, and a reward function for each agent based on collaborative communication data; training and optimizing each agent through an algorithm to obtain a trained model, and finally deploying the trained model in the intelligent vehicle, guiding the operation of the intelligent vehicle through the model to achieve intersection management. Compared with the prior art, the present invention uses the computing power of the vehicle itself to replace the road computing unit, proposes a multi-agent residual attention mechanism deep deterministic strategy network algorithm to control the movement of the intelligent vehicle, and adopts the idea of ​​centralized training distributed execution combined with the residual connection of the attention network to achieve faster convergence and lower average vehicle travel delay.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic technology, and in particular to an intersection management method based on intelligent networked vehicles. Background Art

[0002] In recent years, traffic congestion has been an important issue that needs to be solved urgently. The current traffic lights have poor flexibility, and there is an upper limit to phase changes. Traffic congestion is becoming more and more serious, and intersections are a key node for congestion optimization. With the development of vehicle networking and artificial intelligence technology, it is hoped that the purpose of solving traffic congestion can be achieved through intelligent management of intersections. The intelligent management of intersections with signals is limited by the signal lights themselves, and can quickly achieve certain results, but the upper limit is low. Therefore, most research focuses on the intelligent management of intersections without signals.

[0003] In the field of intelligent transportation, there are three key technologies: low-latency Internet of Vehicles, intelligent networked vehicles, and road computing units. By combining these three technologies, each traffic problem at this stage can be managed intelligently to a certain extent, and intersections are no exception. Internet of Vehicles technology is a basic technology, similar to today's communication networks. Intelligent networked vehicles are the development direction of today's vehicles. Vehicles gradually have computing and communication capabilities, and are divided into two types: smart cars and networked vehicles. Figure 1 As shown, smart cars are generally considered to be vehicles that can perform intelligent driving, and therefore have a decision-making module that can be directly connected to the control module. Connected cars are vehicles that can be connected to the Internet of Vehicles network but are still driven by a driver.

[0004] Road computing units are generally composed of intersection computing units and computing servers. Compared with smart cars and connected cars, road computing units are expensive and costly. It is unrealistic to set up road computing units at every intersection. Therefore, for many small intersections, how to build a distributed strategy and use the computing power of the vehicle itself to replace the expensive road computing units to achieve the goal of intelligent intersection management is an urgent problem that needs to be solved.

[0005] At the same time, although the distributed solution can significantly reduce the cost of building road computing units, it is very difficult to converge and optimize the algorithm. Because of the lack of a central node, the entire intersection does not have a node that receives global vehicle information in most cases, resulting in each distributed node (vehicle) not receiving comprehensive intersection information, which will cause environmental instability and difficulty in convergence and optimization during algorithm optimization. Summary of the invention

[0006] The purpose of the present invention is to provide an intersection management method based on an intelligent connected vehicle in order to overcome the defects of the above-mentioned prior art.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A method for managing an intersection based on an intelligent connected vehicle comprises the following steps:

[0009] Build a multi-agent reinforcement learning model, where an agent is defined as each smart car entering the intersection;

[0010] Set up cooperative data communication between smart cars and connected cars, and define local state space, local action space and reward function for each intelligent agent based on cooperative communication data;

[0011] Each intelligent agent is trained and optimized through the RA-MADDPG algorithm to obtain a trained model. Finally, the trained model is deployed in the smart car, and the model is used to guide the operation of the smart car to achieve intersection management.

[0012] Furthermore, the RA-MADDPG algorithm is a multi-agent residual attention mechanism deep deterministic policy network algorithm, which combines the idea of ​​centralized training distributed execution with the residual connection of the attention network. The algorithm construction includes an algorithm modeling step and an algorithm training and updating step. The algorithm modeling step includes:

[0013] Construct the global state space;

[0014] Constructing a local observation space;

[0015] Constructing action space;

[0016] Construct the reward function.

[0017] Furthermore, constructing the global state space includes:

[0018] Obtaining the operation information of each vehicle at the intersection, and representing the operation information of each vehicle with numbers, wherein the operation information includes position coordinates, speed magnitude and direction;

[0019] The running information of each vehicle is obtained in digital form, recorded as s i , the global state space is expressed as: N represents the total number of vehicles in the intersection.

[0020] Furthermore, the action space is defined as the throttle, brake and steering wheel angle of the smart car.

[0021] Furthermore, the reward function is defined as:

[0022]

[0023] v is the current speed of the vehicle, v max is the maximum speed of the vehicle, v min is the minimum speed of the vehicle, collision indicates whether the vehicle collides, arrived indicates whether the vehicle arrives at the destination, interact indicates whether information is exchanged between vehicles, and η1, η2, η3, and η4 represent the coefficients respectively.

[0024] Furthermore, each agent has an independent policy network, Q-value network, and residual attention network;

[0025] The input of the policy network is the local observation state of the vehicle, that is, the information of nearby vehicles that can be obtained according to the vehicle's own capabilities, which is input into the residual attention network to obtain the corresponding vehicle weight information, and the residual connection is performed with the original nearby vehicle information, and the spliced ​​results are used as the input of the policy network together;

[0026] The output of the policy network is the action space of each intelligent car;

[0027] The Q-value network is a centralized critic architecture. The input of the Q-value network is the global observation state and global action of the vehicle. The output of the Q-value network is the Q-value of the current strategy, which represents the score of the strategy currently adopted by the smart car.

[0028] Furthermore, Gaussian white noise is added to the output of the policy network to facilitate algorithm convergence.

[0029] Furthermore, the algorithm training and updating steps include updating the distributed policy network and performing gradient updates on the global Q-value network. At the beginning of training, the algorithm adopts a random strategy for action output and scoring, and obtains a series of actions A, states X, the next state X' after taking the action, and the immediate reward R obtained at the same time. The four types of data are stored in the form of quadruples. When the size of the stored data meets the batch size for random gradient descent, the network is updated.

[0030] Furthermore, for the Q network, the objective function is expressed as minimizing the mean square error between the target value and the estimated value, expressed as:

[0031]

[0032] represents the output of the vehicle's global observation state after passing through the residual attention network, and y represents the target value, which is:

[0033]

[0034] μ represents the policy network, γ represents the discount factor, and O k represents the output of the k-th vehicle local observation state after the residual attention network; N e represents the e-th smart car, k = 1, 2…N e , r is the immediate reward obtained by the vehicle after selecting the action.

[0035] Substitute the next vehicle local observation state value and the vehicle action value output by the policy network into the Q value network to obtain the next state action value Q', and add it to the immediate reward to get the target value.

[0036] Furthermore, for the estimation strategy network, its output needs to make the output of the Q-value network as large as possible. The strategy gradient is expressed as:

[0037]

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The present invention constructs a multi-agent reinforcement learning model, in which the agent is defined as each smart car entering the intersection. The computing power of the vehicle itself is used to replace the expensive road computing unit to achieve the goal of intelligent intersection management, which can effectively save traffic control costs.

[0040] 2. The present invention proposes the RA-MADDPG algorithm to train and optimize each intelligent agent, deploy the trained model in the smart car, and guide the operation of the smart car through the model to achieve the purpose of distributed intelligent intersection management and fast and safe passage of vehicles. The control method is simple and effective.

[0041] 3. The RA-MADDPG algorithm proposed in this invention adopts the strategy of centralized training and distributed execution. Each agent has an independent strategy network, Q value network and residual attention network. The innovation of the algorithm lies in the combination of the idea of ​​centralized training and distributed execution and the residual connection of the attention network, which accelerates the convergence of the algorithm and achieves better performance. Compared with the mainstream distributed algorithm, the algorithm proposed in this invention can converge faster and achieve lower average vehicle delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the internal structure of the on-board unit of an intelligent connected vehicle;

[0043] Figure 2 A schematic diagram of a distributed automated intersection model in an embodiment of the present invention;

[0044] Figure 3 An example of vehicle relationship in an embodiment of the present invention is given;

[0045] Figure 4 The overall flow chart of the distributed solution proposed by the present invention;

[0046] Figure 5 The algorithm update flow chart proposed by the present invention;

[0047] Figure 6 A schematic diagram of the structure of a distributed policy network in an embodiment of the present invention;

[0048] Figure 7 Schematic diagram of the structure of the Q value network in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0050] In view of the defects in the prior art, this application proposes an intersection management method based on intelligent connected vehicles, which adopts a distributed strategy, such as Figure 2 As shown, the vehicle's own computing power is used to replace the expensive road computing unit to achieve the goal of intelligent intersection management.

[0051] This application is inspired by the idea of ​​distributed execution of centralized training. By adopting the residual attention mechanism, auxiliary features that are equally useful as global information are extracted, such as the potential relationship between vehicles. Figure 3 As shown, in the case of cooperation, there are no potential conflict points, and vehicles can pass directly and quickly. In the case of competition, there are potential conflict points, and conflict avoidance is required. These features are assigned to each vehicle in the form of weights. A large weight indicates that it may be a competition situation and needs attention. A small weight indicates that it may be a cooperation situation. In addition, through residual connections, these information can be used to assist the intelligent agent in strategy optimization based on the original observation space. Experimental evaluation shows that the method proposed in the present invention can converge faster and achieve lower average vehicle passage delay than mainstream distributed algorithms.

[0052] like Figures 4 to 5 As shown in the figure, it is the overall flow chart of the distributed solution proposed in the present invention. First, the problem is modeled and a multi-agent reinforcement learning model is constructed. The agent is defined as each smart car entering the intersection. By designing the collaborative communication data between the smart car and the connected car, the local state space, local action space and reward function are defined. Each agent is trained and optimized through the algorithm. Finally, the trained model is deployed in the smart car to achieve the purpose of distributed intelligent intersection management and fast and safe passage of vehicles.

[0053] The algorithm update adopts a centralized training and distributed execution strategy. Each intelligent agent has an independent strategy network, Q-value network and residual attention network. The innovation of the algorithm lies in the combination of the idea of ​​centralized training and distributed execution and the residual connection of the attention network to accelerate the convergence of the algorithm and achieve better performance.

[0054] The detailed steps of the algorithm include:

[0055] (a) Algorithm Modeling

[0056] Global state space: The position coordinates, speed, direction and other information of each vehicle are expressed as numbers, expressed as s. The global state space can be expressed as: N represents the total number of vehicles in the intersection.

[0057] Local observation space: In distributed scenarios, the global state space does exist, but in most cases no node can obtain this information. Each distributed node vehicle can only obtain partial state information, which is called the local observation space. The size of the information is limited by the communication capability of the vehicle itself. The stronger the communication capability of the vehicle, the larger the local observation space.

[0058] Action space: the accelerator, brake and steering wheel angle of the smart car.

[0059] Reward function: In a distributed scenario, each agent has its own corresponding reward function. For a certain agent, its goal is to maximize its own reward function, but for the entire intersection, it is necessary to maximize the reward function of all vehicles. Therefore, not only safety but also the interaction between vehicles needs to be considered. We hope that vehicles can interact as much as possible to obtain relatively global information. The function is defined as:

[0060]

[0061] (b) Algorithm training and updating

[0062] The training process of the entire algorithm can be divided into two main processes: updating the distributed policy network and updating the global Q value network. The structures of the two networks are as follows: Figure 6 and Figure 7 shown.

[0063] In the policy network, the input of the network is the local observation state of the vehicle, that is, the information of nearby vehicles that can be obtained according to the vehicle's own capabilities, and the information is input into the attention network to obtain the corresponding vehicle weight information, and the residual connection is made with the original nearby vehicle information (residual connection means splicing the two together), and the spliced ​​result is used as the input of the policy network. For the policy network, there are target policy network and estimation policy network, and the network structure of the two is exactly the same. The difference is that the estimation policy network will update the parameters every time, while the target policy network will be updated after a period of time, so as to ensure the reuse of data in the playback experience pool.

[0064] Each network consists of three fully connected layers with dimensions of (1024, 1024), (512, 512), and (256, 256). The output is the action space of each smart car, and a Gaussian white noise is added to facilitate the convergence of the algorithm.

[0065] In the Q-value network, it is essentially a centralized critic architecture. The input of the network is the global observation state and global action of the vehicle. Similar to the strategy network structure, it has residual connections of the attention network, two Q-value networks, etc. The difference is that the output of the Q-value network is the Q-value of the current strategy, which represents the score of the strategy currently adopted by the smart car.

[0066] When the algorithm is updated, the gradients of the two networks are updated separately. The optimization goal can be briefly described as the commentator Q-value network scores more accurately, and the policy network can make decisions with higher scores. At the beginning of training, the algorithm uses a random strategy to output and score actions, and obtains a series of actions A, states X, the next state X' after taking the action, and the immediate reward R obtained at the same time. The four data are stored in a quadruple. When the size of the stored data meets the batch size for random gradient descent, the network is updated.

[0067] For the estimated Q network, the objective function is expressed as minimizing the mean square error between the target value and the estimated value, expressed as:

[0068]

[0069] represents the output of the vehicle's global observation state after passing through the residual attention network, and y represents the target value, which can usually be written as:

[0070]

[0071] μ represents the policy network, γ represents the discount factor, and O kRepresents the output of the k-th vehicle local observation state after passing through the residual attention network; substituting the next vehicle local observation state value and the action value output by the target policy network into the Q value network can obtain the next state action value Q', and adding it to the immediate reward can get the target value.

[0072] For the estimation strategy network, it is necessary to make its output as large as possible so that the output of the Q value network is as large as possible. The strategy gradient can be expressed as:

[0073]

[0074] Through the above process, the model is trained and optimized, which can not only solve the problem of environmental instability, but also use the attention network to extract high-level features about the intersection scene, assist the algorithm to accelerate convergence and achieve better performance. After the training of the entire solution is completed, each smart car will have an independent estimation strategy network.

[0075] (c) Algorithm deployment

[0076] By deploying the strategy network model obtained through the above training into the vehicle computing unit of each smart car, the task of distributed automatic intersection management can be completed.

[0077] Table 1 Experimental results

[0078]

[0079] As shown in Table 1, the experimental results obtained in this embodiment, RA-MADDPG represents the algorithm proposed by the present invention, that is, the multi-agent residual attention mechanism deep deterministic policy network algorithm. Others are more mainstream distributed reinforcement learning algorithms, including: progressive policy gradient optimization algorithm PPO, attention mechanism progressive policy optimization algorithm SA-PPO, deep Q network algorithm DQN, actor commentator algorithm A2C, deep deterministic policy network algorithm MADDPG. It can be seen that the use of the algorithm proposed by the present invention can achieve a lower collision rate, a higher completion rate and the lowest average vehicle delay.

[0080] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for managing intersections based on intelligent connected vehicles, characterized in that: The following steps are involved: Build a multi-agent reinforcement learning model, where an agent is defined as each smart car entering the intersection; Set up cooperative data communication between smart cars and connected cars, and define local state space, local action space and reward function for each intelligent agent based on cooperative communication data; Each intelligent agent is trained and optimized through the RA-MADDPG algorithm to obtain a trained model, which is finally deployed in the smart car to guide the operation of the smart car to achieve intersection management; The reward function is defined as: v is the current speed of the vehicle, v max is the maximum speed of the vehicle, v min is the minimum speed of the vehicle, collision indicates whether the vehicle collides, arrived indicates whether the vehicle arrives at the destination, interact indicates whether information is exchanged between vehicles, η1, η2, η3, η4 represent the coefficients respectively; Each agent has an independent policy network, Q-value network and residual attention network; The input of the policy network is the local observation state of the vehicle, that is, the information of nearby vehicles that can be obtained according to the vehicle's own capabilities, which is input into the residual attention network to obtain the corresponding vehicle weight information, and the residual connection is performed with the original nearby vehicle information, and the spliced ​​results are used as the input of the policy network together; The output of the policy network is the action space of each intelligent car; The Q-value network is a centralized critic architecture. The input of the Q-value network is the global observation state and global action of the vehicle. The output of the Q-value network is the Q-value of the current strategy, which represents the score of the strategy currently adopted by the smart car. The algorithm training and updating steps include updating the distributed policy network and performing gradient updates on the global Q-value network. At the beginning of training, the algorithm adopts a random strategy for action output and scoring, and obtains a series of actions A, states X, the next state X' after taking the action, and the immediate reward R obtained at the same time. The four types of data are stored in the form of quadruples. When the size of the stored data meets the batch size for random gradient descent, the network is updated.

2. The method for managing intersections based on intelligent connected vehicles according to claim 1, characterized in that: The RA-MADDPG algorithm is a multi-agent residual attention mechanism deep deterministic policy network algorithm, which combines the idea of ​​centralized training distributed execution with the residual connection of the attention network. The algorithm construction includes algorithm modeling steps and algorithm training and updating steps. The algorithm modeling steps include: Construct the global state space; Constructing a local observation space; Constructing action space; Construct the reward function.

3. The method for managing intersections based on intelligent networked vehicles according to claim 2, characterized in that: The construction of the global state space includes: Obtaining the operation information of each vehicle at the intersection, and representing the operation information of each vehicle with numbers, wherein the operation information includes position coordinates, speed magnitude and direction; The running information of each vehicle is obtained in digital form, recorded as s i , the global state space is expressed as: N represents the total number of vehicles in the intersection.

4. The method for managing intersections based on intelligent connected vehicles according to claim 1, characterized in that: The action space is defined as the throttle, brake and steering wheel angle of the smart car.

5. The method for managing intersections based on intelligent networked vehicles according to claim 1, characterized in that: The output of the policy network is added with Gaussian white noise to facilitate algorithm convergence.

6. The method for managing intersections based on intelligent networked vehicles according to claim 1, characterized in that: For the Q network, the objective function is expressed as minimizing the mean square error between the target value and the estimated value, expressed as: represents the output of the vehicle's global observation state after passing through the residual attention network, and y represents the target value, which is: μ represents the policy network, γ represents the discount factor, and O k represents the output of the k-th vehicle local observation state after the residual attention network; N e represents the e-th smart car, k = 1, 2…N e , r is the immediate reward obtained by the vehicle after selecting the action; Substitute the next vehicle local observation state value and the vehicle action value output by the policy network into the Q value network to obtain the next state action value Q', and add it to the immediate reward to get the target value.

7. The method for managing intersections based on intelligent networked vehicles according to claim 6, characterized in that: For the estimated policy network, its output needs to make the output of the Q value network as large as possible. The policy gradient is expressed as:

Images