Intelligent connected vehicle intersection coordinated driving control method and system and storage medium

Abstract

The application relates to an intelligent networked vehicle intersection coordinated driving control method, a system and a computer readable storage medium. The intelligent networked vehicle intersection coordinated driving control method comprises the following steps: S1, a directed graph of a road network topology is established by using a graph theory method; S2, a two-dimensional grid coding method is used to extract the state characteristics of an intelligent networked vehicle, and the state of the intelligent networked vehicle is converted into a binary vector; S3, a first reinforcement learning submodel for stimulating a vehicle to quickly reach a terminal point is constructed; S4, a second reinforcement learning submodel for avoiding vehicle collision is constructed; and S5, the first reinforcement learning submodel and the second reinforcement learning submodel are fused to form a fusion model, and an optimal behavior maximizing the global return in the environment is calculated. The application can avoid unnecessary vehicle parking, maximize the global return of the intersection, and has strong generalization ability.

Description

Technical Field

[0001] This application relates to intelligent traffic control technology, and more specifically, to a method, system, and computer-readable storage medium for coordinated driving control of intelligent connected vehicles at intersections. Background Technology

[0002] The most common strategy for managing urban road intersections is to use traffic lights. Traffic lights use different colors to indicate the direction of travel and prohibited directions. Meanwhile, vehicle-level driving decisions, such as acceleration / deceleration and direction changes, are made by human drivers or independent vehicle controllers. However, traffic light control can cause unnecessary vehicle stops, thus failing to fully utilize the time and space margins at intersections.

[0003] To fully utilize the spatial and temporal margins at intersections, a reservation-based autonomous intersection control method has been proposed. Through wireless communication, vehicles approaching the intersection send a passage request to the central controller. Upon receiving the request, the central controller checks whether the requested space is occupied within the requested passage time. If not, the requesting vehicle is allowed to pass; otherwise, passage is blocked. Compared to traffic lights, the reservation-based method determines the vehicle passage order with more flexible strategies, such as a first-come, first-served strategy or an optimization-based strategy. However, reservation-based control relies on a "vehicle request - central controller response" model. This model can lead to vehicles frequently sending requests without prior permission, resulting in unnecessary consumption of communication resources.

[0004] To maximize the temporal and spatial capacity of intersections, the latest autonomous intersection control technologies have moved beyond traffic light-based and reservation-based road-level control frameworks. Instead, they start at the vehicle level, using vehicle driving decision variables as the decision variables for intersection control, and employing optimization control and learning algorithms to achieve optimal overall intersection traffic efficiency. These algorithms can be categorized into centralized control, distributed control, and decentralized control. In a centralized control framework, the intersection controller coordinates the driving decisions of all vehicles approaching the intersection, resulting in an exponential expansion of the decision space with the number of vehicles. Therefore, while a centralized framework theoretically helps in obtaining a globally optimal solution, it is often hampered by computational complexity. In existing technologies, distributed and decentralized algorithms have become more reasonable choices. A decentralized collaborative learning-based method for autonomous intersection management has been proposed (see Wu Y, Chen H, Zhu F. DCL-AIM: Decentralized coordination learning of autonomous intersection management for connected and automated vehicles[J]. Transportation Research Part C: Emerging Technologies, 2019, 103: 246-260). This method employs reinforcement learning, dividing vehicle states into independent and collaborative states. It uses tabular Q-learning to estimate behavioral values ​​in both independent and collaborative states, and selects the optimal driving decision based on these behavioral values. However, compared to function estimation, the tabular method has lower generalization ability and insufficient utilization of samples. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a method, system and computer-readable storage medium for coordinated driving control of intelligent connected vehicles at intersections, which aims to improve traffic efficiency at road intersections and avoid vehicle collisions, in view of the above-mentioned deficiencies of the prior art.

[0006] To address its technical problem, this application proposes a method for coordinated driving control of intelligent connected vehicles at intersections, the method comprising the following steps:

[0007] S1. A directed graph of the road network topology is established using graph theory. Nodes in the directed graph represent the starting point, ending point, and intersection of the driving trajectory. The lines between nodes represent the driving trajectory. The length of the line represents the length of the driving trajectory. The direction of the line represents the driving direction. Each intelligent connected vehicle is defined as an intelligent agent. Each driving trajectory represents a driving intention.

[0008] S2. Use a two-dimensional grid coding method to extract the state features of intelligent connected vehicles and convert the state of intelligent connected vehicles into binary vectors.

[0009] S3. Construct the first reinforcement learning sub-model to incentivize the vehicle to quickly reach the destination. Define the objective function of the first reinforcement learning sub-model as follows:

[0010]

[0011] Where i = 1, 2, ..., N(t), N(t) is the number of agents, which is a function of time, and s i S represents the state of a single agent i, and S represents the set of states of the agents. i It is the acceleration of agent i, Q π (s i ,a i Let ) be a value function whose first row is , and its mathematical form is .

[0012]

[0013] Where E represents expectation, π represents policy, γ is the reward decay factor, p is a positive integer, and γ p Represents γ raised to the power of p, single-step reward function Defined as

[0014]

[0015] S4. Construct a second reinforcement learning sub-model to avoid vehicle collisions, and define the objective function of the second reinforcement learning sub-model as follows:

[0016]

[0017] Where j = 1, 2, ..., N(t), k = 1, 2, ..., N(t), and j ≠ k; a j,k =(a j ,a k ), a j and a k These are the accelerations of agents j and k, respectively; s j,k Yes, the states of agents j and k, where S represents the set of states of the agents; Q π (s j,k ,a j,k The second row is a value function, and its mathematical form is:

[0018]

[0019] Where E represents expectation, π represents policy, γ is the reward decay factor, p is a positive integer, and γ p Represents γ raised to the power of p, single-step reward function Defined as

[0020]

[0021] S5. The first reinforcement learning sub-model and the second reinforcement learning sub-model are fused to form a fusion model, and the global objective function of the fusion model is defined as follows:

[0022]

[0023] Among them, Q i =Q π (s i ,a i ), Q j,k =Q π (s j,k ,a j,k M1(·) and M2(·) are user-defined functions. Let η be the vector consisting of the behaviors of all vehicles in the environment at time t, and let η be the collision balance weight. The optimal behavior that maximizes the global reward in the environment is calculated. for

[0024]

[0025] In one embodiment of the intelligent connected vehicle intersection coordinated driving control method according to the first aspect of this application, step S2 further includes:

[0026] S21. Map each driving trajectory to a two-dimensional velocity-position plane, using N... t N two-dimensional grids, each with N v ×N p Given N subgrids, where each subgrid corresponds to an element in the feature vector, then each driving trajectory corresponds to an N... t ×N v ×N p 3D eigenvectors;

[0027] S22, Regarding N in the environment d The driving trajectory, Let represent the feature vector of agent i operating on the driving trajectory m at time t. express The nth element in the definition

[0028]

[0029] Where M = 1, 2, ..., N d n = 1, 2, ..., N t ×N v ×Np ;

[0030] S23, N d The feature vectors corresponding to the driving trajectories are concatenated to obtain the feature vector of agent i.

[0031]

[0032] in, It is an N d ×N t ×N v ×Np-dimensional vector.

[0033] In one embodiment of the intelligent connected vehicle intersection coordinated driving control method according to the first aspect of this application, the first reinforcement learning sub-model defines a set of variables describing the state of a single vehicle as a single state, the mathematical form of which is:

[0034]

[0035] in, This represents the state of agent i at time t. and Let i represent the driving intention, speed, and position of the agent, i = 1, 2, ..., N(t), where N(t) is the number of agents and is a function of time.

[0036] In one embodiment of the intelligent connected vehicle intersection coordinated driving control method according to the first aspect of this application, the first reinforcement learning sub-model updates its parameters using stochastic gradient descent.

[0037]

[0038] in,

[0039]

[0040]

[0041]

[0042] in, It refers to the behavioral values ​​of a single agent that quickly reaches the destination. and These are the outputs of the online Q-network and its target network, θ. t and These are the parameters of the online Q-network and its target network, respectively, where α is the learning rate and γ is the reward decay factor.

[0043] In one embodiment of the intelligent connected vehicle intersection coordinated driving control method according to the first aspect of this application, the second reinforcement learning sub-model defines a set of variables describing a pair of vehicle states as a dual state, the mathematical form of which is:

[0044]

[0045] in, and Let j and k be single states of vehicles j and k, respectively, j = 1, 2, ..., N(t), k = 1, 2, ..., N(t), and j ≠ k.

[0046] In one embodiment of the intelligent connected vehicle intersection coordinated driving control method according to the first aspect of this application, the second reinforcement learning sub-model updates parameters using stochastic gradient descent.

[0047]

[0048] in,

[0049]

[0050]

[0051]

[0052] in, These are the behavioral values ​​of the two agents targeting collision avoidance. and These are the output values ​​of the online three-switch Q-network and its target network, ω. t and These are the parameters of the online three-switch Q-network and its target network, respectively. and Let each represent a pair of state feature vectors of agents j and k. Let represent the behavioral feature vector of a pair of agents j and k, where β is the learning rate and γ is the reward decay factor.

[0053] In one embodiment of the intelligent connected vehicle intersection coordinated driving control method according to the first aspect of this application, the user-defined functions M1(·) and M2(·) in the global objective function of the fusion model are M1(x) = x and M2(x) = x, respectively. 2 .

[0054] To address its technical problem, this application proposes a second aspect of an intelligent connected vehicle intersection coordinated driving control system, the system comprising:

[0055] The directed graph construction module is used to construct a directed graph of the road network topology using graph theory methods. The nodes in the directed graph represent the starting point, ending point and intersection of the driving trajectory. The lines between the nodes represent the driving trajectory. The length of the line represents the length of the driving trajectory. The direction of the line represents the driving direction. Each intelligent connected vehicle is defined as an intelligent agent. Each driving trajectory represents a driving intention.

[0056] The feature extraction module is used to extract the state features of intelligent connected vehicles using a two-dimensional grid coding method, and convert the state of intelligent connected vehicles into binary vectors.

[0057] The speed incentive module is used to construct the first reinforcement learning sub-model to incentivize the vehicle to quickly reach the destination. The objective function of the first reinforcement learning sub-model is defined as follows:

[0058]

[0059] Where i = 1, 2, ..., N(t), N(t) is the number of agents, which is a function of time, and s i S represents the state of a single agent i, and S represents the set of states of the agents. i It is the acceleration of agent i, Q π (s i ,a i Let ) be a value function whose first row is , and its mathematical form is .

[0060]

[0061] Where E represents expectation, π represents policy, γ is the reward decay factor, p is a positive integer, and γ p Represents γ raised to the power of p, single-step reward function Defined as

[0062]

[0063] The collision avoidance module is used to construct a second reinforcement learning sub-model to avoid vehicle collisions. The objective function of the second reinforcement learning sub-model is defined as follows:

[0064]

[0065] Where j = 1, 2, ..., n(t), k = 1, 2, ..., N(t), and j ≠ k; a j,k =(a j ,a k ), a j and a k These are the accelerations of agents j and k, respectively; s j,k Yes, the states of agents j and k, where S represents the set of states of the agents; Q π (sj,k ,a j,k The second row is a value function, and its mathematical form is:

[0066]

[0067] Where E represents expectation, π represents policy, γ is the reward decay factor, p is a positive integer, and γ p Represents γ raised to the power of p, single-step reward function Defined as

[0068]

[0069] The fusion module is used to fuse the first reinforcement learning sub-model and the second reinforcement learning sub-model to form a fusion model, and defines the global objective function of the fusion model as follows:

[0070]

[0071] Among them, Q i =Q π (s i ,a i ), Q j,k =Q π (s j,k ,a j,k M1(·) and M2(·) are user-defined functions. Let η be the vector consisting of the behaviors of all vehicles in the environment at time t, and let η be the collision balance weight. The optimal behavior that maximizes the global reward in the environment is calculated. for

[0072]

[0073] To address its technical problem, this application proposes a third aspect of an intelligent connected vehicle intersection coordinated driving control system, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the intelligent connected vehicle intersection coordinated driving control method as described above.

[0074] In order to solve its technical problem, this application proposes a computer-readable storage medium in a fourth aspect, which stores a computer program that, when executed by a processor, implements the steps of the intelligent connected vehicle intersection coordinated driving control method as described above.

[0075] The intelligent connected vehicle intersection coordinated driving control method, system, and computer-readable storage medium implementing this application have the following beneficial effects: The intelligent connected vehicle intersection coordinated driving control method according to the embodiments of this application uses vehicle-level control input as the decision variable for intersection management and establishes a reinforcement learning sub-model to evaluate the potential collision probability between vehicles, avoiding unnecessary stopping of vehicles in scenarios where there is no possibility of collision; the method uses a reinforcement learning algorithm to uniformly determine vehicle control input, avoiding the ineffective consumption of communication resources; the method uses a neural network to estimate the vehicle behavior value function, which, compared with existing methods, can estimate the behavior value function corresponding to uncollected samples using a smaller training sample, and has a stronger generalization ability. Attached Figure Description

[0076] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0077] Figure 1 This is a flowchart of an embodiment of the intelligent connected vehicle intersection coordinated driving control method of this application;

[0078] Figure 2 This is a schematic diagram of a directed graph of the road network topology in one embodiment of this application;

[0079] Figure 3 This is a schematic diagram illustrating the use of grid coding to extract state features of intelligent connected vehicles in one embodiment of this application;

[0080] Figure 4 This is the first reinforcement learning sub-model in one embodiment of this application. Network structure diagram;

[0081] Figure 5 This is the second reinforcement learning sub-model in one embodiment of this application. Network structure diagram;

[0082] Figure 6 This is the fusion model C in one embodiment of this application. n The implementation principle diagram;

[0083] Figure 7 yes Figure 6 The fusion model C shown n The implementation flowchart;

[0084] Figure 8 This is a logical structure diagram of the intelligent connected vehicle intersection coordinated driving control system of the same embodiment of this application;

[0085] Figure 9 This is a logical structure diagram of an intelligent connected vehicle intersection coordinated driving control system according to another embodiment of this application. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Furthermore, the embodiments and features described herein can be combined with each other unless otherwise specified.

[0087] This application proposes a reinforcement learning-based method for coordinated driving control of intelligent connected vehicles at intersections. The specific application scenario is a vehicle-road cooperative environment where a central controller is installed at the road intersection. This controller can receive information such as the speed, position, and driving intention of each intelligent connected vehicle approaching the intersection in real time. Furthermore, it can coordinate the driving strategies of all vehicles through algorithms, thereby improving the communication efficiency of the road intersection and avoiding collisions.

[0088] Figure 1 A flowchart of an intelligent connected vehicle intersection coordinated driving control method 100 according to an embodiment of this application is shown. Figure 1 As shown, the intelligent connected vehicle intersection coordinated driving control method 100 includes the following steps:

[0089] In step S1, a directed graph of the road network topology is established using graph theory. Nodes in the directed graph represent the starting point, ending point, and intersection of the driving trajectory. The lines between nodes represent the driving trajectory, the length of the line represents the length of the driving trajectory, and the direction of the line represents the driving direction. Each intelligent connected vehicle is defined as an intelligent agent, and each driving trajectory represents a driving intention.

[0090] In step S2, the state features of the intelligent connected vehicle are extracted using a two-dimensional grid coding method, and the state of the intelligent connected vehicle is converted into a binary vector.

[0091] In step S3, a first reinforcement learning sub-model is constructed to incentivize the vehicle to quickly reach the destination;

[0092] In step S4, a second reinforcement learning sub-model for avoiding vehicle collisions is constructed;

[0093] In step S5, the first reinforcement learning sub-model and the second reinforcement learning sub-model are fused to form a fusion model, and the optimal behavior that maximizes the global reward in the environment is calculated.

[0094] According to the above embodiments of this application, the intelligent connected vehicle intersection coordinated driving control method 100 first establishes a Markov model for intelligent connected vehicle intersection coordinated driving in step S1. For example... Figure 2As shown, step S1 uses graph theory to establish a mathematical model of the road network topology. The road network topology is described as a directed graph, where nodes represent the start, end, and intersection points of driving trajectories, lines between nodes represent driving trajectories, the length of the lines represents the length of the trajectories, and the direction of the lines represents the direction of travel. In the directed graph of the road network, each vehicle is defined as an agent, and each driving trajectory represents a driving intention. Each agent enters from the start of a driving trajectory, with the entry time and initial speed randomly generated, and can only run along the designated driving trajectory. When an agent reaches the end of a driving trajectory or collides with another agent, that agent is considered to have left the environment. The number of agents in the environment changes over time, and the states of all agents, including driving intentions, positions, and speeds, are shared data in real time. Since each agent can only run according to a predetermined driving trajectory, the control input is a one-dimensional variable.

[0095] In step S2 of this method 100, a two-dimensional grid coding method is used to extract the state features of the intelligent connected vehicle, converting the state of the intelligent connected vehicle into a binary vector. Specifically, as follows... Figure 3 As shown, each driving trajectory corresponds to a two-dimensional velocity-position plane. For each velocity-position plane, a two-dimensional grid with different offsets is used for segmentation. A large grid contains several sub-grids, each sub-grid corresponding to an element in the feature vector. If the vehicle appears in a sub-grid, the element corresponding to that sub-grid is 1; otherwise, it is 0. Therefore, step S2 further includes:

[0096] In step S21, each driving trajectory is mapped to a two-dimensional velocity-position plane, using N t N two-dimensional grids, each with N v ×N p Given N subgrids, where each subgrid corresponds to an element in the feature vector, then each driving trajectory corresponds to an N... t ×N v ×N p 3D eigenvectors.

[0097] Then in step S22, for N in the environment d The driving trajectory, Let represent the feature vector of agent i operating on the driving trajectory m at time t. express The nth element in the definition

[0098]

[0099] Where m = 1, 2, ..., N d n = 1, 2, ..., Nt ×N v ×N p This application does not encode the status of all vehicles simultaneously, but instead encodes each vehicle independently.

[0100] In the final step S23, N d The feature vectors corresponding to the driving trajectories are concatenated to obtain the feature vector of agent i.

[0101]

[0102] in, It is an N d ×N t ×N v ×N p A dimensional vector. It's worth noting that if agent i does not appear on the driving trajectory m, all subgrids corresponding to the driving trajectory m are empty, and in this case...

[0103] To incentivize vehicles to quickly reach their destination, the intelligent connected vehicle intersection coordinated driving control method 100 according to the above embodiments of this application constructs a first reinforcement learning sub-model in step S3 to incentivize vehicles to quickly reach their destination. First reinforcement learning sub-model A single state is defined as a set of variables describing the state of a single vehicle, and its mathematical form is:

[0104]

[0105] in, This represents the state of agent i at time t. and Let i represent the driving intention, speed, and position of agent i, respectively, i = 1, 2, ..., N(t), where N(t) is the number of agents and is a function of time.

[0106] Define the first reinforcement learning sub-model The objective function is

[0107]

[0108] Where i = 1, 2, ..., N(t), s i S represents the state of a single agent i, and S represents the set of states of the agents. i The behavior of agent i is defined as the acceleration of agent i, Q. π (s i ,a i Let ) be a value function whose first row is , and its mathematical form is .

[0109]

[0110] Where E represents expectation, π represents policy, γ is the reward decay factor, p is a positive integer, and γ p Represents γ raised to the power of p, single-step reward function Defined as

[0111]

[0112] For a single agent, if it reaches the destination, it receives a reward of 0; otherwise, it receives a reward of -1 for each step it takes. The model encourages the agent to reach the endpoint as quickly as possible. For example... Figure 4 As shown, The model employs a standard Q-network structure. For the goal of quickly reaching the finish line, The model estimates the first behavior value function of a single agent. The model takes single-state features as input and outputs the value of each feasible action in the action space. For example, if an agent's action space is A... 1 = {+1, 0, -1}, the output of the Q-network is the value of +1, 0, and -1. In the hidden layers, h represents the activation function.

[0113] According to a specific embodiment of this application, the first reinforcement learning sub-model Update parameters using stochastic gradient descent.

[0114]

[0115] in,

[0116]

[0117]

[0118]

[0119] in, It refers to the behavioral values ​​of a single agent that quickly reaches the destination. and These are the outputs of the online Q-network and its target network, θ. t and These are the parameters of the online Q-network and its target network, respectively, where α is the learning rate and γ is the reward decay factor. All agents in the environment share this first reinforcement learning sub-model.

[0120] To avoid vehicle collisions, the intelligent connected vehicle intersection coordinated driving control method 100 according to the above embodiments of this application constructs a second reinforcement learning sub-model for avoiding vehicle collisions in step S4. Second reinforcement learning sub-model A set of variables describing a pair of vehicle states is defined as a two-state system, and its mathematical form is:

[0121]

[0122] in, and Let j and k be single states of vehicles j and k, respectively, j = 1, 2, ..., N(t), k = 1, 2, ..., N(t), and j ≠ k.

[0123] Second reinforcement learning sub-model In this context, the objective function is defined as follows:

[0124]

[0125] Where j = 1, 2, ..., N(t), k = 1, 2, ..., N(t), and j ≠ k; a j,k =(a j ,a k ), a j and a k These are the behaviors of agents j and k, respectively, defined as the accelerations of agents j and k; s j,k Yes, the states of agents j and k, where S represents the set of states of the agents; Q π (s j,k ,a j,k The second row is a value function, and its mathematical form is:

[0126]

[0127] Where E represents expectation, π represents policy, γ is the reward decay factor, p is a positive integer, and γ p Represents γ raised to the power of p, single-step reward function Defined as

[0128]

[0129] For the second reinforcement learning sub-model Define the collision state as the endpoint. For a pair of agents that do not collide, award a positive reward. Agents should avoid collisions to obtain higher rewards. Figure 5 As shown, The model is a Q-network with three switching layers, designed for collision avoidance by the agent. The model is used to estimate the behavioral value function of a pair of agents. For example... Figure 5 As shown, The model's input is a pair of agent state features, i.e., two-state features. and To make the second behavior of the value function smoother, method 100 uses two-dimensional grid coding to encode the behavior of a pair of agents into binary vectors as well. Representation. Eigenvector and This is used to construct a three-switch layer in the second row of the value function, where the network parameters connected to 1 are activated, while the remaining parameters are ignored.

[0130] Based on the principle of selecting the maximum and minimum action values, the minimum action value function is first estimated, and then the action with the maximum and minimum values ​​is selected as the optimal action.

[0131] According to a specific embodiment of this application, the second reinforcement learning sub-model Update parameters using stochastic gradient descent.

[0132]

[0133] in,

[0134]

[0135]

[0136]

[0137] in, These are the behavioral values ​​of the two agents targeting collision avoidance. and These are the output values ​​of the online three-switch Q-network and its target network, ω. t and These are the parameters of the online three-switch Q-network and its target network, respectively. and Let each represent a pair of state feature vectors of agents j and k. Let represent the behavioral feature vectors of a pair of agents j and k, where β is the learning rate and γ is the reward decay factor. This second reinforcement learning sub-model is shared by all agents in the environment.

[0138] To maximize the global reward in the environment and to ensure that the optimal behavior balances the goals of quickly reaching the destination and avoiding collisions, the intelligent connected vehicle intersection coordinated driving control method 100 according to the above embodiments of this application, in step S5, uses the first reinforcement learning sub-model Second reinforcement learning sub-model Fusion constitutes fusion model C n Define the fusion model C. n The global objective function is

[0139]

[0140] Among them, Q i =Q π (s i ,a i ), Q j,k =Q π (s j,k ,a j,k M1(·) and M2(·) are user-defined functions, for example, M1(·) and M2(·) are M1(x) = x and M2(x) = x respectively. 2 ; Let η be the vector representing the behavior of all vehicles in the environment at time t, and η be the collision balance weights. The optimal behavior is the one that maximizes the global reward in the environment. Take as

[0141]

[0142] Figure 6 For fusion model C n The implementation principle diagram shows that each vehicle corresponds to one The model outputs behavioral values, one for each vehicle pair. The model outputs behavioral values, and a global objective function is defined based on the behavioral values ​​of all vehicles and vehicle pairs. Each driving decision made by a single vehicle takes into account maximizing the global reward of the intersection. Figure 7 This shows the fusion model C. n The implementation flowchart is shown below. Figure 7 As shown, the fusion model C n The implementation process is as follows:

[0143] First, in step S51, the fusion model C... n initialization: Network Q i , has a random parameter θ; Network Q j,k It has a random parameter ω; Target Network With random parameters Target Network With random parameters Experience storage pool D; environment env, initial state s0, initial behavior a0; simulation step count step=1, set maximum simulation step count M; set integer C, update the target network once every C steps;

[0144] Then, in step S52, a step is performed: the optimal action vector is calculated based on the aforementioned formula (20). In the environment env, from time t to t+1, the following is executed: And observe s t+1 and r t+1 ; Transition state(s) t ,a t ,r t+1 ,s t+1 Store to storage pool D;

[0145] Then, in step S53, sample sampling encoding: randomly sample a small batch of samples (s) from storage pool D. t ,a t ,r t+1 ,s t+1 ); for s t ,a t Encode to obtain state features Behavioral characteristics

[0146] Then, in step S54, the model parameters are updated: for all agents 1,2,…,N(t) in the environment at time t, the parameters are updated based on the aforementioned formula (7). Model parameters; for all agents 1, 2, ..., 0.5N(t)(N(t)-1) in the environment at time t, updated based on formula (15). Model parameters;

[0147] Then, in step S55, it is determined whether the simulation step number is an integer multiple of C. If it is, proceed to step S56, and let... If not, proceed to step S57 to further determine whether the simulation step count has reached the maximum value M. If yes, the process ends; otherwise, return to step S52.

[0148] The intelligent connected vehicle intersection coordinated driving control method 100 according to the above embodiments of this application uses sparse grid coding to achieve parameter sharing between states, and uses a neural network to estimate the vehicle behavior value function. This allows for the estimation of the behavior value function corresponding to uncollected samples using fewer training samples, resulting in strong generalization ability. The model assesses the probability of collision between each pair of vehicles in the environment and can adjust vehicles that are far away and have potential collision risks in advance; a global optimal function is defined as the basis for the agent's behavior selection, and every driving decision of all intelligent connected vehicles is based on global optimality.

[0149] Based on the intelligent connected vehicle intersection coordinated driving control method 100 of the above embodiments of this application, this application also proposes an intelligent connected vehicle intersection coordinated driving control system. Figure 8 A schematic diagram of the logical structure of an intelligent connected vehicle intersection coordinated driving control system 200 according to an embodiment of this application is shown. Figure 4 As shown, the intelligent connected vehicle intersection coordinated driving control system 200 includes a directed graph construction module 210, a feature extraction module 220, a speed excitation module 230, a collision avoidance module 240, and a fusion module 250. The directed graph construction module 210 is used to construct a directed graph of the road network topology using graph theory methods. Nodes in the directed graph represent the start point, end point, and intersection of the driving trajectory; the lines connecting the nodes represent the driving trajectory; the length of the lines represents the length of the driving trajectory; and the direction of the lines represents the driving direction. Each intelligent connected vehicle is defined as an agent, and each driving trajectory represents a driving intention. The feature extraction module 220 is used to extract the state features of the intelligent connected vehicle using a two-dimensional grid coding method, converting the state of the intelligent connected vehicle into a binary vector. The speed excitation module 230 is used to construct a first reinforcement learning sub-model to excite the vehicle to quickly reach the destination. Collision avoidance module 240 is used to construct a second reinforcement learning sub-model to avoid vehicle collisions. The fusion module 250 is used to combine the first reinforcement learning sub-model C1 and the second reinforcement learning sub-model. Fusion constitutes fusion model C n The optimal behavior that maximizes the global reward in the environment is calculated. For further specific implementation of the various modules of the intelligent connected vehicle intersection coordinated driving control system 200, please refer to the foregoing detailed description of the various steps of the intelligent connected vehicle intersection coordinated driving control method 100.

[0150] Figure 9 A logic diagram of an intelligent connected vehicle intersection coordinated driving control system 300 according to another embodiment of this application is shown. See also Figure 9 As shown, the intelligent connected vehicle intersection coordinated driving control system 300 includes a processor 310 and a memory 320, which are communicatively connected. The memory 320 stores a computer program, which, when executed by the processor 310, causes the processor 310 to implement the steps of the intelligent connected vehicle intersection coordinated driving control method 100 of the foregoing embodiments of this application.

[0151] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the intelligent connected vehicle intersection coordinated driving control method 100 of the foregoing embodiments of this application.

[0152] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for intersection coordination driving control of intelligent connected vehicles, characterized in that, The method comprises the following steps: S1, a directed graph of a road network topology is established by using a graph theory method, nodes in the directed graph represent starting points, ending points and intersection points of driving trajectories, connections between the nodes represent the driving trajectories, lengths of the connections represent lengths of the driving trajectories, directions of the connections represent driving directions, each intelligent connected vehicle is defined as an agent, and each driving trajectory represents a driving intention; S2, a state feature of the intelligent connected vehicle is extracted by using a two-dimensional grid coding method, and the state of the intelligent connected vehicle is converted into a binary vector; S3, a first reinforcement learning submodel for encouraging the vehicle to quickly reach the ending point is constructed, and a target function of the first reinforcement learning submodel is defined as ， wherein, , is the number of agents, is a function of time, is the state of a single agent , represents the set of states of the agents, is the acceleration of an agent , is the first action value function, which has the mathematical form ， wherein, represents a desired, represents a policy, is a decay factor for the reward, is a positive integer, represents the power of, one-step reward function is defined as S4, a second reinforcement learning submodel for avoiding vehicle collision is constructed, and a target function of the second reinforcement learning submodel is defined as ， wherein, , , and ; , and are the accelerations of the agents and respectively; is the state of the agent and , denotes the set of states of the agents; is the second action-value function, which is mathematically expressed as ， wherein, represents a desired, represents a policy, is a decay factor for the reward, is a positive integer, represents of power, one-step reward function is defined as S5, the first reinforcement learning submodel and the second reinforcement learning submodel are fused to form a fusion model, and a global target function of the fusion model is defined as ， where, , , and are user-defined functions, is the vector of all vehicles' behaviors in the environment at time is the vector of all vehicles' behaviors in the environment at time η is the collision balancing weight, computed to maximize the global return in the environment is 。 2. The method of claim 1, wherein, The step S2 further comprises: S21. Map each driving trajectory to a two-dimensional velocity-position plane, using... A two-dimensional grid, each two-dimensional grid having Let each sub-grid correspond to an element in the feature vector, then each driving trajectory corresponds to one... 3D eigenvectors; S22, for each of the plurality of agents in the environment a lane trajectory, causing the agent to represent a time instant running on the lane trajectory a feature vector of the agent representing a first element of the vector a first element of the vector is defined wherein , ; S23, will The feature vectors corresponding to each driving trajectory are concatenated to obtain the intelligent agent. State feature vector ， wherein is a vector.

3. The method of claim 2, wherein, The first reinforcement learning submodel defines a set of variables describing a single vehicle state as a single state, and a mathematical form thereof is ， wherein, represents an agent At time , , and represent the driving intention, speed and position of an agent , , is the number of agents, is a function of time.

4. The method of claim 3, wherein, The first reinforcement learning submodel updates parameters by using a stochastic gradient descent method ， wherein, where, is the single-agent behavior value for fast reaching the terminal state, and are the outputs of the online Q-network and its target network, respectively, and are the parameters of the online Q-network and its target network, respectively, is the learning rate, is the decay factor for the reward.

5. The method of claim 4, wherein, The second reinforcement learning submodel defines a set of variables describing a pair of vehicle states as a double state, and a mathematical form thereof is ， wherein, and are single states of a vehicle and , , , and .

6. The method of claim 5, wherein, The second reinforcement learning submodel updates parameters by using a stochastic gradient descent method ， wherein, where, is the double-agent behavior value for the collision avoidance target, and are the output values of the online three-switch Q-network and its target network, respectively, and are the parameters of the online three-switch Q-network and its target network, respectively, and represent the state feature vectors of a pair of agents and respectively, represent the behavior feature vectors of a pair of agents and respectively, is the learning rate, is the decay factor of the reward.

7. The method of claim 1, wherein, a user-defined function in a global objective function of the fusion model and are respectively and .

8. An intelligent connected vehicle intersection coordinated driving control system, characterized in that, The system comprises: a directed graph establishment module, configured to establish a directed graph of a road network topology by using a graph theory method, nodes in the directed graph represent starting points, ending points and intersection points of driving trajectories, connections between the nodes represent the driving trajectories, lengths of the connections represent lengths of the driving trajectories, directions of the connections represent driving directions, each intelligent connected vehicle is defined as an agent, and each driving trajectory represents a driving intention; a feature extraction module, configured to extract a state feature of the intelligent connected vehicle by using a two-dimensional grid coding method, and convert the state of the intelligent connected vehicle into a binary vector; a speed encouragement module, configured to construct a first reinforcement learning submodel for encouraging the vehicle to quickly reach the ending point, and define a target function of the first reinforcement learning submodel as ， where, , is the number of agents, is a function of time, is the state of a single agent , represents the set of states of the agents, is the acceleration of an agent , is the first action value function, which has the mathematical form ， wherein, represents a desired, represents a policy, is a decay factor for the reward, is a positive integer, represents the power of, the one-step reward function is defined as a collision avoidance module, configured to construct a second reinforcement learning submodel for avoiding vehicle collision, and define a target function of the second reinforcement learning submodel as ， wherein, , , and ; , and are the accelerations of the agents and respectively; is the state of the agent and , denotes the set of states of the agents; is the second action-value function, which is mathematically expressed as ， wherein, represents a desired, represents a policy, is a decay factor for the reward, is a positive integer, represents of power, one-step reward function is defined as a fusion module, configured to fuse the first reinforcement learning submodel and the second reinforcement learning submodel to form a fusion model, and define a global target function of the fusion model as ， where, , , and are user-defined functions, is the vector of all vehicles' behavior within the environment at time is the vector of all vehicles' behavior within the environment at time η is the collision balancing weight, computed to maximize the global return in the environment is 。 9. An intelligent connected vehicle intersection coordinated driving control system, characterized in that, The system comprises a processor and a memory, the memory stores a computer program, and the computer program is executed by the processor to realize the steps of the intelligent connected vehicle intersection coordinated driving control method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The system comprises a processor and a memory, the memory stores a computer program, and the computer program is executed by the processor to realize the steps of the intelligent connected vehicle intersection coordinated driving control method according to any one of claims 1-7.

Images