Multi-vehicle collaborative decision-making method and device for mixed traffic flow intersection, medium and product

By constructing a value-decomposition deep reinforcement learning model based on sequential game in multi-agent reinforcement learning, integrating the interaction between smart cars and human vehicles, the problems of low sample efficiency and high collision risk in the existing technology are solved, and more efficient coordinated decisions on traffic flow.

CN118918706APending Publication Date: 2024-11-08INST OF ELECTRICAL ENG CHINESE ACAD OF SCI
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410954591.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-08

Smart Images

  • Figure CN118918706A_ABST
    Figure CN118918706A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-vehicle collaborative decision-making method and device for a mixed traffic flow intersection, a medium and a product, and relates to the technical field of multi-vehicle collaborative decision-making, and the method comprises the steps: constructing a sequential game-based value decomposition multi-agent deep reinforcement learning model of an action filter fusing the interaction between intelligent vehicles and the interaction between the intelligent vehicles and manned vehicles; training a sequential game-based value decomposition multi-agent deep reinforcement learning model; and inputting the global state information of the intelligent vehicle to be subjected to collaborative decision making in the mixed traffic flow and the current collaborative decision making information into the trained sequential game-based value decomposition multi-agent deep reinforcement learning model, and determining the collaborative decision making information of the intelligent vehicle at the next moment. The sequential game-based action filter is added to the value decomposition multi-agent deep reinforcement learning model, so that the sample efficiency and the convergence speed of the algorithm are improved, the intersection passing collision risk of the intelligent network connection vehicle in the mixed traffic flow environment is reduced, and the traffic flow efficiency of the intersection is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of multi-vehicle collaborative decision-making, and in particular to a multi-vehicle collaborative decision-making method, device, medium and product at a mixed traffic flow intersection. Background Art

[0002] The continuous deepening of urbanization and the substantial increase in the number of urban population and vehicles have inevitably led to traffic congestion. This situation not only reduces social production efficiency, but also causes environmental pollution, increases the risk of traffic accidents, and affects the appearance of the city. Faced with these challenges, how to efficiently manage the flow of vehicles at urban intersections has become a pressing issue. Generally speaking, building infrastructure such as traffic lights is one of the effective ways to solve intersection congestion and frequent accidents. Based on these facilities, scholars have conducted extensive research on how to optimize the duration and phase of traffic lights, including the application of optimization theory algorithms and artificial intelligence algorithms based on deep reinforcement learning. In 1992, Michael G. H. Bell [1] pointed out in his paper that traffic lights alone cannot fundamentally solve the problem of traffic congestion, but need to integrate more information. For example, the current road conditions, the location and speed of other vehicles, etc. that autonomous vehicles can obtain in the future. Although traffic lights have always been the mainstream solution for intersection control, the waiting time of vehicles cannot be eliminated in this case. Therefore, a large number of relevant experts have conducted in-depth exploration on how to collaboratively control vehicles passing through intersections without traffic lights. The rapid development of intelligent connected vehicle related technologies, such as Vehicles to Vehicles (V2V) and Vehicles to Infrastructure (V2I), has also promoted this exploration process.

[0003] At present, for the problem of coordinated vehicle control at intersections without signal lights, most academics try to use centralized collaborative methods, for example, based on the model predictive control (MPC) method in the classical control method. First of all, the limitation of centralized methods is that they have high computational complexity and poor real-time and robustness. The development and application of machine learning technology (especially machine learning algorithms based on deep learning) have solved some difficult problems in the past, such as those in the fields of target recognition and robot control. Multi-agent deep reinforcement learning provides a new idea for solving the problem of coordinated control at intersections, for example, centralized multi-agent deep reinforcement learning algorithms. However, the computational complexity of such algorithms will increase exponentially with the number of vehicles passing through the intersection. In addition, centralized algorithms will encounter the problem of credit allocation between agents when solving cooperative tasks. How to fairly and effectively allocate the credit of each agent in the process of cooperation is one of the current problems. Since each agent regards other agents as part of the environment, the environment is constantly changing while the strategies of other agents are changing. Therefore, the direct use of distributed multi-agent reinforcement learning will be affected by the non-stationarity of the environment. Therefore, the more popular paradigm in the field of multi-agent reinforcement learning is centralized training and distributed execution. This method assumes that each agent can only observe part of the environmental information, and can obtain the complete global state information of the environment during the training phase of the algorithm, thereby avoiding the problem of environmental non-stationarity. However, since the algorithm of the prior art does not integrate the action filter based on sequential game that integrates the interaction between smart cars and between smart cars and human cars, the sample efficiency is low, resulting in slow convergence speed and low generalization of the algorithm, which further leads to a high risk of collision of smart connected vehicles in mixed traffic flows. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-vehicle collaborative decision-making method, device, medium and product at a mixed traffic flow intersection, so as to solve the problem that the existing multi-agent reinforcement learning technology does not effectively integrate the interaction between intelligent vehicles and between intelligent vehicles and human-driven vehicles, resulting in low algorithm sample efficiency, slow convergence speed and low strategy generalization, which further leads to a high risk of collision of intelligent connected vehicles at intersections in a mixed traffic flow environment.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A multi-vehicle collaborative decision-making method at a mixed traffic flow intersection, comprising:

[0007] Based on the distributed partially observable Markov decision process model, a sequential game-based value decomposition multi-agent deep reinforcement learning model is constructed; the sequential game-based value decomposition multi-agent deep reinforcement learning model includes an action filter based on sequential game, an action value neural network and a hybrid neural network;

[0008] Taking the global state information and collaborative decision information of each intelligent vehicle at the intersection at the current moment in a mixed traffic flow environment as input, and taking the collaborative decision information of each intelligent vehicle at the next moment as output, the value decomposition multi-agent deep reinforcement learning model based on sequential game is trained to determine the trained value decomposition multi-agent deep reinforcement learning model based on sequential game; the global state information includes the position of the vehicle, the speed of the vehicle and the mark position of whether the vehicle enters the traffic intersection; the collaborative decision information includes the acceleration of the vehicle;

[0009] The global state information of the intelligent vehicles to be coordinated and decided at the intersection in a mixed traffic flow environment and the current coordinated decision information are input into the trained sequential game-based value decomposition multi-agent deep reinforcement learning model to determine the coordinated decision information of the intelligent vehicles in the mixed traffic flow to be decided at the next moment.

[0010] Optionally, taking the global state information and collaborative decision information of each intelligent vehicle at the intersection in a mixed traffic flow environment at the current moment as input, and taking the collaborative decision information of each intelligent vehicle at the next moment as output, training the sequential game-based value decomposition multi-agent deep reinforcement learning model, and determining the trained sequential game-based value decomposition multi-agent deep reinforcement learning model, specifically includes:

[0011] Input the global state information and collaborative decision information of each intelligent vehicle at the intersection in a mixed traffic flow environment into the action filter based on the sequential game, and determine the global state information and collaborative decision information of each intelligent vehicle after filtering;

[0012] Inputting the filtered global state information and collaborative decision information of each intelligent vehicle into the action value neural network to determine the value of each optional action of each intelligent vehicle;

[0013] The value of each optional action of each intelligent vehicle is input into the hybrid neural network to determine the collaborative decision information of each intelligent vehicle at the next moment.

[0014] Optionally, the global state information and collaborative decision information of each intelligent vehicle at the intersection under the mixed traffic flow environment at the current moment are input into the action filter based on the sequential game, and the global state information and collaborative decision information of each intelligent vehicle after filtering are determined, specifically including:

[0015] Determine the priority of each intelligent vehicle according to the initial condition information of each intelligent vehicle at the intersection under the mixed traffic flow environment; the initial condition information includes the initial speed of the vehicle and the initial coordinates of the vehicle entering the traffic intersection;

[0016] Use sequential game theory to build an interaction model between intelligent vehicles and human vehicles;

[0017] Determining a subgame-perfect Nash equilibrium solution of the game according to the interaction model between the intelligent vehicle and the human vehicle;

[0018] According to the priority of each intelligent vehicle and the sub-game refined Nash equilibrium solution of the game, the global state information and collaborative decision-making information of each intelligent vehicle after filtering are determined.

[0019] Optionally, according to the priority of each intelligent vehicle and the sub-game refined Nash equilibrium solution of the game, the filtered global state information and collaborative decision information of each intelligent vehicle are determined, specifically including:

[0020] The collaborative decision information that does not conform to the priority of each intelligent vehicle and the collaborative decision information that does not conform to the sub-game refined Nash equilibrium solution set of the game are masked to determine the global state information and collaborative decision information of each intelligent vehicle after filtering; the sub-game refined Nash equilibrium solution set includes multiple sub-game refined Nash equilibrium solutions.

[0021] Optionally, the filtered global state information and collaborative decision information of each intelligent vehicle are input into the action value neural network to determine the value of each optional action of each intelligent vehicle, specifically including:

[0022] Using the first layer of the fully connected feedforward neural network of the action value neural network to expand the characterization capability of the filtered global state information and collaborative decision-making information of each intelligent vehicle;

[0023] The gated recurrent neural network of the action value neural network is used to evaluate and process the expanded state representation and collaborative decision-making information to determine action evaluation information; the action evaluation information is used to evaluate the quality of the collaborative decision-making information;

[0024] The action evaluation information is selected using the second layer of the fully connected feedforward neural network of the action value neural network to determine the value of each optional action of each intelligent vehicle.

[0025] Optionally, the value of each optional action of each intelligent vehicle is input into the hybrid neural network to determine the collaborative decision information of each intelligent vehicle at the next moment, specifically including:

[0026] The hybrid neural network is used to select an action sequence that can maximize the output value of the hybrid neural network from the values ​​of each optional action of each intelligent vehicle, and determine the collaborative decision information of each intelligent vehicle at the next moment.

[0027] Optionally, based on the distributed partially observable Markov decision process model, a sequential game-based value decomposition multi-agent deep reinforcement learning model is constructed, which also includes:

[0028] Using the formula G = (S, U, P, r, Z, O, n, γ), a distributed partially observable Markov decision process model is constructed;

[0029] Among them, S is the global state information obtained by all intelligent vehicles at each time step, U is the collaborative decision information of each intelligent vehicle at each time step, P is the probability of the intelligent vehicle entering the next state after making an action selection, r is the reward value obtained by all intelligent vehicles at each time step, Z is the local state information obtained by each intelligent vehicle, O is the function of the state and action of the intelligent vehicle, n is the number of intelligent vehicles, and γ is the discount factor.

[0030] A computer device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-vehicle collaborative decision-making method at the mixed traffic flow intersection.

[0031] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-vehicle collaborative decision-making method at a mixed traffic flow intersection.

[0032] A computer program product includes a computer program, characterized in that when the computer program is executed by a processor, the multi-vehicle collaborative decision-making method at the mixed traffic flow intersection is implemented.

[0033] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0034] The multi-vehicle collaborative decision-making method, device, medium and product for a mixed traffic flow intersection provided by the present invention construct a sequential game-based value decomposition multi-agent deep reinforcement learning model based on a distributed partially observable Markov decision process model, which includes a sequential game-based action filter that integrates the interaction between smart vehicles and between smart vehicles and manned vehicles; then, the global state information and collaborative decision information of each smart vehicle at the intersection in a mixed traffic flow environment at the current moment are taken as input, and the collaborative decision information of each smart vehicle at the next moment is taken as output, and the sequential game-based value decomposition multi-agent deep reinforcement learning model is trained to determine the trained sequential game-based value decomposition multi-agent deep reinforcement learning model; then, the global state information and current collaborative decision information of the smart vehicles to be collaboratively decided in the mixed traffic flow are input into the trained sequential game-based value decomposition multi-agent deep reinforcement learning model, so that the collaborative decision information of the smart vehicles at the intersection in the mixed traffic flow environment to be decided at the next moment can be determined. The sequential game-based value decomposition multi-agent deep reinforcement learning model, which outputs the collaborative decision information of each intelligent vehicle at the next moment according to the global state information and collaborative decision information of each intelligent vehicle in the mixed traffic flow to be decided at the current moment, includes a sequential game-based action filter that integrates the interaction between intelligent vehicles and between intelligent vehicles and human vehicles. Therefore, the sample efficiency, convergence speed and generalization of the algorithm are improved, and the collision risk of intelligent connected vehicles in mixed traffic flows is further reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flow chart of the multi-vehicle collaborative decision-making method for a mixed traffic flow intersection provided by the present invention;

[0037] Figure 2 A flow chart of the training process of the sequential game-based value decomposition multi-agent deep reinforcement learning model provided by the present invention;

[0038] Figure 3 A block diagram of the working process of the action filter based on sequential game provided by the present invention;

[0039] Figure 4 A block diagram of the working process of the action value network provided by the present invention;

[0040] Figure 5 A block diagram of the working process of the hybrid neural network provided by the present invention;

[0041] Figure 6 This is a flowchart of the deployment process of the trained sequential game-based value decomposition multi-agent deep reinforcement learning model for this application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The purpose of the present invention is to provide a multi-vehicle collaborative decision-making method, device, medium and product at a mixed traffic flow intersection, aiming to add a sequential game-based action filter that integrates the interaction between intelligent vehicles and between intelligent vehicles and human vehicles to the sequential game-based value decomposition multi-agent deep reinforcement learning model that outputs the collaborative decision information of each intelligent vehicle at the next moment according to the global state information and collaborative decision information of each intelligent vehicle in the mixed traffic flow to be decided at the current moment, thereby improving the sample efficiency, the convergence speed and generalization of the algorithm, and further reducing the collision risk of intelligent connected vehicles in mixed traffic flows.

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Example 1

[0046] like Figure 1 As shown, the present invention provides a multi-vehicle collaborative decision-making method at a mixed traffic flow intersection, comprising:

[0047] Step 101, based on the distributed partially observable Markov decision process model, construct a sequential game-based value decomposition multi-agent deep reinforcement learning model; the sequential game-based value decomposition multi-agent deep reinforcement learning model includes a sequential game-based action filter, an action value neural network and a hybrid neural network.

[0048] Step 101 specifically includes: using the formula G=(S, U, P, r, Z, O, n, γ) to construct a distributed partially observable Markov decision process model.

[0049] Among them, S is the global state information obtained by all intelligent vehicles at each time step, U is the collaborative decision information of each intelligent vehicle at each time step, P is the probability of the intelligent vehicle entering the next state after making an action selection, r is the reward value obtained by all intelligent vehicles at each time step, Z is the local state information obtained by each intelligent vehicle, O is the function of the state and action of the intelligent vehicle, n is the number of intelligent vehicles, and γ is the discount factor.

[0050] The coordinated control problem of intersections without traffic lights is modeled as a distributed partially observable Markov decision process (Decentralized Partially Observable Markov Decision Process, Dec-POMDP), which consists of a tuple G = (S, U, P, r, Z, O, n, γ), where s∈S represents the state information of all agents in the simulation environment that interacts with the algorithm, including the position, speed, whether to enter the intersection, and other marker information of all agents. In each step, each agent a∈A≡{1,2,...,N} selects an action u a ∈U, its joint action is u∈U≡U n , the action space uses discrete vehicle acceleration values. After making an action selection, the environment will enter the next state with probability P(s'|s,u):S×U×S→[0,1]. All agents share the same reward value function r(s,u):S×U→,γ∈[0,1).

[0051] The following principles are followed in the design of the reward function: first, the vehicle speed cannot be too low; second, avoid collisions between vehicles; and third, reward partially successful trajectories. The advantage of following the above principles is that, at the micro level, the reward function mechanism enables each vehicle to pass through the intersection safely and without collision (the third term of the reward function), and encourages vehicles to pass through the intersection quickly, that is, to pass through the intersection in the shortest possible time. In a macro sense, this reward function can improve overall traffic efficiency and traffic safety. In addition, each agent can only observe its local information z∈Z, which is a function of state and action O(s,a):S×U→Z. The historical observation information trajectory of each agent is recorded as τ a ∈T≡(Z×U) * , the agent’s strategy is based on historical observation information, denoted by π a (u a |τ a ):T×U→[0,1]. Since this method uses a value decomposition-based method, each agent's strategy corresponds to a Q-value function, that is, In the formula Here γ represents the discount factor. tIndicates the global state information obtained by each agent at each time step, including the vehicle’s current position, speed, and the marking position information of whether it has entered the intersection. t Represents the discrete decision-making behavior performed by each agent at each time step, specifically its acceleration value. t It represents the reward value obtained by all agents at each time step, and the determination of this value depends on the speed of the vehicle, whether there is a collision, and the number of vehicles that have safely passed the intersection at the current moment. In this embodiment, the agent is an intelligent vehicle.

[0052] Step 102, taking the global state information and collaborative decision information of each intelligent vehicle at the intersection in a mixed traffic flow environment at the current moment as input, and taking the collaborative decision information of each intelligent vehicle at the next moment as output, training the value decomposition multi-agent deep reinforcement learning model based on sequential game, and determining the trained value decomposition multi-agent deep reinforcement learning model based on sequential game; the global state information includes the position of the vehicle, the speed of the vehicle, and the mark position of whether the vehicle enters the traffic intersection; the collaborative decision information includes the acceleration of the vehicle. The value decomposition multi-agent deep reinforcement learning model based on sequential game includes an action filter based on sequential game, an action value neural network, and a hybrid neural network. The output of the value decomposition multi-agent deep reinforcement learning model based on sequential game is used to control the behavior of each intelligent vehicle passing through the intersection area without signal lights.

[0053] like Figure 2 As shown in the figure, the algorithm training framework is a training process diagram of the value decomposition multi-agent deep reinforcement learning model based on sequential games. The input of the value decomposition multi-agent deep reinforcement learning model based on sequential games is The output after the action value neural network is This output represents the evaluation of each action value neural network on the actions of each intelligent vehicle in the current state, and selects the action to be output through the following action selection mechanism (ò-greedy strategy):

[0054]

[0055] Among them, ò represents the probability value obtained by sampling from the interval [0,1] according to a uniform distribution.

[0056] The sequential game-based action filter in the framework of the sequential game-based value decomposition multi-agent deep reinforcement learning model is designed based on sequential game theory. The specific process is as follows Figure 3 The global state information and collaborative decision information of each intelligent vehicle in the mixed traffic flow at the current moment are input into the action filter based on the sequential game to determine the global state information and collaborative decision information of each intelligent vehicle after filtering.

[0057] Specifically, the input vector of the action filter based on sequential game is as follows:

[0058]

[0059] The action filter based on sequential game mainly focuses on That is, the actions of each intelligent vehicle are filtered.

[0060] The filtering process includes: first, determining the priority of each intelligent vehicle according to the initial condition information of each intelligent vehicle at the intersection under a mixed traffic flow environment (initial speed of the intelligent vehicle, initial coordinates of the intelligent vehicle entering the intersection, etc.), and the priority of each intelligent vehicle is used to guide each intelligent networked vehicle to pass through the intersection in a certain order, wherein the initial condition information includes the initial speed of the vehicle and the initial coordinates of the vehicle entering the traffic intersection; second, since intelligent networked vehicles and human vehicles coexist at intersections under a mixed traffic flow environment, sequential game theory is used to construct an interaction model between intelligent vehicles and human vehicles, wherein intelligent vehicles make decisions after human vehicles, and in the process of constructing the interaction model between intelligent vehicles and human vehicles, it is only necessary to specify the size relationship of the utility function; third, according to the interaction model between intelligent vehicles and human vehicles, the sub-game refined Nash equilibrium solution of the game is determined; fourth, according to the priority of each intelligent vehicle and the sub-game refined Nash equilibrium solution of the game, the global state information and collaborative decision-making information of each intelligent vehicle after filtering are determined. The collaborative decision information that does not conform to the priority of each intelligent vehicle and the collaborative decision information that does not conform to the sub-game refined Nash equilibrium solution set of the game are masked to determine the global state information and collaborative decision information of each intelligent vehicle after filtering; the sub-game refined Nash equilibrium solution set includes multiple sub-game refined Nash equilibrium solutions. The final output after filtering is: in, Represents the action of intelligent vehicle a after passing through the action filter based on sequential game.

[0061] The specific working process of the action value network in the framework of the sequential game-based value decomposition multi-agent deep reinforcement learning model is as follows: Figure 4 The filtered global state information and collaborative decision-making information of each intelligent vehicle are input into the action value neural network to determine the value of each optional action of each intelligent vehicle.

[0062] Specifically, the action value network is composed of three layers of neural networks. The first layer is a fully connected feedforward neural network, defined as a fully connected feedforward neural network; the second layer is a gated recurrent neural network (Gated Recurrent Network, GRU); the third layer is still a fully connected feedforward neural network, defined as a second fully connected feedforward neural network. Finally, the above-mentioned ò-greedy strategy is used to output the specific Q values ​​of all the selected actions of the agents. The first fully connected feedforward neural network of the action value neural network is used to expand the representation capability of the filtered global state information and collaborative decision information of each intelligent vehicle to determine the first collaborative decision information; the gated recurrent neural network of the action value neural network is used to process the first collaborative decision information and the historical trajectory of the intelligent vehicle to determine the second collaborative decision information; the second fully connected feedforward neural network of the action value neural network is used to expand the representation capability of the second collaborative decision information to determine the value of each optional action of each intelligent vehicle.

[0063] The input of the action value network is the output vector of the action filter based on the sequential game, and the formula is as follows:

[0064] in, represents the observation of intelligent vehicle a; represents the action of agent a after passing through the filter; f a Represents the tag information of agent a composed of one-hot vectors. For example, the tag of agent 1 is [1, 0, 0, ..., 0]; after passing through the second fully connected feedforward neural network, the output is [Q1(τ 1 ,·),Q2(τ 2 ,·),…,Q N (τ N ,·)], which represents the value of each optional action of each agent; after the ò-greedy strategy, the output is

[0065] The hybrid neural network is composed of a fully connected feedforward neural network. The working process diagram is as follows: Figure 5 The value of each optional action of each intelligent vehicle is input into the hybrid network (i.e. Figure 2 The hybrid neural network in the algorithm is used to determine the collaborative decision information of each intelligent vehicle at the next moment. The hybrid neural network is used to select an action sequence that can maximize the output value of the hybrid neural network from the values ​​of each optional action of each intelligent vehicle to determine the collaborative decision information of each intelligent vehicle at the next moment.

[0066] Specifically, the input of the hybrid neural network combines the global state information of the environment and the output of the action value network, and the specific form is: where s tAfter being input to the fully connected neural network, the output network parameter weights W1, W2 (including absolute value operation on the output) and network parameter biases b1, b2 are obtained by matrix multiplication with the output of the action value network. The physical meaning of this value is the approximation of the shared reward value r. According to the loss function formula, the stochastic gradient descent algorithm is used to update the parameters of each layer in the hybrid neural network. The formula of the trend function is:

[0067]

[0068] Where b represents the total dimension of a batch of data used for algorithm training each time; r represents the shared reward value obtained by all agents in the simulation environment; γ represents the discount factor; Q tot (·) represents the entire hybrid neural network (the hybrid neural network is essentially a complex nonlinear function), where τ′ represents the historical information trajectory at the next moment; s' represents the state information of the environment at the next moment; Represents the action sequence output by the filter at the next moment; θ and θ′ represent the parameters of the hybrid neural network and the target hybrid neural network respectively. After each interaction between the algorithm and the simulation environment, the relationship between the two is θ′=β*θ+(1-β)*θ′, β is an adjustable parameter, and its physical meaning indicates how fast the target network parameters are updated; It means selecting an action sequence that can maximize the output value of the model from the output vector of the hybrid neural network model with parameter θ based on the different action outputs of all agents in different environmental states s' at the next moment.

[0069] The output value q of the hybrid neural network tot With Q1~Q N The relationship between them is shown in the following formula:

[0070]

[0071] Among them, q tot Represents the entire hybrid neural network Q tot The output value of (·) is an implicit nonlinear function.

[0072] The action applications and actual scenarios of the hybrid neural network output are as follows:

[0073] Since the limiting condition of the present invention is that all vehicles are intelligent vehicles, in practical applications, the trajectory of intelligent vehicles passing through the intersection is a fixed curve, for example, the trajectory of an intelligent vehicle going straight is fixed to a straight line, and the trajectory of an intelligent vehicle turning left is fixed to a straight line and a quarter arc with a turning radius r. In this application, the vehicle only needs to consider the longitudinal kinematic behavior derived from the discrete acceleration value, and the acceleration line is defined as: u a =[-ahigh ,-a medium ,-a low ,0,a low ,a medium ,a high ], where [a low ,a medium ,a high ] represent low, medium and high acceleration values ​​respectively.

[0074] Q tot The relationship with the reward value r is: It is an approximate estimate of the reward value r shared by all intelligent vehicles. The closer it is to r, the better the training effect of the value decomposition multi-agent deep reinforcement learning model based on sequential game is.

[0075] The formula for the stochastic gradient descent algorithm is as follows:

[0076]

[0077] Among them, θ represents all the parameters of the hybrid neural network model, Represents the gradient of the loss function, α represents the learning rate. This solution uses the autograd module in the deep learning framework Pytorch (whose function is to automatically solve the gradient of the custom loss function) to automatically calculate this value and convert it into It is passed to the action value network Q(·) through linear transformation, and the Adam optimizer is used, with the learning rate set to 5×10 -4 The update process formula of the action value network parameters is:

[0078]

[0079] Among them, θ Q represents all the parameters of the action-value network, and f(·) represents the function that maps the loss value of the hybrid neural network to the action-value network after automatically solving the gradient mechanism in the deep learning framework pytorch.

[0080] The evaluation process of the multi-agent deep reinforcement learning model after training based on the value decomposition of sequential games actively closes the hybrid neural network model in the model evaluation stage, and transmits the status information of the intelligent vehicle, the collaborative decision-making information, and the ID information of each intelligent vehicle, namely: Input to the trained action-value neural network, based on its output Adopting a ò-greedy strategy, using Select actions and output action sequences It is transmitted to the SUMO simulation environment to control the coordination of vehicles in the simulation environment.

[0081] Step 103, input the global state information of the intelligent vehicles to be coordinated and decided at the intersection in the mixed traffic flow environment and the current coordinated decision information into the trained sequential game-based value decomposition multi-agent deep reinforcement learning model to determine the coordinated decision information of the intelligent vehicles in the mixed traffic flow to be decided at the next moment. Figure 6 As shown, it is a flowchart of the algorithm deployment process, that is, a flowchart of the deployment process of the trained sequential game-based value decomposition multi-agent deep reinforcement learning model.

[0082] Example 2

[0083] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-vehicle collaborative decision-making method at a mixed traffic flow intersection in Example 1.

[0084] Example 3

[0085] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-vehicle collaborative decision-making method at a mixed traffic flow intersection in Example 1.

[0086] Example 4

[0087] A computer program product includes a computer program, which, when executed by a processor, implements the multi-vehicle collaborative decision-making method at a mixed traffic flow intersection in Example 1.

[0088] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-vehicle collaborative decision-making method for mixed traffic flow intersections. It is characterized in that include: Based on the distributed partially observable Markov decision process model, a sequential game-based value decomposition multi-agent deep reinforcement learning model is constructed; the sequential game-based value decomposition multi-agent deep reinforcement learning model includes an action filter based on sequential game, an action value neural network and a hybrid neural network; Taking the global state information and collaborative decision information of each intelligent vehicle at the intersection at the current moment in a mixed traffic flow environment as input, and taking the collaborative decision information of each intelligent vehicle at the next moment as output, the value decomposition multi-agent deep reinforcement learning model based on sequential game is trained to determine the trained value decomposition multi-agent deep reinforcement learning model based on sequential game; the global state information includes the position of the vehicle, the speed of the vehicle and the mark position of whether the vehicle enters the traffic intersection; the collaborative decision information includes the acceleration of the vehicle; The global state information of the intelligent vehicles to be coordinated and decided at the intersection in a mixed traffic flow environment and the current coordinated decision information are input into the trained sequential game-based value decomposition multi-agent deep reinforcement learning model to determine the coordinated decision information of the intelligent vehicles in the mixed traffic flow to be decided at the next moment.

2. The multi-vehicle collaborative decision-making method at a mixed traffic flow intersection according to claim 1, It is characterized in that Taking the global state information and collaborative decision information of each intelligent vehicle at the intersection in a mixed traffic flow environment at the current moment as input, and taking the collaborative decision information of each intelligent vehicle at the next moment as output, training the sequential game-based value decomposition multi-agent deep reinforcement learning model, and determining the trained sequential game-based value decomposition multi-agent deep reinforcement learning model, specifically includes: Input the global state information and collaborative decision information of each intelligent vehicle at the intersection in a mixed traffic flow environment into the action filter based on the sequential game, and determine the global state information and collaborative decision information of each intelligent vehicle after filtering; Inputting the filtered global state information and collaborative decision information of each intelligent vehicle into the action value neural network to determine the value of each optional action of each intelligent vehicle; The value of each optional action of each intelligent vehicle is input into the hybrid neural network to determine the collaborative decision information of each intelligent vehicle at the next moment.

3. The multi-vehicle collaborative decision-making method at a mixed traffic flow intersection according to claim 2, It is characterized in that The global state information and collaborative decision information of each intelligent vehicle at the intersection in the mixed traffic flow environment are input into the action filter based on the sequential game, and the global state information and collaborative decision information of each intelligent vehicle after filtering are determined, specifically including: Determine the priority of each intelligent vehicle according to the initial condition information of each intelligent vehicle at the intersection under the mixed traffic flow environment; the initial condition information includes the initial speed of the vehicle and the initial coordinates of the vehicle entering the traffic intersection; Use sequential game theory to build an interaction model between intelligent vehicles and human vehicles; Determining a subgame-perfect Nash equilibrium solution of the game according to the interaction model between the intelligent vehicle and the human vehicle; According to the priority of each intelligent vehicle and the sub-game refined Nash equilibrium solution of the game, the global state information and collaborative decision-making information of each intelligent vehicle after filtering are determined.

4. The multi-vehicle collaborative decision-making method at a mixed traffic flow intersection according to claim 3, It is characterized in that According to the priorities of the intelligent vehicles and the sub-game refined Nash equilibrium solutions of the game, the global state information and collaborative decision-making information of the intelligent vehicles after filtering are determined, specifically including: The collaborative decision information that does not conform to the priority of each intelligent vehicle and the collaborative decision information that does not conform to the sub-game refined Nash equilibrium solution set of the game are masked to determine the global state information and collaborative decision information of each intelligent vehicle after filtering; the sub-game refined Nash equilibrium solution set includes multiple sub-game refined Nash equilibrium solutions.

5. The multi-vehicle collaborative decision-making method at a mixed traffic flow intersection according to claim 2, It is characterized in that Inputting the filtered global state information and collaborative decision information of each intelligent vehicle into the action value neural network to determine the value of each optional action of each intelligent vehicle, specifically including: Using the first layer of the fully connected feedforward neural network of the action value neural network to expand the characterization capability of the filtered global state information and collaborative decision-making information of each intelligent vehicle; The gated recurrent neural network of the action value neural network is used to evaluate and process the expanded state representation and collaborative decision-making information to determine action evaluation information; the action evaluation information is used to evaluate the quality of the collaborative decision-making information; The action evaluation information is selected using the second layer of the fully connected feedforward neural network of the action value neural network to determine the value of each optional action of each intelligent vehicle.

6. The multi-vehicle collaborative decision-making method at a mixed traffic flow intersection according to claim 2, It is characterized in that Inputting the value of each optional action of each intelligent vehicle into the hybrid neural network to determine the collaborative decision information of each intelligent vehicle at the next moment, specifically including: The hybrid neural network is used to select an action sequence that can maximize the output value of the hybrid neural network from the values ​​of each optional action of each intelligent vehicle, and determine the collaborative decision information of each intelligent vehicle at the next moment.

7. The multi-vehicle collaborative decision-making method at a mixed traffic flow intersection according to claim 1, It is characterized in that Based on the distributed partially observable Markov decision process model, a value decomposition multi-agent deep reinforcement learning model based on sequential games is constructed, which also includes: Using the formula G = (S, U, P, r, Z, O, n, γ), a distributed partially observable Markov decision process model is constructed; Among them, S is the global state information obtained by all intelligent vehicles at each time step, U is the collaborative decision information of each intelligent vehicle at each time step, P is the probability of the intelligent vehicle entering the next state after making an action selection, r is the reward value obtained by all intelligent vehicles at each time step, Z is the local state information obtained by each intelligent vehicle, O is the function of the state and action of the intelligent vehicle, n is the number of intelligent vehicles, and γ is the discount factor.

8. A computer device, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-vehicle collaborative decision-making method at a mixed traffic flow intersection as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, it implements the multi-vehicle collaborative decision-making method at a mixed traffic flow intersection as described in any one of claims 1-7.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, it implements the multi-vehicle collaborative decision-making method at a mixed traffic flow intersection as described in any one of claims 1-7.

Citation Information

Cited By

  • Multi-mode traffic simulation method based on multi-agent sequential decision

    CN121600715A

  • A multi-modal traffic simulation method based on multi-agent sequential decision

    CN121600715B