Complex traffic cooperative control method based on heterogeneous multi-agent reinforcement learning

Abstract

The invention discloses a complex traffic cooperative control method based on heterogeneous multi-agent reinforcement learning, and belongs to the technical field of intelligent traffic, and the method comprises the steps: designing a heterogeneous mixed traffic scene abstract modeling method based on a heterogeneous graph structure for three types of heterogeneous agents of human driving vehicles, automatic driving vehicles and signal lamps; an efficient communication mechanism among the objects is realized; an encoder suitable for a heterogeneous graph structure is designed, and efficient embedding integration of heterogeneous object observation features can be achieved; establishing a multi-agent action generation network and a value evaluation network based on a heterogeneous graph convolutional deep network by sharing a heterogeneous graph encoder for action generation of each agent and value evaluation for the generated action; and constructing a deep reinforcement learning method suitable for heterogeneous multi-agents to carry out model training, inputting environment observation data of each heterogeneous multi-agent, and outputting control quantities of all controlled objects, thereby realizing heterogeneous multi-agent cooperative control in a mixed traffic flow scene.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent transportation, and specifically to a complex traffic collaborative control method based on heterogeneous multi-agent reinforcement learning. Background Art

[0002] With the continuous maturity of autonomous driving technology, the intelligence level of vehicles has been increasing day by day. However, in the foreseeable future for a long time, human-driven vehicles and autonomous driving vehicles will still coexist in the road environment, forming a complex hybrid traffic flow. Traditional traffic control methods are mostly based on a single type of vehicle or only consider signal control, lacking sufficient modeling of the interaction between multiple types of agents, and it is difficult to meet the real-time and efficient collaborative control requirements in complex environments. In addition, existing research based on multi-agent reinforcement learning usually assumes that agents are homogeneous, and it cannot well accommodate the multi-dimensional characteristics of heterogeneous entities such as autonomous driving vehicles, human-driven vehicles, and traffic signals, thus restricting the accuracy and generalization ability of policy optimization. Summary of the Invention

[0003] (I) Technical Problems to be Solved

[0004] Aiming at the deficiencies of the prior art, the present invention effectively integrates the observation information of different types of agents by using a heterogeneous graph structure, and realizes the learning of collaborative control strategies for heterogeneous multi-agents through a shared heterogeneous graph encoder and corresponding action generation network and value evaluation network. At the same time, independent experience storage and exploration strategies are designed for different types of agents, which can significantly improve the learning efficiency and control performance. Through the above method, it is possible to effectively reduce the collision risk, alleviate traffic congestion, and improve the safety and traffic efficiency of the overall traffic flow in a mixed traffic scenario.

[0005] (II) Technical Solutions

[0006] To achieve the above object, the present invention provides the following technical solution: A complex traffic collaborative control method based on heterogeneous multi-agent reinforcement learning, comprising the following steps:

[0007] S1. Design a heterogeneous multi-agent graph structure including autonomous driving vehicles, human-driven vehicles, and traffic signals;

[0008] S2. Based on the heterogeneous multi-agent graph structure in step S1, construct a heterogeneous observation encoder for autonomous driving vehicles and traffic signals, input the environmental observation data of the heterogeneous multi-agents, and output the encoded environmental observation data;

[0009] S3. Design the action generation network and the corresponding value evaluation network for heterogeneous multi - agents, and share the heterogeneous observation encoder in step S2 for the action generation network and the value evaluation network. That is, the action generation network outputs the actions of the heterogeneous multi - agents according to the encoded environmental observation data, and the value evaluation network predicts the corresponding value based on the actions of the heterogeneous multi - agents;

[0010] S4. Construct a multi - level exploration and exploitation strategy suitable for the actions of heterogeneous multi - agents, and the output exploration value acts on the actions generated by the action generation network in step S3 for exploration regularization;

[0011] S5. Design two experience pools for autonomous vehicles and traffic lights, and store the environmental observation data, actions, and rewards of the agents. Specifically, each heterogeneous agent (autonomous vehicle, traffic light) outputs an action according to its action generation network. After the environment executes this action, new observations and corresponding rewards are obtained, and the quadruple information at the th moment is stored in the experience pool. For autonomous vehicles, record the quadruple information at the th moment and store it in the experience pool of the autonomous vehicle, where , , and respectively represent the environmental observation data, the action generated by the action generation network, the obtained reward, and the environmental observation data of the th autonomous vehicle at the th moment; for traffic lights, similarly record the quadruple information at the th moment , , and respectively represent the environmental observation data, the action generated by the action generation network, the obtained reward, and the environmental observation data of the th traffic light at the

[0012] S6. Establish a heterogeneous multi-agent deep reinforcement learning framework. During the learning process, collect the experiences of heterogeneous multi-agents, input them into the action generation network designed in step S3, and obtain the action values through the corresponding value evaluation network. Update the value evaluation network according to the accuracy of the value evaluation, and update the action generation network according to the magnitude of the value of the action generated by the action generation network, so as to continuously iterate to achieve the update of the network, that is, the action learning of heterogeneous multi-agents;

[0013] S7. Save the model parameters after learning is completed, load and deploy the heterogeneous multi-agent collaborative control model, and input the environmental observation data of each heterogeneous multi-agent to output the control quantities of all controlled objects.

[0014] As a preferred technical solution of the present invention, step S1 includes the following steps:

[0015] S1.1. The autonomous vehicle and the traffic signal identify the surrounding objects, i.e., human-driven vehicles, and judge whether they should be linked to each other according to the distance threshold If the distance is less than the distance threshold, it is considered that the autonomous vehicle or the traffic signal should be connected to the human-driven vehicle, otherwise it is not connected. Since the node types in this structure are not completely consistent, the generated graph structure belongs to a heterogeneous graph;

[0016] S1.2. Convert the heterogeneous graph in step S1.1 into a mathematical expression: Use to represent the set of autonomous vehicle nodes, where represents the autonomous vehicle numbered , …… ; represents the number of autonomous vehicles; Use to represent the set of human-driven vehicle nodes, where represents the human-driven vehicle numbered , …… ; represents the number of human-driven vehicles; Use to represent the set of traffic signal nodes, where represents the traffic signal numbered , …… ; Therefore, the overall node set can be represented as ; Therefore, for any three types of nodes in this heterogeneous graph, if the spatial Euclidean distance between any two nodes is less than the set distance threshold , then connect these two nodes in the heterogeneous graph structure and set the connection edge weight equal to 1; furthermore, obtain the adjacency matrix and the feature matrix , the adjacency matrix of the elements :

[0017]

[0018] In the formula, the subscript represents the th autonomous vehicle, the th human-driven vehicle, and the th traffic signal, represents the connection situation between the th autonomous vehicle, the th human-driven vehicle, and the th traffic signal; and obtain the heterogeneous feature matrix , where is the feature information of the autonomous vehicle, and represent the th autonomous vehicle's , coordinates; and respectively represent the speed and acceleration of the th autonomous vehicle; represents the lane where the th autonomous vehicle is located; represents the feature information of the human-driven vehicle, and represent the th human-driven vehicle's , coordinates; and respectively represent the speed and acceleration of the th human-driven vehicle; represents the lane where the th human-driven vehicle is located; , where represents the phase in which the th traffic signal is currently located, represents the time required for the th traffic signal to switch to the next phase;

[0019] As a preferred technical solution of the present invention, the step S2 includes the following steps:

[0020] S2.1. Since there are differences in attributes, observation dimensions, etc. between autonomous vehicles and traffic signals, design a heterogeneous encoder to extract different features for different types of nodes, so that it can uniformly output a high-dimensional representation for the subsequent decision-making network:

[0021]

[0022] In the formula, represents the heterogeneous encoder mapping process; is the parameter of the encoder; is the encoded feature;

[0023] S2.2. For the basic principle of the encoder described in step S2.1, design modules such as multi-head self-attention graph convolution, ordinary graph convolution, heterogeneous graph convolution, and time series convolution as the main structure of the observation encoder for signal light type nodes; among them, the calculation process of the multi-head self-attention graph convolution can be expressed as:

[0024]

[0025] In the formula, represents the environmental observation data of the th autonomous driving vehicle or the th human-driven vehicle or the th signal light in the heterogeneous graph; represents the environmental observation data of an agent with a different node type from , and the subscript represents a node type different from ; is the learnable linear mapping matrix of the multi-head self-attention graph convolution; is the attention weight vector; represents the unnormalized attention coefficient vector; represents activation function; represents activation function; represents activation function; is the attention coefficient normalized by the activation function; represents the environmental observation data of the th autonomous driving vehicle or the th human-driven vehicle or the th signal light after passing through the multi-head self-attention graph convolution; for the ordinary graph convolution, its calculation process can be expressed as:

[0026]

[0027] In the formula, and respectively represent the learnable weight matrix and bias term of the ordinary graph convolution, is the th autonomous driving vehicle or the the environmental observation data of a human-driven vehicle or the nth traffic signal; representing the nth autonomous vehicle or the nth human-driven vehicle or the set of other nodes that are connected to the node represented by the nth traffic signal in the heterogeneous graph structure;

[0028] Based on the above two forms of graph convolution, the heterogeneous graph convolution process of the present invention is constructed as follows: An edge relationship is constructed between any two node types to represent the action relationship of one node type on another node type; therefore, for any edge type, in one information aggregation and propagation process, the process of heterogeneous graph convolution can be expressed as:

[0029]

[0030] wherein, represents the environmental observation data of the nth autonomous vehicle or the nth human-driven vehicle or the nth traffic signal after heterogeneous graph convolution; represents the graph convolution operator acting on different edge types, which can be specifically expressed as the above-mentioned multi-head self-attention graph convolution process and the ordinary graph convolution process in the present invention; since one node type may establish edge relationships with multiple other types of nodes, it is necessary to unify the outputs of different convolution operators to the same dimension and perform aggregation, which can be specifically expressed as:

[0031]

[0032] wherein, represents the summation aggregation function; represents the environmental observation data of the traffic signal after global encoding through aggregation;

[0033] Furthermore, the node features processed by heterogeneous graph convolution are input into the network:

[0034]

[0035] wherein, and respectively represent the update gate and the reset gate at the t-th moment. The update gate is used to determine how much information at the t-th moment needs to be retained, and the reset gate determines to what extent the hidden state at the t-th moment is ignored; and and Denote the parameter matrix and bias vector for updating the gate; Denote the hidden state at time Denote the environmental observation data of each node after heterogeneous graph convolution at time Denote the activation function; and denote the weight parameter and bias vector of the reset gate respectively; use the reset gate to control the dependency degree of and calculate the candidate hidden state Denote the activation function; and are the parameter matrix and bias of the candidate hidden state; after the above steps, the environmental observation data of the traffic signal can realize the high-dimensional feature space embedding coding of the original environmental observation data;

[0036] S2.3. For autonomous vehicles, their observations have a strong real-time feedback property, that is, the intelligent agent learning strategy of autonomous vehicles needs to rely on the single-step "state-action-reward" set. Therefore, the encoder of autonomous vehicles is designed as a static single-step heterogeneous graph encoder. Input the environmental observation data of each autonomous vehicle at the current time step, and output the encoded environmental observation data of all autonomous vehicles at the current time step. Specifically, as described in step S2.2, use the same heterogeneous graph convolution operation to perform convolution on the environmental observation data of autonomous vehicles. Denote the environmental observation data of autonomous vehicles after heterogeneous graph convolution as and then improve the non-linear expression ability through a linear mapping:

[0037]

[0038] In the formula, is the weight parameter of the linear mapping layer; is the encoded environmental observation data of autonomous vehicles after linear mapping; through the two heterogeneous graph encoder structures shown in the above steps S2.1 to S2.3, effective feature abstraction and fusion can be performed on heterogeneous intelligent agents such as autonomous vehicles, human-driven vehicles, and traffic signals, providing unified and strongly expressive input features for the subsequent action generation network and value evaluation network;

[0039] As a preferred technical solution of the present invention, the step S3 includes the following steps:

[0040] S3.1. Share the encoder part defined in step S2 between the "action generation network" and the "value evaluation network" to reduce model redundancy and ensure encoding consistency. That is, design a heterogeneous graph convolutional layer to further fuse the encoded heterogeneous graph features:

[0041]

[0042] In the formula, represents the heterogeneous graph convolution process described in step S2.2, represents the shared high-dimensional feature representation after heterogeneous graph convolution;

[0043] S3.2. Design the action generation network for autonomous vehicles and traffic lights, which can be specifically described as follows: For autonomous vehicles: The actions include the probability distribution of acceleration and lane-changing actions; For traffic lights: The action can be the signal timing for the next phase. Use a multi-layer perceptron as the action generation model as follows:

[0044]

[0045] In the formula, represents the action of the th autonomous vehicle, specifically including the lane-changing action of the th autonomous vehicle and the acceleration action of the th autonomous vehicle , where is a probability distribution quantity, and , where each element represents the probability of each strategy in the strategy set . The , , in the strategy set represent left lane change, no lane change, and right lane change respectively; represents the action generation network mapping process of the th autonomous vehicle, are the weight parameters of the autonomous vehicle action generation network; is the signal timing for the next phase of the rd traffic light ; represents the action generation network mapping process of the traffic lights, are the weight parameters of the traffic light action generation network;

[0046] S3.3. The value evaluation network is used to evaluate the value of the current agent's action decision. For the heterogeneous multi-agent scenario of the present invention, a centralized value evaluation network mode is adopted. The actions and state observations of all agents are input into the value evaluation network to obtain the action values corresponding to each agent:

[0047]

[0048] In the formula, represents the value brought by the actions of each autonomous vehicle and traffic signal; represents the input of the globally shared feature after the second encoding , the actions of all autonomous vehicles and the actions of all traffic signals so as to output the mapping process of the action value of each agent; is the weight parameter of the centralized value evaluation network;

[0049] As a preferred technical solution of the present invention, the step S4 includes the following steps:

[0050] S4.1. In multi-heterogeneous agent reinforcement learning, in order to balance the exploration depth and policy stability of multi-dimensional vehicle and traffic signal control, the present invention proposes a multi-level exploration and exploitation strategy. This strategy switches between the following three modes by defining and dynamically regulating three threshold parameters: 1) Random exploration: completely random action sampling, indicating that the agent conducts action exploration; 2) Soft exploitation: sampling actions according to the discrete action probability distribution output by the action generation network; 3) Hard exploitation: directly selecting the action with the highest probability output by the action generation network; For the lane-changing actions of autonomous vehicles, in order to represent the exploration and exploitation mechanisms of the acceleration actions of autonomous vehicles and the phase timing actions of traffic signals, the present invention introduces a noise attenuation strategy, and continuously attenuates a threshold to make the continuous actions closer and closer to the actions generated by the action generation network.

[0051] S4.2. Based on the basic description of step S4.1, detailed threshold definitions and theoretical elaborations are carried out; Denote as a decay threshold, whose initial value is set to , and its final decay value is , and the decay step is , so as the agent is trained, the decay of can be specifically expressed as:

[0052]

[0053] In the formula, is a decay coefficient, and by continuously calculating the decay coefficient to maintain Decay; This parameter is mainly used in the exploration and exploitation processes of the acceleration actions of autonomous vehicles and the phase timing actions of signal lights. Specifically, design a random Gaussian noise generator as follows:

[0054]

[0055] In the formula, represents the action noise to be applied; and respectively represent the mean and standard deviation of the noise; represents a standard normal distribution random number; Since the noise application objects are the acceleration actions of autonomous vehicles and the phase timing actions of signal lights, the standard deviation , ; where represents that the standard deviation of the acceleration action noise of the autonomous vehicle is 3, and the standard deviation of the timing action of the signal light is 10; To represent the smooth transition of the exploration and exploitation processes of the acceleration of the autonomous vehicle and the timing of the signal light, apply to express the continuously decaying noise value:

[0056]

[0057] In the formula, represents the noise value after the decay effect of ;

[0058] Similarly, denote and as two increasing thresholds, and set their initial values to be and , respectively, and after the increasing step number is , reach their termination values of and respectively; Their updates follow:

[0059]

[0060] In the formula, and are both decay increasing coefficients, and by continuously calculating the increasing coefficients, the increases of and are maintained;

[0061] S4.3. Based on the description in step S4.2, in each training step, the system obtains a random number , and determines which exploration and exploitation strategy to choose by judging which of the following intervals it falls into: 1) When When it is time, enter the random exploration mode. At this time, the lane-changing action of the autonomous vehicle will be randomly selected from left lane-changing, no lane-changing, and right lane-changing, and the acceleration action of the autonomous vehicle and the phase timing action of the traffic signal will also be randomly selected; 2) When When it is time, enter the soft exploitation mode. At this time, the lane-changing action of the autonomous vehicle will be sampled and selected according to the probability distribution given by its action generation network. The acceleration action of the autonomous vehicle and the phase timing action of the traffic signal will be added with the noise value after attenuation described in step S4.2; 3) When it is time, enter the hard exploitation mode. At this time, the lane-changing action of the autonomous vehicle will directly select the lane-changing action with the highest probability according to the probability distribution given by its action generation network. The acceleration action of the autonomous vehicle and the phase timing action of the traffic signal will act on the environment directly without any modification; This multi-level exploration and exploitation strategy can not only provide sufficient exploration in the early training stage, but also gradually strengthen exploitation in the later stage to ensure the convergence effect and generalization performance of the learned strategy; As a preferred technical solution of the present invention, the step S6 includes the following steps:

[0062] S6.1. Based on the experience pool for the autonomous vehicle and the traffic signal in step S5, regularly and uniformly extract a batch of experience samples from it to form two sequences containing a number of samples

[0063] where represents the sampling index; and represents the number of experience entries in one sampling; represents the number of experience entries in one sampling;

[0064] S6.2. As described in step S3, in the training stage, it is necessary to input the samples extracted in step S6.1 into the action generation network and the value evaluation network to obtain the corresponding value estimation and policy gradient information; For the th autonomous vehicle, given the environmental observation data at the th moment, its action generation network predicts a lane-changing action probability distribution and an acceleration action; For the th traffic signal, given the environmental observation data at the th moment, its action generation network predicts the duration of the next phase; After obtaining all the actions of the autonomous vehicle and the traffic signal at the th moment, the environmental observation data at the th moment is merged with them and input into the value evaluation network to obtain the action value of each agent at the th moment. At this time, the target action values of the autonomous vehicle and the traffic signal can be calculated according to the Bellman equation as follows:

[0065] ​

[0066]

[0067] In the formula, is the discount factor, taking 0.95; and respectively represent the target action values of the th autonomous driving vehicle and the th traffic signal; and respectively represent the action values of the th autonomous driving vehicle and the th traffic signal predicted by the value evaluation network at the th moment;

[0068] S6.3. In step S6.2, the present invention respectively calculates the target action values for the traffic signal and the autonomous driving vehicle. For the value evaluation network, the present invention defines the loss function for updating its own weight parameters as:

[0069]

[0070] In the formula, represents the loss of the value evaluation network at the th moment; and respectively represent the action values of the autonomous driving vehicle and the traffic signal predicted by the action value network at the th moment; and respectively represent the total numbers of the autonomous driving vehicle and the traffic signal; at this time, the weight parameters of the value evaluation network can be updated according to minimizing this prediction loss , and the update optimizer selects optimizer, with a learning rate of 0.0001;

[0071] S6.4. After updating the weight parameters of the value evaluation network in step S6.3, input the environmental observation data of the autonomous driving vehicle and the traffic signal at the th moment into their corresponding action generation networks to predict the agent actions in the environment at the th moment, and input the predicted actions and the environmental observation data at the th moment into the value evaluation network again to predict the action value at the th moment. Therefore, the loss of the action generation network of the autonomous driving vehicle can be obtained as:

[0072]

[0073]

[0074] In the formula, represents the loss of the action generation network of the autonomous vehicle at the th moment; represents the lane-changing action probability distribution of the th autonomous vehicle at the th moment; and respectively represent the values of the lane-changing action and the acceleration action of the th autonomous vehicle predicted by the value evaluation network at the th moment; represents the loss of the signal light action generation network at the th moment; represents the value of the phase duration action of the th signal light at the th moment; It can be seen that whether it is the action generation network of the autonomous vehicle or the action generation network of the signal light, its loss value is always optimized in the direction of increasing the action value; The present invention selects optimizer to optimize the weight parameters, and the learning rate is set to 0.0001;

[0075] Compared with the prior art, the present invention provides a complex traffic cooperative control method based on heterogeneous multi-agent reinforcement learning, which has the following beneficial effects:

[0076] Compared with the prior art, the present invention has the following beneficial effects: First, by using a heterogeneous graph structure to perform multi-agent modeling on autonomous vehicles, human-driven vehicles and signal lights, it accurately captures the complex interactions between different types of nodes, and combines two encoders to take into account both time series and single-step observations, greatly enhancing the dynamic scene perception ability. Second, introducing heterogeneous multi-agent deep reinforcement learning and a three-level exploration and exploitation strategy, it realizes fast convergence and efficient cooperation in large-scale mixed traffic scenarios, ensuring sufficient search depth in the early stage and strengthening the stability of the optimal strategy in the later stage. Finally, through decentralized experience pool independent storage and unified value evaluation, it reduces the mutual interference between different agents, and can be flexibly extended and deployed according to different road scales or traffic structures, thus effectively improving driving safety and road traffic efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 is a schematic flow chart of the present invention;

[0078] Figure 2 is a schematic diagram of the multi-level exploration and exploitation strategy of the present invention;

[0079] Figure 3 is a schematic diagram of the heterogeneous multi-agent deep reinforcement learning framework of the present invention; Detailed implementation manners

[0080] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0081] Please refer to Figures 1 - 3 , a complex traffic collaborative control method based on heterogeneous multi-agent reinforcement learning, which is characterized in that it includes the following steps:

[0082] S1. Design a heterogeneous multi-agent graph structure including autonomous vehicles, human-driven vehicles, and traffic lights.

[0083] The step S1 includes the following steps:

[0084] S1.1. The autonomous vehicle and the traffic light identify the surrounding objects, i.e., human-driven vehicles, and judge whether they should be linked to each other according to the distance threshold . If the distance is less than the distance threshold, it is considered that the autonomous vehicle or the traffic light should be connected to the human-driven vehicle; otherwise, they are not connected. Since the node types in this structure are not completely the same, the generated graph structure belongs to a heterogeneous graph.

[0085] S1.2. Convert the heterogeneous graph in step S1.1 into a mathematical expression: use to represent the set of autonomous vehicle nodes, where represents the autonomous vehicle numbered , …… ; represents the number of autonomous vehicles; use to represent the set of human-driven vehicle nodes, where represents the human-driven vehicle numbered , …… ; represents the number of human-driven vehicles; use to represent the set of traffic light nodes, where represents the traffic light numbered , …… ; Therefore, the overall node set can be represented as ; Therefore, for any three types of nodes in this heterogeneous graph, if the spatial Euclidean distance between any two nodes is less than the set distance threshold , then connect these two nodes in the heterogeneous graph structure and set the weight of the connection edge to 1; then obtain the adjacency matrix for the heterogeneous graph and the feature matrix , the elements in the adjacency matrix are: :

[0086]

[0087] In the formula, the subscript represents the th autonomous vehicle, the th human-driven vehicle, and the th traffic signal. represents the connection situation between the th autonomous vehicle, the th human-driven vehicle, and the th traffic signal; and obtain the heterogeneous feature matrix , where is the feature information of the autonomous vehicle, and represent the th autonomous vehicle's , coordinates; and respectively represent the speed and acceleration of the th autonomous vehicle; represents the lane where the th autonomous vehicle is located; represents the feature information of the human-driven vehicle, and represent the th human-driven vehicle's , coordinates; and respectively represent the speed and acceleration of the th human-driven vehicle; represents the lane where the th human-driven vehicle is located; , where represents the current phase of the th traffic signal, represents the time required for the th traffic signal to switch to the next phase;

[0088] S2. Based on the heterogeneous multi-agent graph structure in step S1, design a heterogeneous observation encoder for autonomous vehicles and traffic signals, input the environmental observation data of the heterogeneous multi-agent, and output the encoded environmental observation data.

[0089] Step S2 includes the following steps:

[0090] S2.1. Since there are differences between autonomous vehicles and traffic lights in terms of attributes, observation dimensions, etc., design a heterogeneous encoder to extract different features for different types of nodes, enabling it to uniformly output high-dimensional representations for subsequent decision-making networks:

[0091]

[0092] In the formula, represents the heterogeneous encoder mapping process; are the parameters of the encoder; is the encoded feature;

[0093] S2.2. For the basic principle of the encoder described in step S2.1, design modules such as multi-head self-attention graph convolution, ordinary graph convolution, heterogeneous graph convolution, and time series convolution as the main structures of the observation encoder for traffic light type nodes; among them, the calculation process of multi-head self-attention graph convolution can be expressed as:

[0094]

[0095] In the formula, represents the environmental observation data of the th autonomous vehicle or the th human-driven vehicle or the th traffic light in the heterogeneous graph; represents the environmental observation data of an agent with a different node type from , and the subscript represents a node type different from ; is the learnable linear mapping matrix of multi-head self-attention graph convolution; is the attention weight vector; represents the unnormalized attention coefficient vector; represents activation function; represents activation function; represents activation function; is the attention coefficient normalized by the activation function; represents the environmental observation data of the th autonomous vehicle or the th human-driven vehicle or the th traffic light after multi-head self-attention graph convolution; for ordinary graph convolution, its calculation process can be expressed as:

[0096]

[0097] In the formula, and respectively represent the learnable weight matrix and bias term of the ordinary graph convolution, is the environmental observation data of the th autonomous vehicle or the th human-driven vehicle or the th traffic signal after passing through the ordinary graph convolution; represents the set of other nodes that are connected to the node represented by the th autonomous vehicle or the th human-driven vehicle or the th traffic signal in the heterogeneous graph structure;

[0098] Based on the above two forms of graph convolution, the heterogeneous graph convolution process of the present invention is constructed as follows: An edge relationship is constructed between any two node types to represent the action relationship of one node type on another node type; Therefore, for any edge type, in one information aggregation and propagation process, the process of heterogeneous graph convolution can be expressed as:

[0099]

[0100] In the formula, represents the environmental observation data of the th autonomous vehicle or the th human-driven vehicle or the th traffic signal after passing through the heterogeneous graph convolution; represents the graph convolution operator acting on different edge types, which can be specifically expressed as the above-mentioned multi-head self-attention graph convolution process and ordinary graph convolution process in the present invention; Since one node type may establish edge relationships with multiple other types of nodes, it is necessary to unify the outputs of different convolution operators to the same dimension and perform aggregation, which can be specifically expressed as:

[0101]

[0102] In the formula, represents the sum aggregation function; represents the environmental observation data after the global encoding of the traffic signal after aggregation;

[0103] Furthermore, the node features processed by the heterogeneous graph convolution are input into the network:

[0104]

[0105] In the formula, and respectively represent the update gate and the reset gate at the moment. The update gate is used to determine how much information at the moment needs to be retained, and the reset gate determines to what extent the hidden state at the moment is ignored; and represent the parameter matrix and bias vector of the update gate; represents the hidden state at the moment; represents the environmental observation data of each node after heterogeneous graph convolution at the moment; represents the activation function; and respectively represent the weight parameter and bias vector of the reset gate; the reset gate is used to control the dependency degree and calculate the candidate hidden state ; represents the activation function; and are the parameter matrix and bias of the candidate hidden state; after the above steps, the environmental observation data of the traffic signal can realize the high-dimensional feature space embedding encoding of the original environmental observation data;

[0106] S2.3. For autonomous vehicles, their observations have a strong real-time feedback property, that is, the intelligent agent learning strategy of autonomous vehicles needs to depend on the single-step "state-action-reward" set. Therefore, the encoder of autonomous vehicles is designed as a static single-step heterogeneous graph encoder, which inputs the environmental observation data at the current time step of each autonomous vehicle and outputs the encoded environmental observation data of all autonomous vehicles at the current time step. Specifically, as described in step S2.2, the same heterogeneous graph convolution operation is used to implement the convolution of the environmental observation data of autonomous vehicles, and the environmental observation data of autonomous vehicles after heterogeneous graph convolution is denoted as , and then the non-linear expression ability is enhanced through a linear mapping:

[0107]

[0108] In the formula, is the weight parameter of the linear mapping layer; ​Encoding environmental observation data for an autonomous vehicle after linear mapping; through the two heterogeneous graph encoder structures shown in the above steps S2.1 to S2.3, effective feature abstraction and fusion can be performed on heterogeneous intelligent agents such as autonomous vehicles, human-driven vehicles, and traffic lights, providing unified and highly expressive input features for the subsequent action generation network and value evaluation network;

[0109] S3. Design an action generation network and a corresponding value evaluation network for heterogeneous multi-intelligent agents, sharing the heterogeneous observation encoder in step S2 for the action generation network and the value evaluation network, that is, the action generation network outputs the actions of heterogeneous multi-intelligent agents based on the encoded environmental observation data, and the value evaluation network predicts the corresponding value based on the actions of heterogeneous multi-intelligent agents.

[0110] The above step S3 includes the following steps:

[0111] S3.1. Share the defined encoder part in step S2 between the "action generation network" and the "value evaluation network" to reduce model redundancy and ensure encoding consistency. That is, design a heterogeneous graph convolutional layer to further fuse the encoded heterogeneous graph features:

[0112]

[0113] where represents the heterogeneous graph convolution process described in step S2.2, represents the shared high-dimensional feature representation after heterogeneous graph convolution;

[0114] S3.2. Design an action generation network for autonomous vehicles and traffic lights, which can be specifically described as: for autonomous vehicles: the actions include the action probability distribution of acceleration and lane change; for traffic lights: the action can be the signal timing for the next stage. Use a multi-layer perceptron as the action generation model as follows:

[0115]

[0116] where represents the action of the th autonomous vehicle, specifically including the lane change action of the th autonomous vehicle and the acceleration action of the th autonomous vehicle , where is a probability distribution quantity, and , where each element represents the probability of each strategy in the strategy set , and in the strategy set , , represent left lane change, no lane change, and right lane change respectively; represents the action generation network mapping process of the th autonomous vehicle, is the th signal timing for the next stage of the traffic light set; represents the action generation network mapping process of the th traffic light,

[0117] S3.3. The value evaluation network is used to evaluate the value of the current agent's action decision. For the heterogeneous multi-agent scenario of the present invention, a centralized value evaluation network mode is adopted. The actions and state observations of all agents are input into the value evaluation network to obtain the action values corresponding to each agent:

[0118]

[0119] In the formula, represents the value brought by the actions of each autonomous vehicle and traffic light; represents the input of the globally shared feature after secondary encoding , the actions of all autonomous vehicles and the actions of all traffic lights so as to output the mapping process of the action value of each agent; is the weight parameter of this centralized value evaluation network.

[0120] S4. As Figure 2 shown, a multi-level exploration and exploitation strategy suitable for the actions of heterogeneous multi-agents is constructed, and the output exploration value is used to regularize the exploration of the actions generated by the action generation network in step S3.

[0121] The said step S4 includes the following steps:

[0122] S4.1. In multi - heterogeneous agent reinforcement learning, to balance the exploration depth and policy stability for multi - dimensional vehicle and signal light control, the present invention proposes a multi - level exploration and exploitation strategy. This strategy switches the heterogeneous agent among the following three modes by defining and dynamically regulating three threshold parameters: 1) Random exploration: completely random action sampling, indicating that the agent conducts action exploration; 2) Soft exploitation: sampling actions according to the discrete action probability distribution output by the action generation network; Hard exploitation: directly selecting the action with the highest probability output by the action generation network. For the lane - changing actions of autonomous vehicles, in order to represent the exploration and exploitation mechanisms of the acceleration actions of autonomous vehicles and the phase timing actions of signal lights, the present invention also introduces a noise attenuation strategy, which continuously attenuates a threshold to make continuous actions increasingly close to the actions generated by the action generation network.

[0123] S4.2. Based on the basic description in step S4.1, detailed threshold definitions and theoretical elaborations are carried out; denote as a decay threshold, whose initial value is set to and its final decay value is , and the number of decay steps is . Therefore, as the agent is trained, the decay of

[0124]

[0125] can be specifically expressed as: where is a decay coefficient, and by continuously calculating the decay coefficient, the decay of

[0126]

[0127] is maintained; this parameter is mainly used in the exploration and exploitation processes of the acceleration actions of autonomous vehicles and the phase timing actions of signal lights. Specifically, a random Gaussian noise generator is designed as follows: where and represent the action noise to be applied, the mean and standard deviation of the noise respectively; represents a standard normal distribution random number; since the noise application objects are the acceleration actions of autonomous vehicles and the phase timing actions of signal lights, the standard deviation is taken, ; where means that the standard deviation of the acceleration action noise of the autonomous vehicle is 3, and the standard deviation of the signal light timing action is 10; to represent the smooth transition of the exploration and exploitation processes of the acceleration and signal light timing actions of autonomous vehicles, the noise application is used to represent the continuously decaying noise value:

[0128]

[0129] In the formula, represents the noise value after attenuation;

[0130] Similarly, denote and as two increasing thresholds respectively, and set their initial values to be and respectively, and after the increasing step number is , their termination values reach and respectively; their update follows:

[0131]

[0132] In the formula, and are both attenuation increasing coefficients, and by continuously calculating the increasing coefficients, the increase of and is maintained;

[0133] S4.3. Based on the description in step S4.2, in each training step, the system obtains a random number , and determines which of the following intervals it falls into to determine which exploration and exploitation strategy to choose: 1) When , enter the random exploration mode. At this time, the lane-changing action of the autonomous vehicle will be randomly selected from left lane-changing, no lane-changing, and right lane-changing, and the acceleration action of the autonomous vehicle and the phase timing action of the traffic signal will also be randomly selected; 2) When , enter the soft exploitation mode. At this time, the lane-changing action of the autonomous vehicle will be sampled and selected according to the probability distribution given by its action generation network, and the acceleration action of the autonomous vehicle and the phase timing action of the traffic signal will be added with the noise value after attenuation described in step S4.2; 3) When , enter the hard exploitation mode. At this time, the lane-changing action of the autonomous vehicle will directly select the lane-changing action with the highest probability according to the probability distribution given by its action generation network, and the acceleration action of the autonomous vehicle and the phase timing action of the traffic signal will act on the environment without any modification; This multi-level exploration and exploitation strategy can not only provide sufficient exploration in the early training stage, but also gradually strengthen exploitation in the later stage to ensure the convergence effect and generalization performance of the learned strategy.

[0134] S5. Design two experience pools for autonomous vehicles and traffic lights, and store the environmental observation data, actions, and rewards of the agents. Specifically, for each heterogeneous agent (autonomous vehicle, traffic light), the network outputs an action according to its action generation network. After the environment executes this action, new observations and corresponding rewards are obtained, and the quadruple information at the th moment is stored in the experience pool. For autonomous vehicles, record the quadruple information at the th moment and store it in the experience pool of the autonomous vehicle, where , , and respectively represent the environmental observation data, the action generated by the action generation network, the obtained reward, and the th moment of the th autonomous vehicle, and the environmental observation data of the th moment of the th autonomous vehicle; for traffic lights, similarly record the quadruple information at the th moment and store it in the experience pool of the traffic light, where , , and respectively represent the environmental observation data, the action generated by the action generation network, the obtained reward, and the th moment of the th traffic light, and the environmental observation data of the th moment of the th traffic light; it should be noted that in order to avoid the untimely feedback of the reward signal, the present invention stores the experience of autonomous vehicles once per step, while the experience of traffic lights is stored once after the end of one phase cycle;

[0135] S6. As shown in Figure 3 , establish a heterogeneous multi-agent deep reinforcement learning framework: during the learning process, collect the experience of heterogeneous multi-agents, input it into the action generation network designed in step S3, and obtain the action value through the corresponding value evaluation network. Update the value evaluation network according to the accuracy of the value evaluation, and update the action generation network according to the magnitude of the value of the action generated by the action generation network, so as to continuously iterate to realize the update of the network, that is, the action learning of heterogeneous multi-agents.

[0136] The said step S6 includes the following steps:

[0137] S6.1. Based on the experience pools of autonomous vehicles and traffic lights in step S5, regularly and uniformly extract a batch of experience samples from them to form two sequences containing several samples , , where represents the sampling index; represents the number of empirical entries for one sampling;

[0138] S6.2. As described in step S3, in the training phase, the samples extracted in step S6.1 need to be input into the action generation network and the value evaluation network to obtain the corresponding value estimation and policy gradient information; for the th autonomous driving vehicle, given the environmental observation data at the th moment, its action generation network predicts a lane-changing action probability distribution and an acceleration action; for the th traffic signal, given the environmental observation data at the th moment, its action generation network predicts the duration of the next phase; after obtaining all the actions of the autonomous driving vehicle and the traffic signal at the th moment, the environmental observation data at the th moment is merged with them and input into the value evaluation network to obtain the action value of each agent at the th moment. At this time, the target action values of the autonomous driving vehicle and the traffic signal can be calculated according to the Bellman equation as follows:

[0139]

[0140]

[0141] In the formula, is the discount factor, taking 0.95; and respectively represent the target action values of the th autonomous driving vehicle and the th traffic signal; and respectively represent the action values of the th autonomous driving vehicle and the th traffic signal predicted by the value evaluation network at the th moment;

[0142] S6.3. In step S6.2, the present invention respectively realizes the calculation of the target action values for the traffic signal and the autonomous driving vehicle. For the value evaluation network, the present invention defines its loss function for updating its own weight parameters as:

[0143]

[0144] In the formula, represents the loss of the value evaluation network at the th moment; and respectively represent the action values of the autonomous vehicle and the traffic signal predicted by the action value network at the moment; and respectively represent the total numbers of the autonomous vehicle and the traffic signal; at this time, the weight parameters of the value evaluation network can be updated according to minimizing the prediction loss , and the update optimizer selects optimizer with a learning rate of 0.0001;

[0145] S6.4. After the weight parameters of the value evaluation network are updated in step S6.3, input the environmental observation data of the autonomous vehicle and the traffic signal at the moment into their corresponding action generation networks to predict the agent action in the environment at the moment, and input the predicted action and the environmental observation data at the moment into the value evaluation network again to predict the action value at the moment. Therefore, the loss of the action generation network of the autonomous vehicle can be obtained as:

[0146]

[0147]

[0148] In the formula, represents the loss of the action generation network of the autonomous vehicle at the moment; represents the lane-changing action probability distribution of the th autonomous vehicle at the moment; and respectively represent the values of the lane-changing action and the acceleration action of the th autonomous vehicle predicted by the value evaluation network at the moment; represents the loss of the traffic signal action generation network at the moment; represents the value of the phase duration action of the th traffic signal at the moment; it can be seen that whether it is the action generation network of the autonomous vehicle or the action generation network of the traffic signal, their loss values are always optimized in the direction of increasing the action value; for all action generation networks in the present invention, optimizer is selected to optimize the weight parameters, and the learning rate is set to 0.0001.

[0149] S7. Save the model parameters after learning is completed, load and deploy the heterogeneous multi-agent collaborative control model, and input the environmental observation data of each heterogeneous multi-agent to output the control quantities of all controlled objects;

[0150] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A complex traffic cooperative control method based on heterogeneous multi-agent reinforcement learning, characterized in that: The following steps are involved: S1. Constructing heterogeneous multi-agent environmental observation data based on heterogeneous graph structures including autonomous vehicles, human-driven vehicles, and traffic lights; S2. Based on the heterogeneous multi-agent graph structure in step S1, a heterogeneous observation encoder for the autonomous driving vehicle and the traffic light is designed, the environmental observation data of the heterogeneous multi-agent is input, and the encoded environmental observation data is output; S3. Design an action generation network for heterogeneous multi-agents and a corresponding value evaluation network. The action generation network outputs the actions of the heterogeneous multi-agents according to the encoded environmental observation data. The value evaluation network predicts the corresponding values ​​based on the actions of the heterogeneous multi-agents. S4, construct a multi-level exploration and utilization strategy suitable for the actions of heterogeneous multi-agents, and the output exploration value acts on the actions generated by the action generation network in step S3 to perform exploration regularization; S5. Design two experience pools for autonomous vehicles and traffic lights to store the agent's environmental observation data, actions, and rewards. Each heterogeneous agent generates a network output action based on its action. After the environment executes the action, it obtains new observation data and corresponding rewards, and stores the four-tuple information at time t into the experience pool. S6. Establish a heterogeneous multi-agent deep reinforcement learning framework. During the learning process, collect the experience data of the heterogeneous multi-agent, input it into the action generation network designed in step S3, and obtain the action value through the corresponding value evaluation network. Update the value evaluation network according to the accuracy of the value evaluation, and update the action generation network according to the value of the action generated by the action generation network. In this way, the network is updated continuously iteratively, i.e., the action learning of the heterogeneous multi-agent; S7. Save the learned model parameters, load and deploy the heterogeneous multi-agent collaborative control model, input the environmental observation data of each heterogeneous multi-agent, output the control quantity of all controlled objects, and realize the heterogeneous multi-agent collaborative control in the mixed traffic flow scenario.

2. According to claim 1, a complex traffic coordinated control method based on heterogeneous multi-agent reinforcement learning is characterized in that: The step S1 comprises the following steps: S1.

1. Autonomous vehicles and traffic lights recognize the distance between surrounding objects and human-driven vehicles and Determine whether they should be linked to each other. If the distance is less than the threshold, it is considered that the autonomous driving vehicle or signal light should be connected to the human-driven vehicle, otherwise it is not connected; S1.

2. Convert the heterogeneous graph in step S1.1 into a mathematical expression: represents the set of autonomous driving vehicle nodes, where Indicates the number , … of self-driving vehicles, represents the number of autonomous vehicles; represents the set of human-driven vehicle nodes, where Indicates the number , … of human-driven vehicles, Indicates the number of human-driven vehicles; Represents a set of traffic light nodes, where Indicates the number , … The signal light; therefore, the overall node set can be expressed as ; Therefore, for any three types of nodes in the heterogeneous graph, if the spatial Euclidean distance between any two nodes is Less than the set distance threshold , then in the heterogeneous graph structure, the two nodes are connected and the connection edge weight is set equal to 1; then the adjacency matrix for the heterogeneous graph is obtained and the feature matrix , the adjacency matrix Elements in : ； In the formula, the superscript Indicates Self-driving cars, Human-driven car and A signal light, Indicates Self-driving cars, Human-driven car and The connection between the traffic lights; and obtain the heterogeneous feature matrix ,in is the characteristic information of the autonomous driving vehicle, and Indicates of autonomous vehicles , coordinate; and Respectively represent The speed and acceleration of the autonomous vehicle; Indicates The lane where the autonomous vehicle is located; Represents the characteristic information of human driving the vehicle, and Indicates Human-driven vehicles , coordinate; and Respectively represent The speed and acceleration of the human-driven vehicle; Indicates The lane where the human-driven vehicle is located; ,in Indicates The current phase of the signal light. Indicates The time required for a signal light to switch to the next phase.

3. According to claim 1, a complex traffic coordinated control method based on heterogeneous multi-agent reinforcement learning is characterized in that: The specific process of step S2 is as follows: S2.

1. Since autonomous vehicles and traffic lights differ in attributes and observation dimensions, a heterogeneous encoder is designed to extract different features from different types of nodes, so that they can uniformly output high-dimensional representations for subsequent decision networks: ； In the formula, represents the heterogeneous encoder mapping process; is the parameter of the encoder; is the encoded feature; S2.

2. Design multi-head self-attention graph convolution, common graph convolution, heterogeneous graph convolution and time series convolution modules as the main structure of the observation encoder of the traffic light type node; S2.

3. For autonomous driving vehicles, the encoder of the autonomous driving vehicle is designed as a static single-step heterogeneous graph encoder, which inputs the environmental observation data of the current time step of each autonomous driving vehicle and outputs the encoded environmental observation data of the current time step of all autonomous driving vehicles.

4. According to claim 1, a complex traffic coordinated control method based on heterogeneous multi-agent reinforcement learning is characterized in that: The specific process of step S3 is as follows: S3.

1. Design a heterogeneous graph convolution layer to further fuse the encoded heterogeneous graph features so that the heterogeneous observation encoder in step S2 is shared between the action generation network and the value assessment network. The heterogeneous graph convolution process of the heterogeneous graph convolution layer is expressed as follows: ； In the formula, represents the heterogeneous graph convolution process, Represents the shared high-dimensional feature representation after heterogeneous graph convolution; S3.

2. Design the action generation network for the autonomous vehicle and the traffic light. Specifically, it can be expressed as follows: for the autonomous vehicle: the action includes the probability distribution of acceleration and lane change; for the traffic light: the action can be the signal timing for the next stage. The multi-layer perceptron is used as the action generation model as follows: ； In the formula, Indicates The actions of the autonomous vehicle include Lane changing maneuvers of an autonomous vehicle and The acceleration of a self-driving car ,in is a probability distribution, and , where each element represents a set of strategies The probability of each strategy, the strategy set , , They represent left lane change, no lane change, and right lane change respectively; Indicates The process of generating a network map of the actions of a self-driving car. It is The weight parameters of the action generation network of an autonomous vehicle; For the The signal timing of the next stage of the signal A collection of; express The action of each signal light generates a network mapping process. It is The weight parameters of the action generation network of each signal light; S3.

3. Input the actions and state observations of all agents into the value evaluation network to obtain the action value corresponding to each agent. The expression is as follows: ； In the formula, Represents the value of each action taken by the autonomous vehicle and the traffic light; Represents the global shared features of the input after secondary encoding , all actions of the autonomous vehicle And all the signal lights Thus, the mapping process of the action value of each agent is output; The weight parameters of the network are evaluated for this centralized value.

5. According to claim 1, a complex traffic coordinated control method based on heterogeneous multi-agent reinforcement learning is characterized in that: For autonomous vehicles, record Quadruple information at the moment Stored in the experience pool of the autonomous driving vehicle, where , , as well as Respectively Moment The environment observation data of the autonomous vehicle, the actions generated by the action generation network, the rewards obtained, and Moment Environmental observation data of autonomous vehicles; For signal lights, the Quadruple information at the moment The experience pool stored in the signal light, where , , as well as Respectively Moment The environmental observation data of the traffic lights, the actions generated by the action generation network, the rewards obtained, and Moment Environmental observation data of a traffic light.

6. The complex traffic coordinated control method based on heterogeneous multi-agent reinforcement learning according to claim 1 is characterized in that: In step S4, three threshold parameters are defined and dynamically adjusted through a multi-level exploration and utilization strategy, so that the heterogeneous agent switches between the following three modes: 1) random exploration: completely random action sampling, indicating that the agent performs action exploration; 2) soft utilization: sampling actions according to the discrete action probability distribution output by the action generation network; 3) hard utilization: directly selecting the action with the highest probability output by the action generation network; the above three modes are aimed at the lane changing action of the autonomous driving car, indicating the exploration and utilization mechanism of the acceleration action of the autonomous driving car and the phase timing action of the traffic light; Moreover, the noise attenuation strategy continuously attenuates a threshold value to achieve continuous actions that are closer and closer to the actions generated by the action generation network. In the process of multi-heterogeneous agent reinforcement learning, in each training step, the system obtains a random number : when , it enters the random exploration mode. At this time, the lane change action of the autonomous driving vehicle will be randomly selected from left lane change, no lane change, and right lane change. The acceleration action of the autonomous driving vehicle and the phase timing action of the traffic light will also be randomly selected. when , the soft utilization mode is entered. At this time, the lane change action of the autonomous driving vehicle will be sampled and selected according to the probability distribution given by its action generation network. The acceleration action of the autonomous driving vehicle and the phase timing action of the traffic light are added with the process described in step S4.

2. Noise value after attenuation; when When the vehicle enters the hard utilization mode, the lane-changing action of the autonomous vehicle will directly select the lane-changing action with the maximum probability according to the probability distribution given by its action generation network, and the acceleration action of the autonomous vehicle and the phase timing action of the traffic light will directly act on the environment without any changes; in and There are two increasing thresholds respectively.

Images