Expressway super-large-scale road network management and control response method based on heterogeneous multi-agent reinforcement learning

Abstract

The invention discloses a highway super-large-scale road network management and control response method based on heterogeneous multi-agent reinforcement learning, and the method comprises the steps: dividing a super-large-scale road network based on a graph theory and a fuzzy C-means clustering algorithm to form different sub-road network agents, and constructing a multi-agent reinforcement learning method to search for an optimal strategy for each sub-road network. Traffic event parameters are abstracted, macroscopic, mesoscopic and microscopic traffic state input of each intelligently formulated sub-road network and a lane control strategy set of the sub-road network are determined, and a reward function form is comprehensively determined through average speed, accident punishment and vehicle emission. Furthermore, on the basis of an entropy function of a maximized joint strategy and meeting security constraints, a heterogeneous agent soft actor-commentator algorithm is constructed to construct a commentator and actor network, and action constraints are met during full exploration; and finally, asynchronously updating the network parameters of the actor, and carrying out soft updating on the reviewer network so as to iteratively obtain an optimal sub-road network lane strategy.

Description

Technical Field

[0001] The present invention belongs to the fields of vehicle safety, traffic management and machine learning, and in particular relates to a control and response method for ultra-large-scale highway networks based on heterogeneous multi-agent reinforcement learning. Background Art

[0002] Expressways are a vital component of China's comprehensive, three-dimensional transportation network, playing a vital role in promoting economic and social development, improving people's living standards, ensuring national security, and facilitating emergency rescue. With the rapid development of my country's expressway network, the scale, density, complexity, and operational pressures of expressways have continued to increase, posing significant challenges to their management and services. Expressways operate in a diverse and complex environment. Compared to urban road networks, expressways have a larger scope and fewer control mechanisms. Coupled with lower network density and fewer alternative routes, compliance with management is often low, placing higher demands on the precision and targeted nature of traffic control.

[0003] There are a large number of heterogeneous road networks in the highway network. That is, the number of lanes and road conditions of different sub-networks vary, resulting in inconsistent action spaces. Some sub-networks have more lanes and a richer range of adjustable lane control strategies, such as more options in dynamic speed limits and lane opening hours; while sub-networks with fewer lanes have relatively limited action options.

[0004] At present, there are still some problems and challenges in the development of active traffic management and control technology on my country's expressways. There is a lack of fully effective coordination mechanisms and execution capabilities, as well as customized solutions for different scenarios and needs. This makes it difficult to achieve effective intervention and management of the traffic system, and difficult to achieve effective guidance and incentives for traffic participants.

[0005] Existing reinforcement learning-driven learning strategies mostly use homogeneous agents. Homogeneous agent algorithms such as MAPPO and MASAC are not adaptable to heterogeneous road networks (for example, the control strategies in different road network areas may be different). Heterogeneous intelligent algorithms HAPPO and HAPPDG lack explicit safety constraints on traffic rules (such as speed limits). In addition, their synchronous update mechanisms are prone to policy conflicts in large-scale road networks, resulting in low training efficiency.

[0006] Existing research strategies are unable to fully utilize network-level data and information to accurately describe, predict, evaluate, and optimize complex traffic systems under various factors and constraints, such as demand, traffic supply, and traffic safety. This makes it difficult to achieve customized solutions for different scenarios and needs. It is necessary to conduct in-depth research on the digital description of traffic events at the network level, both theoretically and practically, to achieve multi-objective, multi-level, multi-agent, and multi-strategy collaboration and coordination, such as traffic control at the network, regional, section, and intersection levels. This will enable the development of optimal, personalized traffic control measures, such as speed limits, diversions, guidance, signaling, tolls, and services, to inform traffic decision-making for governments, operators, vehicles, drivers, and passengers. Summary of the Invention

[0007] To address the challenges of existing technologies, this paper proposes a heterogeneous multi-agent reinforcement learning-based approach for ultra-large-scale highway network control and response. By constructing a heterogeneous multi-agent reinforcement learning framework, this approach decouples the ultra-large-scale network control problem into a two-stage task: subnetwork partitioning and collaborative optimization. This innovative approach utilizes a dynamic network partitioning method that combines graph-theoretic features (node ​​weights / adjacency relationships) with fuzzy C-means clustering. Furthermore, a constrained heterogeneous soft actor-critic algorithm is designed. Through the dual mechanisms of maximum entropy policy optimization and safety constraints, this approach enables collaborative decision-making regarding lane speed limits and closures across subnetworks.

[0008] Technical Solution A super-large-scale highway network control and response method based on heterogeneous multi-agent reinforcement learning includes the following steps: S1: Model the highway network as a weighted directed graph, set the network partition optimization objective function, and use fuzzy C-means clustering to generate the optimal sub-network partitioning scheme.

[0009] S2: Define the sub-network agent and control the state input traffic parameters, road parameters and environmental parameters.

[0010] Constructing an action space that includes the speed limit and open state of each lane; The designed reward function integrates global traffic volume, collision penalty and emission penalty.

[0011] S3: Define a joint policy entropy function to maximize the benefits of agent collaboration and introduce a safety constraint function to limit dangerous actions. A heterogeneous agent soft actor-critic algorithm is used to establish an actor network and a critic network. Before formal training begins, the actor and critic networks are assigned random network parameters and interact with the environment to generate state-action-value data to populate the experience replay pool.

[0012] S4: Deploy an asynchronous training framework, with multiple actor network threads executing in parallel and interacting with the environment: each thread generates sub-network actions based on the current actor network, collects state-reward data and calculates gradients, and asynchronously updates the actor network parameters of each sub-network; the critic network uses a soft update mechanism to mix the current network and target network parameters in a fixed ratio.

[0013] S5: Randomly sample batches of data from the experience pool, calculate the temporal difference error of the critic network, update its parameters, and optimize the action probability distribution through policy gradient ascent. During the iterative process, the safety constraint verification module is used to filter conflicting actions, ultimately outputting a lane-level control policy that complies with cross-region coordination constraints.

[0014] Specifically, step S1 is a sub-network division method based on graph theory and fuzzy C-means clustering: Ultra-large-scale road networks involve a large amount of geographic information, traffic flow, signal control and other data. There are complex interactions and influences between road network structure, traffic demand, traffic control, traffic behavior, and traffic environment factors. Analysis and modeling involve a large amount of calculation and simulation. Therefore, dividing the road network into sub-networks can reduce the scale and complexity of the road network, enhance the robustness and fault tolerance of the road network, adapt to the dynamic changes and growth of the road network, and thus improve the efficiency of road network analysis and management.

[0015] The present invention is based on graph theory and fuzzy C-means clustering, and its purpose is to divide a large road network into several small sub-road networks, so that the nodes in each sub-road network have a higher similarity, while the nodes between different sub-road networks have a lower similarity.

[0016] Similarity is measured using a node's weight and neighbor set. The weight represents the node's traffic volume or importance, while the neighbor set represents the nodes connected to it. A higher weight and a larger neighbor set indicate a more important node, necessitating its own subnetwork.

[0017] Fuzzy C-means clustering is a clustering algorithm that can assign nodes to one or more subnetworks, rather than just one, to better reflect the complexity and diversity of nodes. The fuzzy C-means clustering process continuously updates the node membership and subnetwork centers until an optimal partitioning result is achieved.

[0018] The optimal partitioning result is one that minimizes a network partitioning optimization objective function. This objective function is composed of the node weight and the set of neighbors, and reflects the distance and connectivity between the node and the subnetwork. The smaller the distance, the higher the connectivity, indicating that the node and the subnetwork are more similar, and the smaller the objective function.

[0019] Specifically, step S2 is a highway control response model based on the soft actor-critic algorithm of the constrained heterogeneous sub-road network intelligent agent. When making decisions, each sub-network manager only needs to consider the current road conditions and traffic flow status, which is not related to the historical traffic flow status. Therefore, the management and control response problem based on the highway network status has Markov properties and can be modeled using reinforcement learning.

[0020] For each sub-network obtained in step S1, a sub-network agent is constructed, and the state, action, and reward function of the sub-network agent are determined as follows: State function: Abstract typical traffic event parameters and determine the road network traffic status input that can be obtained by the sub-road network.

[0021] Status items include: traffic parameters, road parameters and environmental parameters.

[0022] Among them, traffic parameters are obtained through gantry video, including the average speed and average density in each section of the sub-road network; Road parameters are obtained through road administration data and road surface detectors, including lane width and number of lanes in each section of the sub-road network; Environmental parameters include weather type, precipitation intensity, visibility distance, temperature and humidity of the sub-road network.

[0023] Action function: Based on the typical active control scheme, the sub-road network strategy set with lanes (speed, opening time) within the sub-road network as the control object is determined.

[0024] Among them, the constraint network control objects include: Dynamic speed limit (speed km / h); Lane opening status (closure time h, select 1 road section in the sub-road network to be closed (id)).

[0025] Reward function: According to the reward function y based on the global road network flow and sub-road network flow, the specific form is Among them, w1, w2, w3 are the weights of each part of the reward, ; is the average speed, is the speed of the i-th car, is the total number of vehicles; is the number of vehicles involved in the collision; Average vehicle emissions ,in is the exhaust gas emission (g / km), is the emission factor (g / km) for the i-th vehicle type under the j-th fuel and the k-th pollution control level, is the mileage (km) of the i-th type of vehicle using the j-th fuel and the k-th pollution control level, is the frequency distribution of the i-th vehicle under the j-th fuel and the k-th pollution control level. is the evaporative emission (g), is the evaporative emission rate (g / time) of the i-th type of vehicle under the j-th fuel and the k-th pollution control level, is the number of times the i-th type of vehicle travels under the j-th fuel and the k-th pollution control level, To calculate the time.

[0026] Specifically, step S3: Since different sub-networks have different numbers of lanes, their action spaces are inconsistent. Therefore, the collaborative optimization problem of agents in different sub-networks can be modeled as a heterogeneous strategy learning problem.

[0027] A heterogeneous agent soft actorcritic (HASAC) algorithm is designed. An action maximum entropy regularization term is added to the loss function in the actor network, and a sequential update scheme is adopted, that is, only the strategy of one sub-network agent is updated at a time, while the strategies of other sub-network agents remain unchanged.

[0028] Among them, the actor network: The actor network is based on the agent state function as input, the action function as output, and the action value composed of the reward function as the optimization target.

[0029] Maximum entropy term: In order to prevent the sub-network agent from falling into a single strategy choice within the established safety constraints and losing the exploration of the environment, The sub-network agent is constructed, and the maximum entropy term of the action is used as the optimization goal of the actor network to avoid the action value output by the strategy being too concentrated. The maximum entropy term can be expressed as: in, is the state distribution, It is The strategy of each sub-network agent, are the strategies of other subnetwork agents; It is The action value function (joint action-state value function) of the sub-network agent, that is, given a state , No. Actions of the sub-network agents , and other subnetwork agents' strategies , predict the The expected reward of each sub-network agent.

[0030] Security constraint function: When executing an action, the following constraints need to be met: in, It is The constraint function of each sub-network agent, It is The constraint threshold of each sub-network agent, the sub-network action execution needs to meet the following constraints: 1) Dynamic speed limit The constraints are: in, is the set of all lanes; It's a lane Speed ​​range; and Lanes minimum and maximum speed limits; Represents the set of all special events; It's a special event The speed range is specified below.

[0031] 2) Lane opening status The constraints are the road network connectivity conditions: in, is the set of all lanes in the road network; It is a temporarily closed lane; out( )enter( ) port path; Entrance and exit collection.

[0032] Construct an actor network to output the random strategy of each sub-network agent, that is, given the state and the strategy of other sub-network agents, output the action probability distribution of the current sub-network agent. The parameters of the actor network are updated by the following formula: in, It is The parameters of the actor network of the sub-network agents, is the learning rate, It is The objective function of the sub-network agent is to maximize the The policy entropy and expected reward (including the maximum entropy regularization term) of each sub-network agent are calculated by the following formula: in, is the state distribution, It is The joint action-state value function of the sub-network agents, that is, given a state , No. Actions of the sub-network agents , and other subnetwork agents' strategies , predict the The expected reward of each sub-network agent.

[0033] Construct a constraint network to estimate the safety constraint function of each sub-network agent. That is, given the state and the strategies of all sub-network agents, directly output the actions that meet the constraints; for actions that do not meet the constraints, output any actions that meet the constraints and the constrained actions of the current sub-network agent. The parameters of the constraint network are updated by the following formula: in, It is The parameters of the constraint network of the sub-network agents, is the learning rate, It is The loss function of the constraint network of the sub-network agent, namely the log-likelihood loss, can be calculated by the following formula: in, It is Safety constraint function of each sub-network agent; It is The probability function of the constraint network of the sub-network agent is expressed in the state , No. Actions of the sub-network agents strategies of other subnetwork agents , No. The degree of constraint violation of each sub-network agent The probability distribution of .

[0034] The actor network is responsible for generating action strategies. The constraint network is responsible for generating action strategies through security constraint functions. To constrain the actions output by the actor network. The security constraint function is defined as follows: A critic network is constructed to estimate the joint action-state value function of each sub-network agent. This function is the policy of each sub-network agent, which is the output of the constraint network after processing. The parameters of the critic network are updated by the following formula: in, It is The parameters of the critic network of the sub-network agent, is the learning rate, It is The loss function of the critic network of each sub-network agent, namely the mean square error loss, can be calculated by the following formula: in, It is The reward function of each sub-network agent, is the state transition function, is the discount factor, It is The sub-network agent is in the next state Next action.

[0035] In the SAC algorithm of this invention, the critic network's parameters are updated using a soft update mechanism. Unlike hard updates (which completely replace old parameters with new ones), soft updates blend the parameters of the current evaluation network with those of the target network in a certain ratio, achieving a smooth parameter transition. This soft update approach stabilizes network training and prevents drastic parameter updates from causing model performance fluctuations. In ultra-large-scale road network management scenarios, where traffic conditions are constantly changing, the soft update mechanism allows the agent to retain some previous experience while learning new information, gradually adjusting its strategy to better adapt to the dynamically changing road network environment, thereby improving the algorithm's convergence speed and stability.

[0036] Beneficial effects: This invention decouples the problem of ultra-large-scale road network management into a two-stage task: subnetwork partitioning and collaborative optimization. The subnetwork partitioning stage utilizes a method combining graph-theoretic features with fuzzy C-means clustering, effectively reducing the complexity of network analysis and management while enhancing the network's robustness and fault tolerance. During the collaborative optimization stage, a constrained, heterogeneous soft actor-critic algorithm, utilizing a dual mechanism of maximum entropy policy optimization and safety constraints, enables collaborative decision-making regarding lane speed limits and closures across subnetworks. Compared to uncontrolled and route-guided approaches, the reinforcement learning-based control response of this invention outperforms measures such as average speed. In comparison with a typical multi-agent reinforcement learning algorithm, the heterogeneous multi-agent reinforcement learning algorithm (Constrainted - HASAC) of this invention demonstrates superior post-collision global reward and average speed, as well as superior convergence performance. This demonstrates the effectiveness of this invention in improving road network management efficiency, ensuring traffic safety, and enhancing algorithm performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a processing flow chart of a super-large-scale highway network control and response method based on heterogeneous multi-agent reinforcement learning according to the present invention; Figure 2 It is a flow chart of a road network partitioning method based on graph theory and fuzzy C-means clustering implemented according to the present invention; Figure 3 It is a highway network control response model based on the constraint-based heterogeneous sub-network intelligent agent soft actor-critic algorithm implemented in accordance with the present invention; Figure 4 is a comparison chart of the algorithm of the present invention and other control response methods implemented according to the present invention at different traffic levels; Figure 5 This is a comparison chart of the convergence performance of the algorithm of the present invention and other multi-agent reinforcement learning algorithms implemented according to the present invention. DETAILED DESCRIPTION

[0038] The technical solution provided by this application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of this application will become more apparent with reference to the following description.

[0039] Figure 1 This is a processing flow chart of a super-large-scale highway network control and response method based on heterogeneous multi-agent reinforcement learning according to the present invention.

[0040] The method of the present invention includes two stages: sub-network division and collaborative optimization. The sub-network division stage corresponds to step S1, and the collaborative optimization stage corresponds to steps S2-S5. The details are as follows: Sub-network division stage The road network is represented as an undirected graph G = (V, E), where V is the set of nodes and E is the set of road segments. Each node v ∈ V has a weight w(v), which represents the traffic volume at that node. Each road segment e ∈ E has a length l(e), which represents its length or distance. Calculate the degree d(v) of each node, which represents the number of road segments connected to it. Based on the degree of each node, we divide nodes into three categories: internal nodes (d(v) = 2), boundary nodes (d(v) = 1), and connecting nodes (d(v) > 2). Internal and boundary nodes belong to the same subnetwork, while connecting nodes belong to multiple subnetworks. Calculate the neighbor set N(v) of each node, which represents the other nodes connected to it. According to the neighbor set of the node, the nodes are divided into two categories: adjacent nodes ( , where u is any connected node) and isolated nodes ( , where u is any connected node). Adjacent nodes belong to the same subnetwork, and isolated nodes belong to different subnetworks. Using the fuzzy C-means clustering algorithm, the nodes are clustered according to their weights and neighbor sets to obtain k sub-networks. The goal of the fuzzy C-means clustering algorithm is to minimize the following objective function: in, is the fuzzy membership matrix, which indicates the degree to which each node belongs to each sub-network. is the cluster center vector, representing the central node of each sub-network, is a fuzzy parameter that controls the fuzziness of clustering. To balance parameters, control the weight of the node and the influence of the neighbor set. The objective function aims to optimize the calculation to make each node as close as possible to the central node of the sub-network to which it belongs in terms of spatial distribution. At the same time, it also maintains the connectivity between the node and its neighboring nodes to the greatest extent possible to ensure the rationality and coherence of the sub-network structure.

[0041] Based on the fuzzy membership matrix U, each node is assigned to one or more subnetworks to obtain the final network partitioning result. If a node's membership is greater than a given threshold, it is assigned to the corresponding subnetwork; otherwise, it is ignored. If a node is assigned to multiple subnetworks, it is considered a boundary node; otherwise, it is considered an internal node.

[0042] To prevent the sub-network from being too large, each sub-network should not exceed 20 nodes. The above process is as follows: Figure 2 shown.

[0043] Collaborative optimization stage: According to the division of different road networks, construct sub-network agents and conduct multi-agent collaborative training (such as Figure 3 ) Assume there are N heterogeneous subnetwork agents. Input parameters: At the beginning of the algorithm, some basic parameters need to be set. These parameters will determine how the algorithm operates. Temperature τ is an important parameter that controls exploration. The coefficient ρ is used to smooth the update of the target network. The batch size B determines the number of experiences considered in each update. The number of subnetwork agents n is the number of subnetwork agents participating in the learning process. K is the total number of rounds of the algorithm. The number of steps per episode T is the number of steps in each round.

[0044] Before learning begins, for both the critic network and the actor network, the target network and the main network need to be initialized. The parameters of the target network are initially the same as those of the main network, including: Critics Network , which are responsible for evaluating the quality of the current strategy.

[0045] Actor Network and , which are responsible for generating actions.

[0046] Replay buffer B is used to store experience for subsequent learning.

[0047] The algorithm learns through multiple rounds of iterations. In each round, the subnetwork agent observes the current state , and according to the actor network Select an action After executing the action, the sub-network agent will observe the new state and corresponding rewards This information is stored in the replay buffer B. Then, a mini-batch of experiences is randomly sampled from B for learning.

[0048] The critic objective is calculated as follows: Update the Q function and update the critic network by minimizing the difference between the predicted value and the target value through the gradient descent algorithm. The update formula is as follows: In order to increase the randomness of the learning process, the algorithm randomly extracts the order of the sub-network agents.

[0049] Each sub-network agent will be updated based on its own experience. The goal of the update is to maximize the expected return while keeping the entropy of the policy high enough to promote exploration. The formula for sub-network agent update is as follows: The critic network uses a soft update mechanism, and the target network is smoothly updated according to the coefficient ρ to stabilize the learning process. The update formula is as follows.

[0050] During each training, the new parameters of the target network are composed of a certain proportion of the target network parameters plus a certain proportion of the current network parameters, thereby improving training stability.

[0051] A simulation platform was built to verify the research method proposed in this paper. A distributed training framework was built using an Intel i7 13700 + RTX 3090 platform. A large-scale highway network model was constructed using SUMO as the simulation platform, including road sections and traffic settings. A collision accident in a lane at a certain location on the road network was considered a traffic event. First, experiments were conducted on large-scale road networks with different traffic conditions (1500, 2000, and 2500 vehicles / (h·lane)) and different control measures (no control, route guidance, and reinforcement learning-based control response). Figure 4 Secondly, in the case of 50 sub-networks, the heterogeneous multi-agent reinforcement learning (Constrainted-HASAC) proposed in this paper is compared with the typical multi-agent reinforcement learning algorithms MASAC, MAPPO, HAPPO, MAPDDG, HAPDDG and QMIX, as shown in the figure below. Figure 5 As shown. In terms of indicator evaluation, the global reward and average speed after the collision are used as indicators for evaluation. Among them, the parameter design of this research method is: τ=0.5, ρ=0.995, B=1000, K=100,000, T=500, w1=0.5, w2=0.3, w3=0.5. When compared with no control measures and route induction measures, the control response strategy based on reinforcement learning adopted by the present invention shows significant advantages in key indicators such as average speed. When compared with typical multi-agent reinforcement learning algorithms such as MASAC, MAPPO, HAPPO, MAPDDG, HAPDDG and QMIX, the present invention performs better in terms of global rewards and average speed, and has a faster convergence speed, which fully proves that the present invention has practical effectiveness in improving road network control efficiency, ensuring traffic safety and enhancing algorithm performance.

[0052] The above description is only a description of the preferred embodiments of the present application and does not limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical content should be regarded as equivalent valid embodiments and fall within the scope of protection of the technical solution of the present application.

Claims

1. A super-large-scale highway network control and response method based on heterogeneous multi-agent reinforcement learning, characterized by: The following steps are involved: S1: Model the highway network as a weighted directed graph, set the network partition optimization objective function, and use fuzzy C-means clustering to generate the optimal sub-network partitioning scheme; S2: Define the sub-network agent and control the state input traffic parameters, road parameters and environmental parameters; Constructing an action space that includes the speed limit and open state of each lane; Design a reward function that integrates global traffic volume, collision penalty, and emission penalty; S3: Define a joint policy entropy function to maximize the benefits of agent collaboration and introduce a safety constraint function to limit dangerous actions. Adopt a heterogeneous agent soft actor-critic algorithm to establish an actor network and a critic network. Before formal training begins, the actor and critic networks are given random network parameters to interact with the environment, generating state-action-value data and filling the experience replay pool. S4: Deploy an asynchronous training framework, with multiple actor network threads executing in parallel and interacting with the environment: each thread generates sub-network actions based on the current actor network, collects state-reward data and calculates gradients, and asynchronously updates the actor network parameters of each sub-network; the critic network uses a soft update mechanism, mixing the current network and target network parameters in a fixed ratio; S5: Randomly sample batches of data from the experience pool, calculate the temporal difference error of the critic network and update the parameters, and optimize the action probability distribution through policy gradient ascent. During the iterative process, use the safety constraint verification module to filter conflicting actions, and finally output a lane-level control strategy that meets the cross-region coordination constraints.

2. The method for super-large-scale highway network control and response based on heterogeneous multi-agent reinforcement learning according to claim 1 is characterized in that: In step S1, the sub-network division method based on graph theory and fuzzy C-means clustering is as follows: Based on graph theory and fuzzy C-means clustering, the goal is to divide a large road network into several small sub-networks, so that the nodes within each sub-network have a high similarity, while the nodes between different sub-networks have a low similarity; The similarity metric is the weight of the node and the neighbor set. The weight represents the traffic flow or importance of the node, and the neighbor set represents other nodes connected to the node. Fuzzy C-means clustering assigns nodes to one or more subnetworks instead of just one subnetwork. The process of fuzzy C-means clustering is to continuously update the node membership and subnetwork center until an optimal partitioning result is achieved. The optimal partitioning result is to minimize the objective function of the road network partitioning optimization. The objective function is composed of the weight of the node and the neighbor set, which reflects the distance and connectivity between the node and the sub-road network. The smaller the distance, the higher the connectivity, which means the more similar the node is to the sub-road network, and the smaller the objective function.

3. The method for super-large-scale highway network control and response based on heterogeneous multi-agent reinforcement learning according to claim 1 is characterized in that: In step S2, the highway control response model based on the soft actor-critic algorithm of the constrained heterogeneous sub-road network agent is as follows; For each sub-network obtained in step S1, a sub-network agent is constructed, and the state, action, and reward function of the sub-network agent are determined as follows: State function: abstract typical traffic event parameters and determine the road network traffic state input that can be obtained by the sub-road network; Status items include: traffic parameters, road parameters and environmental parameters; Among them, traffic parameters are obtained through gantry video, including the average speed and average density in each section of the sub-road network; Road parameters are obtained through road administration data and road surface detectors, including lane width and number of lanes in each section of the sub-road network; Environmental parameters include weather type, precipitation intensity, visibility distance, temperature and humidity of the sub-road network; Action function: Determine the sub-network strategy set with lanes in the sub-network as the control object based on typical active control schemes; Among them, the constraint network control objects include: Dynamic speed limit; Lane opening conditions; Reward function: Based on the reward function y based on the global road network flow and sub-road network flow, the specific form is Among them, w1, w2, w3 are the weights of each part of the reward, ; is the average speed, is the speed of the i-th car, is the total number of vehicles; is the number of vehicles involved in the collision; Average vehicle emissions ,in is the exhaust gas emission (g / km), is the emission factor (g / km) for the i-th vehicle type under the j-th fuel and the k-th pollution control level, is the mileage (km) of the i-th type of vehicle using the j-th fuel and the k-th pollution control level, is the frequency distribution of the i-th vehicle type under the j-th fuel and the k-th pollution control level; is the evaporative emission (g), is the evaporative emission rate (g / time) of the i-th type of vehicle under the j-th fuel and the k-th pollution control level, is the number of times the i-th type of vehicle travels under the j-th fuel and the k-th pollution control level, To calculate the time.

4. The method for super-large-scale highway network control and response based on heterogeneous multi-agent reinforcement learning according to claim 1 is characterized in that: In step S3, a soft actor-critic HASAC algorithm for heterogeneous sub-network agents is designed, and an action maximum entropy regularization term is added to the loss function in the actor network; a sequential update scheme is adopted, that is, only the strategy of one sub-network agent is updated each time, while the strategies of other sub-network agents remain unchanged.

5. The method for super-large-scale highway network control and response based on heterogeneous multi-agent reinforcement learning according to claim 4 is characterized in that: The actor network is based on the agent state function as input, the action function as output, and the action value composed of the reward function as the optimization target; Maximum entropy term: In order to prevent the sub-network agent from falling into a single strategy choice within the established safety constraints and losing the exploration of the environment, The sub-network agent is constructed and the maximum entropy term of the action is used as the optimization goal of the actor network to avoid the action value output by the strategy being too concentrated. The maximum entropy term is expressed as: in, is the state distribution, It is The strategy of each sub-network agent, are the strategies of other subnetwork agents; It is The action value function (joint action-state value function) of the sub-network agent, that is, given a state , No. Actions of the sub-network agents , and other subnetwork agents' strategies , predict the The expected reward of each sub-network agent; Security constraint function: When executing an action, the following constraints need to be met: in, It is The constraint function of each sub-network agent, It is The constraint threshold of each sub-network agent, the sub-network action execution needs to meet the following constraints: 1) Dynamic speed limit The constraints are: in, is the set of all lanes; It's a lane Speed ​​range; and Lanes minimum and maximum speed limits; Represents the set of all special events; It's a special event The speed range is specified below; 2) Lane opening status The constraints are the road network connectivity conditions: in, is the set of all lanes in the road network; It is a temporarily closed lane; out( )enter( ) port path; Entrance and exit collection; Construct an actor network to output the random strategy of each sub-network agent, that is, given the state and the strategy of other sub-network agents, output the action probability distribution of the current sub-network agent; the parameters of the actor network are updated by the following formula: in, It is The parameters of the actor network of the sub-network agents, is the learning rate, It is The objective function of the sub-network agent is to maximize the The policy entropy and expected reward of each sub-network agent are calculated by the following formula: in, is the state distribution, It is The joint action-state value function of the sub-network agents, that is, given a state , No. Actions of the sub-network agents , and other subnetwork agents' strategies , predict the The expected reward of each sub-network agent; Construct a constraint network to estimate the safety constraint function of each sub-network agent. That is, given the state and the strategies of all sub-network agents, directly output the actions that meet the constraints; for actions that do not meet the constraints, output any actions that meet the constraints and the constrained actions of the current sub-network agent. The parameters of the constraint network are updated by the following formula: in, It is The parameters of the constraint network of the sub-network agents, is the learning rate, It is The loss function of the constraint network of the sub-network agent, namely the log-likelihood loss, is calculated by the following formula: in, It is Safety constraint function of each sub-network agent; It is The probability function of the constraint network of the sub-network agent is expressed in the state , No. Actions of the sub-network agents strategies of other subnetwork agents , No. The degree of constraint violation of each sub-network agent The probability distribution of The actor network is responsible for generating action strategies, and the constraint network is responsible for generating action strategies through security constraint functions. To constrain the actions output by the actor network; the security constraint function is defined as follows: 。 6. The method for super-large-scale highway network control and response based on heterogeneous multi-agent reinforcement learning according to claim 4 is characterized in that: The critic network is used to estimate the joint action-state value function of each sub-network agent, that is, given the state and the strategy of all sub-network agents, the strategy is the output processed by the constraint network, and outputs the expected reward of the current sub-network agent; The parameters of the critic network are updated by the following formula: in, It is The parameters of the critic network of the sub-network agent, is the learning rate, It is The loss function of the critic network of each sub-network agent, namely the mean square error loss, can be calculated by the following formula: in, It is The reward function of each sub-network agent, is the state transition function, is the discount factor, It is The sub-network agent is in the next state Next action.

Images